Package: dt4dds
🧬 dt4dds - Digital Twin for DNA Data StorageOverviewWeb-based ToolSystem RequirementsInstallation GuideLicenseOverviewdt4ddsis a Python package  providing a customizable, digital representation of the widely-used DNA data storage workflow involving array synthesis, PCR, Aging, and Sequencing-By-Synthesis. By modelling each part of such user-defined workflows with fully customizable experimental parameters,dt4ddsenables data-driven experimental design and rational design of redundancy.dt4ddsalso includes a pipeline for comprehensively analyzing errors in sequencing data, both from experiments and simulation. This Python package is used in the following publication:Andreas L. Gimpel, Wendelin J. Stark, Reinhard Heckel, Robert N. Grass. Experimental quantification of errors and biases in DNA data storage for a digital twin. Manuscript in preparation.The Jupyter notebooks and associated code used for generating the figures in the manuscript are found in thedt4dds_notebooks repository.Web-based ToolA web-based version ofdt4ddswith an easy-to-use graphical user interface and the most common workflows is available atdt4dds.ethz.ch.System requirementsHardware requirementsThis package only requires a standard computer. Depending on the size and complexity of the simulated workflows, sufficient RAM to support the in-memory operations is required. The required amount of RAM can be reduced by decreasing the number of cores used for parallelization (see config below), at the cost of increased run time.Software requirementsThis package is compatible with Windows, macOS and Linux. The package has been developed and tested on Ubuntu 20.04 using Python 3.10. The following Python packages are required:numpy
scipy
pandas
biopython
edlib
numba
plotly
PyYAML
rapidfuzz
ruamel.yaml
tqdmFor the analysis pipeline, BBMap (v38.99) is required and assumed to be installed in~/.local/bin/.Installation guideTo install this package from PyPi, usepip3installdt4ddsTo install this package from Github, usegitclonehttps://github.com/fml-ethz/dt4ddscddt4dds
python3setup.pyinstall.DemoExemplary scripts using this package are providedin the demo folder:Basic demoshowing a simple data storage workflowAdvanced demoshowcasing a workflow with custom experimental parametersAll scripts run within seconds on a standard computer and require less than 1 GB of RAM.Scripts for reproductionAll scripts used in the manuscript for internal and external validation, as well as the case study are providedin the scripts folder:Interal validationreproducing the experimental conditions of the manuscriptGenerational experiments by Koch et al.for external validationCase studyas an example application for the rational design of redundancyTo run these scripts, provide a valid filepath where output files will be stored as the first argument to the command line. As these scripts run multiple complex workflows with a large number of sequencing endpoints at high coverages sequentially, run-times of around 1-3 h are to be expected when using four cores of a standard desktop CPU. Around 64 GB of RAM are recommended.Instructions for useImport the package in your script:importdt4ddsAt this point, you can set custom configuration options and enable logging, if desired:dt4dds.default_logging()# enable logging outputdt4dds.config.enable_multiprocessing=True# enable parallelizationdt4dds.config.n_processes=2# limit the number of parallel processes to 2For the full documentation of configuration options, refer toconfig.py.Representing sequences and oligo poolsTwo datastructures are used to represent individual sequences and oligo pools:SeqandSeqPool. TheSeqdatastructure encapsulates a string representation of the oligo sequence, while theSeqPooldatastructure encapsulates adictassociating each sequence with an abundance. Both datastructures are accessible in thedt4dds.datastructuresnamespace, however they are usually generated by other functions such as the synthesis process (see below) or those inthe generators module.Convenience methodsBoth theSeqandSeqPooldatastructures implement convenience methods for common operations. In the case of theSeqPooldatastructure for example, parameters of the oligo pool such as its mass concentration or the number of individual sequences may be accessed:print(pool.mass_concentration)# in ng/uLprint(pool.n_sequences)The pool may also be sampled by weight or volume:sampled_pool=pool.sample_by_mass(50)# in ngsampled_pool=pool.sample_by_volume(50)# in uLOr exported to other formats:pool.save_as_csv("my/path.csv")pool.save_as_fastq("my/path.fastq")For the full documentation of convenience methods, refer toseq.pyandseqpool.py.Building a workflowAll processes are provided in thedt4dds.processesnamespace. In general, all process modules are implemented as classes that require instantiation, during which non-default experimental parameters can be optionally provided (see below). Process instances do not perform their actions directly, but must be executed with theirprocessmethod. With the exception of the synthesis module, all modules'processmethods require aSeqPooldatastructure as input, representing the oligo pool the process is performed with. With the exception of the sequencing module, all modules'processmethods return aSeqPooldatastructure as output, representing the state of the oligo pool after the process.Using theSeqPooldatastructure returned by one process as the input to another process starts a workflow. AnySeqPooldatastructure may be (re-)used in multiple processes to investigate the impacts of different workflows on the same oligo pool.SynthesisThedt4dds.processes.ArraySynthesis()class models array-based oligo pool synthesis. It requires the design sequences (without primer regions) as input, and aSeqPoolcan be sampled by mass or moles after invocation of itsprocess()method:# read sequences from referenceseq_list=dt4dds.tools.fasta_to_seqlist('./design_files.fasta')# set up synthesis, add non-default parameters as arguments if desiredarray_synthesis=dt4dds.processes.ArraySynthesis()# perform synthesis and sample to receive a SeqPool datastructurearray_synthesis.process(seq_list)pool=array_synthesis.sample_by_mass(1)For the full documentation of methods, refer toarray_synthesis.py. To see customization options and more details, see thedemo filesandreproduction scripts.PCRThedt4dds.processes.PCR()class models amplification by polymerase chain reaction for a user-defined number of cycles. It requires aSeqPoolas input, and yields aSeqPoolrepresentative of the amplified oligo pool by invocation of itsprocess()method:# set up PCR with 20 cycles, add further non-default parameters as arguments if desiredpcr=dt4dds.processes.PCR(n_cycles=20)# perform PCR and receive the oligo pool representation after PCRpost_PCR_pool=pcr.process(pool)For the full documentation of methods, refer topcr.py. To see customization options and more details, see thedemo filesandreproduction scripts.AgingThedt4dds.processes.Aging()class models decay for a user-defined number of half-lives. It requires aSeqPoolas input, and yields aSeqPoolrepresentative of the decayed oligo pool by invocation of itsprocess()method:# set up aging for one half-life, add further non-default parameters as arguments if desiredaging=dt4dds.processes.Aging(fixed_decay_ratio=0.5)# perform aging and receive the oligo pool representation after decaypost_decay_pool=aging.process(pool)For the full documentation of methods, refer toaging.py. To see customization options and more details, see thedemo filesandreproduction scripts.SequencingThedt4dds.processes.SBSSequencing()class models sequencing-by-synthesis based on Illumina's sequencing platforms. It requires theSeqPoolas input, and invocation of itsprocess()method yields a user-defined number of sequencing reads saved at the provided output directory:# set up sequencing to a specific folder with 1M readssbs_sequencing=dt4dds.processes.SBSSequencing(output_directory="./sequencing/",n_reads=100000)# perform sequencing, FASTQ files will be deposited in the output directorysbs_sequencing.process(pool)For the full documentation of methods, refer tosbs_sequencing.py. To see customization options and more details, see thedemo filesandreproduction scripts.Setting custom parametersAll process classes support full customization of the experimental settings and error parameters used for simulation. Custom parameters must be supplied during instantiation, either as keyword arguments or by providing the appropriate, customized settings object fromdt4dds.settings. For example, to customize some parameters for PCR, both options can be used:# set up PCR with keyword argumentspcr=dt4dds.processes.PCR(n_cycles=20,polymerase_fidelity=5,efficiency_mean=0.9)# set up PCR with a re-usable settings objectpcr_settings=dt4dds.settings.PCRSettings(n_cycles=20,polymerase_fidelity=5,efficiency_mean=0.9)pcr=dt4dds.processes.PCR(pcr_settings)For documentation of all settings and details about the settings objects indt4dds.settings, refer tosettings.py.Provided defaultsThedt4dds.settings.defaultsmodule provides multiple default settings objects for some processes. For example, for the two different synthesis providers characterised in the manuscript:# set up electrochemical synthesissynthesis_settings=dt4dds.settings.defaults.ArraySynthesis_CustomArray()# set up synthesis by material-depositionsynthesis_settings=dt4dds.settings.defaults.ArraySynthesis_Twist()For documentation of all defaults, refer todefaults.pyand the configuration files in thesettings folder.Documentation of all parametersFor documentation of all settings, refer tosettings.py.Analysing errors and biasesThe pipeline for analysing errors and biases is provided in thedt4dds.analysismodule. The convenience functions, provided as a command-line interface inrunner.pyand described below, require the sequencing datasets as gzipped FASTQ (i.e.,R1.fq.gzandR2.fq.gz) and the reference sequences as a FASTA-formatted file (i.e.,design_files.fasta) in a common directory.Characterising errorsTo characterize the errors present in a single- or paired-read sequencing dataset in gzipped FASTQ format, the three convenience methods inrunner.pycan be used:# analyse 100000 paired readspython3runner.pypaired<path/to/folder/>-n100000# analyse 100000 single readspython3runner.pysingle<path/to/folder/>-n100000# analyse 100000 PhiX readspython3runner.pyphix<path/to/folder/>-n100000In all cases, the scripts deposit the analysis results in the same folder, stratified by similarity group. The remaining command line arguments are described inrunner.py.Quantifying coverageTo generate coverage data for a single- or paired-read sequencing dataset, the two BBMMap-based convenience methods inrunner.pycan be used:# analyse coverage of paired readspython3runner.pyscafstats_paired<path/to/folder/># analyse coverage of single readspython3runner.pyscafstats_single<path/to/folder/>In both cases, the scripts deposit the analysis results in the same folder, in thescafstats.txtfile.LicenseThis project is licensed under the GPLv3 license, seehere.