Package: dnadna
DNADNADeep Neural Architecture for DNA.The goal of this package is to provide utility functions to improve
development of neural networks for population genetics.dnadnashould allow researchers to focus on their research project, be it the
analysis of population genetic data or building new methods, without the need to
focus on proper development methodology (unit test, continuous integration,
documentation, etc.). Results will thus be more easily reproduced and shared.
Having a common interface will also decrease the risk of bugs.InstallationUsing conda/mambaBecause DNADNA has some non-trivial dependencies (most notablyPyTorch) the easiest way to install it is in a conda
environment.
We will show how to install it withmamba(which is a faster
implementation ofconda).Note that all following commands will work withcondainstead ofmamba, unless otherwise specified.If you don't havecondayet, follow theofficial
instructionsto install either the fullAnacondadistribution or, advisably, the smallerMinicondadistribution.
Once you've installedconda, you can follow theMambainstallation.You can install DNADNA in an existing conda environment, but the easiest way
is to install it withmambain a new environment.  Start by creating a new
environment:$mambacreate-ndnadnawhere the name of the environment (-n) isdnadna(it won't install
DNADNA yet).Activate the environment withmamba activate dnadna.Then DNADNA can be installed and updated by running:$mambainstall-cmlgenetics-cpytorch-cbioconda-cconda-forgednadnaEach-cflag specifies a conda "channel" that needs to be searched for
some of the dependencies.  If you want to add the "mlgenetics" channel and
those of its dependencies to your list of default conda channels, you can
run:$condaconfig--addchannelsmlgenetics\--addchannelspytorch\--addchannelsconda-forge\--addchannelsbiocondaNote: The previous command works only withconda, not withmamba(yet)One can make surednadnais properly installed by runningdnadna -h.Optional dependenciesDNADNA supports logging losses (and eventually other statistics) during
training in theTensorBoardformat
for betterloss
visualization.To ensure TensorBoard support is enabled, you can install it in your active
conda environment with:$mambainstall-cconda-forgetensorboardDNADNA can be used inside Jupyter notebooks. If you are using conda/mamba environments, you need to install jupyterlab in the same environement as DNADNA:$mambaactivatednadna$mambainstall-cconda-forgejupyterlabUsing PyPIIf you prefer not to use conda, and to get DNADNA and its dependencies
directly using PyPI+pip you can run:$pipinstalldnadnaNote: We still advise to install it within a conda environment.DevelopmentAlternatively, you mayclonethe DNADNA git repository and install from
there:$gitclonehttps://gitlab.com/mlgenetics/dnadna.git$cddnadna$mambaenvcreate$mambaactivatednadna$pipinstall.Or using the lighter CPU-only environment:$gitclonehttps://gitlab.com/mlgenetics/dnadna.git$cddnadna$mambaenvcreate--fileenvironment-cpu.yml$mambaactivatednadna-cpu$pipinstall.If you plan to dodevelopmenton the package, it is advisable to choose
thegit clonesolution and install in "editable" mode by running instead:$pipinstall-e.UpdatingIf installed with conda/mamba, DNADNA can be updated by running:$mambaupdate-cmlgenetics-cpytorch-cconda-forgednadnaOr, if you already added the release channels to your default channels,
with simply:$mambaupdatednadnaNote for internal users/developersUsers of the private repository onhttps://gitlab.inria.frshould seethe development
documentationfor
notes on how to access the private repository.DockerADockerfileis also included for building a Docker image containing DNADNA (it is based
on conda, so it essentially recreates the installation environment explained
above, including GPU support.  To build the image run:$dockerbuild--tagdnadna.Make sure to run this from the root of the repository, as the entire
repository needs to be passed to the Docker build context.If run without specifying any further commands, it will open a shell with
thednadnaconda environment enabled:$dockerrun-tidnadnaHowever, you can also run it non-interactively, e.g. by specifying thednadnacommand.  Here it is also likely a good idea to mount the data
directory for your simulation/training files.  For example:$dockerrun-t-v/path/to/my/data:/data--workdir/datadnadnadnadnainitAdditional notes on the Docker image:By default it logs in as a non-root user named "dnadna" with UID 1001
and GID also of 1001.  If you start the container with the--userflag
and your own UID it will change the UID and GID of the "dnadna" user.
Make sure to specify both a UID and a GID, otherwise the group of all
files owned by "dnadna" will be changed to "root".  E.g., run:docker run -u $(id -u):$(id -g).The default workdir is/home/dnadna/dnadnawhich contains thednadnapackage source code.If you want to install your own conda packages (e.g. to use this container
for development), after starting the container it is best to create a new
conda environment cloned from the base environment, then re-install
dnadna:$condacreate-ndnadna--clonebase$condaactivatednadna$pipinstall-e.This is not done by default by the image because it would require
additional start-up time in the non-development case.  However, as
a short-cut for the above steps you can run the container withdocker run -e DEV=1.Afterwards, you can create a snapshot of this container in another
terminal, e.g. by runningdocker commit <my-container> dnadna-dev.Dependenciespython >= 3.6pytorchpandasnumpymatplotlibmsprimejsonschemapyyamltqdm(For a complete list, seesetup.cfg, orrequirements.txt.)Quickstart TutorialAfter successful installation you should have a command-line utility calleddnadnainstalled:$dnadna--helpusage: dnadna [-h] [COMMAND]dnadna version ... top-level command.See dnadna <sub-command> --help for help on individual sub-commands.optional arguments:-h, --help       show this help message and exit--plugin PLUGIN  load a plugin module; the plugin may be specified either as the file path to a Python module, or the name of a module importable on the current Python module path (i.e. sys.path); plugins are just Python modules which may load arbitrary code (new simulators, loss functions, etc.) during DNADNA startup); --plugin may be passed multiple times to load multiple plugins--trace-plugins  enable tracing of plugin loading; for most commands this is enabled by default, but for other commands is disabled to reduce noise; this forces it to be enabled-V, --version    show the dnadna version and exitsub-commands:init          Initialize a new model training configuration and directorystructure.preprocess    Prepare a training run from an existing model trainingconfiguration.train         Train a model on a simulation using a specified pre-processedtraining config.predict       Make parameter predictions on existing SNP data using an alreadytrained model.simulation    Run a registered simulation.This implements a number of different sub-commands for different training
and simulation steps.  Thednadnacommand can be used either starting with
an existing simulation dataset (which may need to be first be converted to
theDNADNA Dataset
Format,
or you may usednadna's simulator interface to create a new simulation
dataset.Here we step through the complete process from configuring and generating a
simulation, to running data pre-processing on the simulation, and training a
network based off that simulation.If you already have simulation data in theDNADNA Data
Formatyou can skip straight to theinitializationstep.Simulation initialization and configurationTo initialize a simulation, we must first generate a config file and output
folder for it, using thednadna simulation initcommand:$dnadnasimulationinitmy_datasetone_eventWriting sample simulation config to my_dataset/my_dataset_simulation_config.yml ...Edit the config file as needed, then run this simulation with the command:dnadna simulation run my_dataset/my_dataset_simulation_config.ymlThis will create a directory in the current directory, namedmy_dataset/,
and initialize it with a config file pre-populated with sample parameters
for the built-inone_event example
simulator.Before running the simulation we may want to adjust some of the parameters.
Openmy_dataset/my_dataset_simulation_config.ymlin your favorite text
editor.  By default we see thatn_scenariosis100withn_replicatesof 3 per scenario (in the one_event examplen_replicatesis the number of independently simulated genomic regions). You can change it to20scenarios with2replicates, i.e. 100 simulations total, which is good for this quick demo but too low for training a real model. For a real training with enough computing resources, we would recommend changing these numbers to higher values such as20000and100(2 million simulations which take a very long time). You may
also set theseedoption to seed the random number generator for
reproducible results.  The resulting file (with none of the other settings
changed) should look like:# my_dataset/my_dataset_simulation_config.ymldata_root:.n_scenarios:20n_replicates:2seed:2...Running the simulationNow to run the simulation we configured, we rundnadna simulation run,
passing it the path to the config file we just edited.  If you run this in a
terminal it will also display a progress bar:$dnadnasimulationrunmy_dataset/my_dataset_simulation_config.yml... INFO;  Running one_event simulator with n_scenarios=20 and n_replicates=2... INFO;  Simulation complete!... INFO;  Initialize model training with the command:... INFO;... INFO;      dnadna init --simulation-config=my_dataset/my_dataset_simulation_config.yml <model-name>Model initializationThe main command for initialize DNADNA isdnadna init, which assumes we
already have a simulation (such as the one we just generated) in the
standardDNADNA Data Format.  Although this command can be
run without any arguments (producing a default config file), if we pass it
the path to our simulation config file it will output a config file
appropriate for use with that simulation:$dnadnainit--simulation-config=my_dataset/my_dataset_simulation_config.ymlmy_modelWriting sample preprocessing config to my_model/my_model_preprocessing_config.yml ...Edit the dataset and/or preprocessing config files as needed, then run preprocessing with the command:dnadna preprocess my_model/my_model_preprocessing_config.ymlAfter runningdnadna init, it is expected that the user will manually edit
the sample config file that it outputs, in order to exactly specify how they
want to train their model, and on which parameters.  In fact, the default
template is going to be good enough for our demo simulation, except for one
bit that will give us trouble.The optiondataset_splits:has a default value meaning 70% of our
scenarios will be used for training, and only 30% for validation.  Since,
for this quick demo, we only have 20 scenarios, the validation set will be
too small.  Open the filemy_model/my_model_preprocessing_config.ymlin your
editor and change this so that our dataset is split 50/50 between training
and validation:# my_model/my_model_preprocessing_config.yml# ...dataset_splits:training:0.5validation:0.5Under normal use you would set these ratios however you prefer.  You can
also include atestset of scenarios to be set aside for testing your
model.Pre-processingBefore training a model, some data pre-processing must be performed on the
data set; the output of this pre-processing can depend on the settings in
the preprocessing config file that was output bydnadna init.  To do this,
simply run:$dnadnapreprocessmy_model/my_model_preprocessing_config.yml... INFO;  Removing scenarios with:... INFO;   - Missing replicates... INFO;   - Fewer than 500 SNPs... INFO;  ...... INFO;  Using ... CPU for checking scenarios... INFO;  20 scenarios out of 20 have been kept, representing 40 simulations... INFO;  Splitting scenarios between training and validation set... INFO;  Standardizing continuous parameters... INFO;  Writing preprocessed scenario parameters to: .../my_model/my_model_preprocessed_params.csv... INFO;  Writing sample training config to: .../my_model/my_model_training_config.yml... INFO;  Edit the training config file as needed, then start the training run with the command:... INFO;... INFO;      dnadna train .../my_model/my_model_training_config.ymlThis will produce a<model_name>_training_config.ymlfile containing the
config file prepared for training your model.TrainingTo run a model training, after pre-processing usednadna train, giving it
the path to the pre-processed training config file as output by the last
step.In order to make the training run a little faster (just for this example)
let's also edit the training config file to limit it to one epoch:# my_model/my_model_training_config.yml# ...# name and parameters of the neural net model to trainnetwork:name:CustomCNN# number of epochs over which to repeat the training processn_epochs:1...Then rundnadna trainon the training config file:$dnadnatrainmy_model/my_model_training_config.yml... INFO;  Preparing training run... INFO;  20 samples in the validation set and 20 in the training set... INFO;  Start training... INFO;  Networks states are saved after each validation step... INFO;  Starting Epoch #1... INFO;  Validation at epoch: 1 and batch: 1... INFO;  Compute all outputs for validation dataset...... INFO;  Done... INFO;  training loss = 1.0222865343093872 // validation loss = 1.2975229024887085... INFO;  Better loss found on validation set: None --> 1.2975229024887085... INFO;  Saving model to ".../my_model/run_000/my_model_run_000_best_net.pth" ...... INFO;  Compute all outputs for validation dataset...... INFO;  Done... INFO;  --- 3.185938596725464 seconds ---... INFO;  --- Best loss: 3.892427444458008... INFO;  Saving model to ".../my_model/run_000/my_model_run_000_last_epoch_net.pth" ...... INFO;  You can test the model's predictions on a test dataset by running the command:... INFO;... INFO;      dnadna predict .../my_model/run_000/my_model_run_000_last_epoch_net.pth <dataset config file or paths to .npz files>By default this will output a directory for your training run undermodel_name/run_NNNwhereNNNis an integer run ID.  The run ID starts at
0, and by default the next unused run ID is used.  However, you may also
pass the--run-idargument to give a custom run ID, which may be either an
integer, or an arbitrary string.Following a successful training run will output a<model_name>_run_<run_id>_last_epoch_net.pthfile in the run directory,
containing the final trained model in a pickled format, which can be loaded
by thetorch.loadfunction.Under the run directory this will also produce a<model_name>_<run_id>_training_config.ymlfile containing the final config
file prepared for this training run.  This contains a complete copy of the
"base" training config use used to rundnadna trainas well as a
complete copy of the simulation config.  This information is copied in full
for the purpose of provenance and reproducibility of a training run.The expectation is that between multiple training runs, you may modify
the "base" config to tune the training, either by modifying the original
config file directly, or by copying it and editing the copy.  In any case,
the final configuration used to perform the training run is saved in the run
directory and should not be modified.You can keep track of all training and validation losses with TensorBoardPredictionGiven the trained network, we can now use it to make (or confirm)
predictions on new datasets.  To demonstrate we'll run thednadna predictmodel over part of the existing dataset we just used to train the model,
though in practice it could be run on any data that conforms (e.g. in
dimensions) to the dataset the model was trained on.  The output is a CSV
file containing the parameter predictions for each input:$dnadnapredictmy_model/run_000/my_model_run_000_last_epoch_net.pth\my_dataset/scenario_04/*.npzpath,event_time,recent_size,event_size.../my_dataset/scenario_04/my_dataset_04_0.npz,-0.06392800807952881,-0.11097482591867447,-0.12720556557178497.../my_dataset/scenario_04/my_dataset_04_1.npz,-0.06392764300107956,-0.11097370833158493,-0.12720371782779694Data formatDNADNA has a prescribed filesystem layout and file format for the datasets
its works on.  Some of the details of this layout can be modified in the
configuration files, and in a future version will be further customizable by
plugins.But the default format assumes that SNP data (SNP matrices and associated
SNP position arrays) are stored in NumPy'sNPZformat with one file per SNP.  They are organized on disk by scenario like:\_ my_simulation/
    \_ my_simulation_params.csv  # the scenario parameters table
    |_ my_simulation_dataset_config.yml  # the simulation config file
    |_ scenario_000/
        \_ my_simulation_000_00.npz  # scenario 0 replicate 0
        ...
        |_ my_simulation_000_NN.npz
    |_ scenario_001/
        \_ my_simulation_001_00.npz  # scenario 1 replicate 0
        ...
        |_ my_simulation_001_NN.npz
    ...
    |_ scenario_NNN/
        \_ my_simulation_NNN_00.npz
        ...
        |_ my_simulation_NNN_NN.npzThe filemy_simulation_params.csvcontains the known target values for
each parameter of the simulation, on a per-scenario basis.  It is currently
a plain CSV file which must contain at a minimum 3 columns:Ascenario_idxcolumn giving the scenario number.Ann_replicatescolumn which specifies the number of replicates in that
scenario.One or more additional columns containing arbitrary parameter names, and
their values each scenario in the dataset.For example:scenario_idx,mutation_rate,recombination_rate,event_time,n_replicates
0,1e-08,1e-08,0.3865300203456797,-0.497464473948751,100
1,1e-08,1e-08,0.19344551118300793,0.16419897912977574,100
...An associated config file here namedmy_simulation_dataset_config.ymlprovides further details about how to load the dataset to the DNADNA
software.  An example dataset config can be generated by runningdnadna init.SeeThe DNADNA Dataset
Formatfor more details.DevelopmentSeethe development
documentationfor full details on how to set up and use a development environment and
contribute to DNADNA.Detailed usageFor the full usage manual see theDNADNA
Documentation.