Package: neural-admixture
Neural ADMIXTURENeural ADMIXTURE is an unsupervised global ancestry inference technique based on ADMIXTURE. By using neural networks, Neural ADMIXTURE offers high quality ancestry assignments with a running time which is much faster than ADMIXTURE's. For more information, we recommend readingour corresponding article.The software can be invoked via CLI and has a similar interface to ADMIXTURE (e.g.the output format is completely interchangeable). While the software runs in both CPU and GPU, we recommend using GPUs if available to take advantage of the neural network-based implementation.System requirementsHardware requirementsThe successful usage of this package requires a computer with enough RAM to be able to handle the large datasets the network has been designed to work with. Due to this, we recommend using compute clusters whenever available to avoid memory issues.Software requirementsThe package has been tested on both Linux (CentOS 7.9.2009, Ubuntu 18.04.5 LTS) and MacOS (BigSur 11.2.3, Intel and Monterey 12.3.1, M1). If using GPUs, make sure CUDA drivers are properly installed.We recommend creating a fresh Python 3.9 environment usingvirtualenv(orconda), and then install the packageneural-admixturethere. As an example, forvirtualenv, one should launch the following commands:$virtualenv--python=python3.9~/venv/nadmenv$source~/venv/nadmenv/bin/activate(nadmenv)$pipinstallneural-admixtureInstallation GuideThe package can be easily installed in at most a few minutes usingpip(make sure to add the--upgradeflag if updating the version):(nadmenv)$pipinstallneural-admixtureUsageRunning Neural ADMIXTURETo train a model from scratch, simply invoke the following commands from the root directory of the project. For more info about all the arguments, please runneural-admixture train --help. If training a single-head version of the network suffices, please use the flag--kinstead of--min_kand--max_k. Note that VCF, BED, PGEN and HDF5 files are supported as of now.For unsupervised Neural ADMIXTURE (single-head):$neural-admixturetrain--kK--nameRUN_NAME--data_pathDATA_PATH--save_dirSAVE_PATH--init_fileINIT_FILEFor unsupervised Neural ADMIXTURE (multi-head):$neural-admixturetrain--min_kMIN_K--max_kMAX_K--nameRUN_NAME--data_pathDATA_PATH--save_dirSAVE_PATH--init_fileINIT_FILEFor supervised Neural ADMIXTURE:$neural-admixturetrain--kK--supervised--populations_pathPOPS_PATH--nameRUN_NAME--data_pathDATA_PATH--save_dirSAVE_PATH# only single-head support at the momentAs an example, the following ADMIXTURE call$./admixturesnps_data.bed8-s42would be mimicked in Neural ADMIXTURE by running$neural-admixturetrain--k8--data_pathsnps_data.bed--save_dirSAVE_PATH--init_fileINIT_FILE--namesnps_data--seed42with some parameters such as the decoder initialization or the save directories not having a direct equivalent.Several files will be output to theSAVE_PATHdirectory (thenameparameter will be used to create the whole filenames):If the unsupervised version is run, aPicklebinary file containing the PCA object (using theinit_nameparameter), as well as an image file containing a PCA plot.A.Pfile, similar to ADMIXTURE.A.Qfile, similar to ADMIXTURE.A.ptfile, containing the weights of the trained network.A.jsonfile, with the configuration of the network.The last two files are required to run posterior inference using the network, so be aware of not deleting them accidentally! Logs are printed to thestdoutchannel by default. If you want to save them to a file, you can use the commandteealong with a pipe:$neural-admixturetrain--k8...|teerun.logInitialization methodAs described in the article, Neural ADMIXTURE's decoder(s) can be initialized using several methods, which will be indicated by the requiredinitializationargument. The best-performing initialization method depends, mainly, on the structure of the data. The main options are:pckmeans: initialize using PCK-Means (Algorithm 1 in the paper). Performs best if data contains a large number of single-ancestry individuals.pcarchetypal: initialize using PCArchetypal (Algorithm 2 in the paper). Performs best if data contains a large number of admixed individuals.Thesupervisedinitialization method is used by default (and can only be used) when using supervised mode.Inference mode (projective analysis)ADMIXTURE allows reusing computations in theprojective analysismode, in which theP(F, frequencies) matrix is fixed to an already known result and only the assignments are computed. Due to the nature of our algorithm, assignments can be computed for unseen data by simply feeding the data through the encoder. This mode can be run by typinginferinstead oftrainright after theneural-admixturecall.For example, assuming we have a trained Neural ADMIXTURE (namednadm_test) in the path./outputs, one could run inference on unseen data (./data/unseen_data.vcf) via the following command:$neural-admixtureinfer--namenadm_test--save_dir./outputs--out_nameunseen_nadm_test--data_path./data/unseen_data.vcfFor this command to work, files./outputs/nadm_test.ptand./outputs/nadm_test_config.json, which are training outputs, must exist. In this case, only a.Qwill be created, which will contain the assignments for this data (the parameter of the flagout_namewill be used to generate the output file name). This file will be written in the--save_dirdirectory (in this case,./outputs).Pretrained Neural ADMIXTUREAs also mentioned in the paper, Neural ADMIXTURE can be used to learn a function (through the encoder) given the results of the frequency matrixP/Fof another algorithm so out-of-training data inference can be performed using the structure learnt by the other algorithm. The following arguments should be used to run Neural ADMIXTURE in this mode:--initialization pretrained: indicates that an ADMIXTURE-like.Pfile must be loaded to initialize the decoder weights. The path of the file is specified using the--init_fileargument.--freeze_decoder: indicates that the decoder weights will be frozen during training.If the second argument is skipped, then the decoder weights will be updated and the solution won't yield exactly the sameP/Fmatrix that was used as input.Supervised Neural ADMIXTUREThe supervised version of the algorithm can be used when all samples have a corresponding population label. This can be very benificial, especially when dealing with large imbalances in the data (e.gdata contains 1K samples from Pop1 and 50 samples from Pop2).In order to use the supervised mode, the flag--supervisedmust be passed when invoking the software, along with a--populations_pathpointing to the file where the ancestries are defined. The latter file must be a single-column, headerless, plain text file where rowidenotes the ancestry for thei-th sample in the data. The character-must be used for samples whose ancestry is missing/unknown. If validation data is provided, then the corresponding populations file path must be passed through the--validation_population_pathargument.The supervised mode works by adding a scaled classification loss to the bottleneck of the algorithm (Equation 5 of the paper). The scaling factor can have a big impact on the performance. If it is too small, then the supervised loss will have little impact on the training, so results would be similar to an unsupervised run. On the other hand, if it is too large, then the supervision will dominate training, making the network overconfident in its predictions: essentially, one would get only binary assignments. The default value of the scaling factor isη=0.05, and can be controlled using the parameter--supervised_loss_weight.Basically, if on validation data you are getting single-ancestry estimations when you expect admixed estimations, try setting a smaller value for the supervised loss scaling factorη(--supervised_loss_weight) and/or a larger value for the L2 penalty termλ(--l2_penalty, defaults to 0.0005); and vice versa.Moreover, note that the initialization method chosen will have no effect, as the supervised method is always used when using the supervised version.Other optionswarmup_epochs: number of warmup epochs to get a good initialization for the encoder. During the warmup epochs, the decoder is frozen and only the encoder weights, which compute Q, are updated. Set to 0 to skip warmup. Defaults to 10.l2_penalty: indicates penalty to apply to encoder weights. Defaults to 0.0005. The greater the penalty, the "softer" the cluster assignments will be, limiting to a uniform distribution.batch_size: number of samples used at every update. If you have memory issues, try setting a lower batch size. Defaults to 400.pca_components: dimension of the PCA projection for the PC-KMeans and PCArchetypal initializations. Defaults to 8.max_epochs: maximum number of times the whole training dataset is used to update the weights. Defaults to 50.tol: will stop optimization when difference in objective function between two iterations is smaller than this value (after epoch 15). Defaults to 1e-6.learning_rate: dictates how large an update to the weights will be. If you find the loss function oscillating, try setting a lower value. If convergence is slow, try setting a higher value. Defaults to 1e-5.seed: RNG seed for replication purposes. Defaults to 42.Using Plink2 binary files (.pgen)If the data format you will be working on isPlink2 Binary Files (.pgen, .psam, .pvar)then you also need to install the packagepgenlib. This package is not available in PyPi, but is included in theplink repository. Installation instructions can be found in thecorrespondingREADME.mdfile. While you will need to clone the whole repository, you can remove it after installing the package, unless you plan to work with it.Experiments replicationThe datasetsAll-Chms,Chm-22andChm-22-Simused in the Experiments section of the article can be found infigshare. For descriptions of the datasets, please refer to the corresponding section in the paper. The exact hyperparameters used in the experiments to allow replication can be found in the Supplementary Table 3 of the article.DemoTo run the software with a small demo dataset, check the instructions inthe corresponding folder of the repository.LicenseNOTICE: This software is available for use free of charge for academic research use only. Academic users may fork this repository and modify and improve to suit their research needs, but also inherit these terms and must include a licensing notice to that effect. Commercial users, for profit companies or consultants, and non-profit institutions not qualifying as "academic research" should contact the authors for a separate license. This applies to this repository directly and any other repository that includes source, executables, or git commands that pull/clone this repository as part of its function. Such repositories, whether ours or others, must include this notice.CiteWhen using this software, please cite the following paper:@article{dominguezmantes23,
	abstract = {Characterizing the genetic structure of large cohorts has become increasingly important as genetic studies extend to massive, increasingly diverse biobanks. Popular methods decompose individual genomes into fractional cluster assignments with each cluster representing a vector of DNA variant frequencies. However, with rapidly increasing biobank sizes, these methods have become computationally intractable. Here we present Neural ADMIXTURE, a neural network autoencoder that follows the same modeling assumptions as the current standard algorithm, ADMIXTURE, while reducing the compute time by orders of magnitude surpassing even the fastest alternatives. One month of continuous compute using ADMIXTURE can be reduced to just hours with Neural ADMIXTURE. A multi-head approach allows Neural ADMIXTURE to offer even further acceleration by computing multiple cluster numbers in a single run. Furthermore, the models can be stored, allowing cluster assignment to be performed on new data in linear time without needing to share the training samples.},
	author = {Dominguez Mantes, Albert and Mas Montserrat, Daniel and Bustamante, Carlos D. and Gir{\'o}-i-Nieto, Xavier and Ioannidis, Alexander G.},
	doi = {10.1038/s43588-023-00482-7},
	id = {Dominguez Mantes2023},
	isbn = {2662-8457},
	journal = {Nature Computational Science},
	title = {Neural ADMIXTURE for rapid genomic clustering},
	url = {https://doi.org/10.1038/s43588-023-00482-7},
	year = {2023},
	bdsk-url-1 = {https://doi.org/10.1038/s43588-023-00482-7}}