dataesr/scientific-paragraphs-categorization huggingface dataset:
We assessed the AGs generated by these models with multiple statistics measuring data quality and possible privacy leakage. First, we compared the new models with the ones from  [21]  using the same 10,000 SNPs extracted from the 1000 Genomes human dataset  [31]  and then trained these models with a larger 65,535-SNP dataset to generate long sequence AGs. We performed multiple tests to evaluate (i) the quality of the generated haplotypes via allele frequency spectrum, linkage disequilibrium, population structure and complex haplotypic integrity analyses such as 3-point correlations and distribution of k-mer motifs; and (ii) privacy preservation via distance-based metrics, membership inference attacks and overfitting/underfitting scores.
Data The two sets of 1000 Genomes  [31]  data we used include: (i) 10,000 SNPs from chromosome 15 (between 27,379,578-29,625,035 base pairs, *2 megabase pairs) identical to the ones picked by  [21] , and (ii) 65,535 SNPs from chromosome 1 between 534,247-81,813,279 base pairs, *80 megabase pairs) within the Omni 2. 5 genotyping array framework. Further downsampling of the array framework was performed to create a dataset with a reasonable SNP number for faster training trials and the specific number of 65,535 SNPs was decided to be in the form of (2n-1) for easy implementation of convolutional scaling. For the 10,000-SNP dataset, we used padding with zeros to match the (2n-1) form. Both datasets have the same 2504 individuals and 5008 phased haplotypes used for training the models. The data format is the same as  [21]  where the rows are phased haplotypes and columns are positions which hold alleles represented by 0 (reference) and 1 (alternative).
RBM implementation An RBM is a generative stochastic neural network  [33]  defined as a probability distribution over a set of visible units, s, representing SNPs in our case, and hidden units, t, where both type of variables interact via a pairwise weight matrix w and the local biases y and Z help to adjust the mean value of each unit: pðs ; tÞ ¼ 1 Z expð X ia w ia s i t a þ X i y i s i þ X a Z a t a Þ In this study, we used binary units {0, 1} for both the visible and hidden nodes, and the number of hidden units was chosen to be about the same order as the number of visible ones. The likelihood of such model is given by: L ¼ P M m¼1 h P i y i s ðmÞ i þ P a logð1 þ expð P i w ia s ðmÞ i þ Z a ÞÞ i À M logðZÞ where the index m is indexing the samples of the dataset (M being the total number of samples) and Z is the normalization constant, or partition function, of the probability distribution.
In the provided experiments, the initial conditions were chosen uniformly at random (each unit having equal probability to be 0 or 1) and the number of MC steps was either 100 (for the 65,535-SNP dataset) or 200 (for the 10,000-SNP dataset) but the qualitative results were mostly not affected. The learning rate of the model was chosen such that the eigenvalues of the learning weight matrix are smoothly increasing during the first epochs from almost zero to values of *O(1). For the generation of very long sequences, we designed a novel procedure based on conditional-RBMs (CRBMs)  [23, 35]  (Fig  2 ). The CRBM consists in the learning of correlation patterns conditionally to some input variables. Hence, instead of considering all the dataset X, we separate it into two parts X = X P [ X I , and the CRBM will learn to generate X I (the inferred variables) based on X P (the pinned variables).
Therefore, we design a gradient that will learn how to generate the variables X I given that we provide the variables X p. In practice, denoting s as the variables to be inferred and x as the pinned ones, we will maximize the following quantity: pðs 2 Ijx 2 PÞ ¼ 1 Z expð X ia w ia s i t a þ X i y i s i þ X a Z a t a þ X a t a X j w ja x j Þ Therefore, for each sample of the dataset, this construction is similar to a classical RBM with additional biases for the hidden nodes which depend on the pinned variables. During the learning, we infer the parameters of the model such that when showing a configuration of pinned variables X P , the model will generate a sample that is correlated to it accordingly. This conditional model can be used to learn very long chains of variables. Let us consider that we have a dataset with 100,000 input variables.
3. We follow the same procedure until we finally generate the whole 100,000 variables. This method was used to generate the large-scale 65,535-SNP dataset. We first trained a "normal" RBM on the first 5,000 input variables. Then, we made a set of 10,000-input conditional RBMs, where the first 5,000 variables were used as pinned variables. All the models were trained with 1000 hidden nodes, and a learning rate of β = 0. 005. The learning dynamics uses the Rdm-k method: each chain was generated starting from the uniform Bernoulli distribution, with k = 100 for the training of the RBM and the CRBMs. A recurrent issue with RBM (or training in general) is to decide when to stop the training. While in supervised setting it is easy to monitor the loss function (or the number of correctly classified samples), it is not the case for RBM since the partition function is intractable.
In this work, the learning was affected by the small number of samples given the dimension of the inputs. Therefore (and to avoid overfitting), the meta-parameters such as the number of hidden nodes and the number of epochs were fixed a posteriori, by investigating various trained machines with different values of these parameters and choosing the one giving a good AA TS score. The detailed python scripts can be accessed at https://gitlab. inria. fr/ml_ genetics/public/artificial_genomes. The RBM implementations are based on the pytorch library and handle GPU to perform both training and sampling. Another example of the out-of-equilibrium code applied to images can be found at https://github. com/ AurelienDecelle/TorchRBM.
01, except for the final layers. The decoder final layer is followed by a sigmoid function, whereas the encoder final μ and σ layers have linear activation. We used Adam optimizer with default settings and the learning rate set to 0. 001. Similarly to the WGAN, we evaluated the training based on coherence of the PCA performed on real and generated genomes (components 1 to 4). We could not successfully train a VAE model which generates plausible AGs for the 65,535-SNP dataset. Further architecture and hyperparameter optimization is needed to better assess VAE models in this context. VAE models were coded with python-3. 9 and pythorch-1. 11  [32]. The detailed python scripts can be accessed at https://gitlab. inria. fr/ ml_genetics/public/artificial_genomes.
Nearest neighbour adversarial accuracy (AA TS ) Similarly to  [21] , we assessed the overfitting/underfitting characteristics of AGs using the AA TS score  [37]. AA TS is calculated as follows: AA truth ¼ 1 n X n i¼1 1ðd TS ðiÞ > d TT ðiÞÞ AA syn ¼ 1 n X n i¼1 1ðd ST ðiÞ > d SS ðiÞÞ AA TS ¼ 1 2 ðAA truth þ AA syn Þ where n is the number of samples in each dataset (real and generated), d TS (i) is the distance between the real (truth-T) sample indexed by i and its nearest neighbour in the generated (synthetic-S) dataset, d TT (i) is the distance between a real sample i and its nearest neighbour in the real dataset, d ST (i) is the distance between a generated sample i and its nearest neighbour in the real dataset, d SS (i) is the distance between a generated sample i and its nearest neighbour in the generated dataset, and 1 is the indicator function which returns 1 if the argument is true and 0 otherwise.
Based on this equation, an AA TS value below 0. 5 indicates overfitting and an AA TS value above 0. 5 indicates underfitting. We additionally obtained a privacy score in another analysis where we separated the datasets into two equal-sized train and test sets (2504 phased haplotypes each). The score is defined as follows: Privacy score ¼ Test AA TS À Train AA TS where Test AA TS (resp. Train AA TS ) is the AA TS computed with the test (resp. training) samples as truth. The expected value for the privacy score is 0 when there is no privacy leakage, with higher scores indicating higher leaks.
Summary statistics For the 65,535-SNP dataset, LD was computed only on a subset of pairs in order to fasten computation. To sample these pairs in an efficient way (i. e. approximately uniformly along the SNP distance log scale without computing the full matrix of SNP distances), we used the script from  [38]. The remaining summary statistics (allele frequencies, haplotypic pairwise distances, PC scores, 3-point correlations and LD for the 10,000-SNP dataset) were computed as in  [21]  using the publicly available scripts at https://gitlab. inria. fr/ml_genetics/public/artificial_ genomes. For the radar plots, we transformed the scores so that they span values between 0 and 1, where 0 represents poor performance and 1 high or perfect performance. Precisely, we used the allele frequency correlation for alleles with low frequency (<= 0.
In addition, the new WGAN and RBM models preserved 3-point correlations better than the other models (S3 Fig). To assess whether AGs could preserve short motifs in real genomes, we also analyzed the correlation of non-overlapping 4-mer and 8-mer motifs between real and generated datasets (S4 Fig). AGs generated by all models showed high correlation and good fit overall, although RBM_prev had the highest variance despite the high correlation coefficient. None of the models have produced identical sequences and no full sequences were copied from the training dataset except for the benchmark VAE. In addition, the distribution of haplotypic pairwise differences between real and generated datasets overlapped well with the distribution of haplotypic pairwise differences within the real dataset (S5 Fig).
Membership inference attacks and privacy leakage Assessing privacy leakage and performing membership inference attacks require an additional test set but we could not obtain good quality AGs using a subset of 65,535-SNP dataset. Therefore, we trained the new WGAN and RBM models using half of the samples from the 10,000-SNP dataset (2504 haplotypes). AGs generated via the WGAN model had similar summary statistics to the AGs generated via the model trained with the whole dataset. For the RBM model, results were slightly worse but better than the previous RBM model (RBM_prev) Hypothetically, this might occur when the generator generates new instances based on average information from a small collection of samples, causing low local variation. d) An extreme case of a specific type of generative aberration in which the nearest neighbours of both real and synthetic data points are real (AA syn << 0.
5 and AA truth >> 0. 5). This might possibly be observed when the generative model learns the main modes in real data but fails to mimic the densities and generates instances in the axes of the main modes with high variance. https://doi. org/10. 1371/journal. pcbi. 1011584. g004 trained with the whole dataset (S14A-S14C Fig). AA TS analysis showed underfitting for WGAN AGs (AA TS = 0. 80, AA truth = 0. 76, AA syn = 0. 83) and almost perfect scores for RBM AGs (AA TS = 0. 50, AA truth = 0. 49, AA syn = 0. 51). Based on the test set, WGAN AGs had good privacy score (0. 03) and RBM AGs showed possible privacy leakage (0. 23) (S14D Fig). Furthermore, both white-box attack on WGAN AGs and black-box attack on RBM AGs reached high accuracies (WGAN: 1%: 0. 82, 10%: 0. 826, 25%: 0. 754, 50%: 0. 677; RBM: 1%: 0.
Discussion In this study, we implemented generative neural networks for large-scale genomic data generation and assessed various characteristics of the artificial genomes (AGs) generated by these networks. Initially, we generated AGs from the 10,000-SNP dataset to be able to compare to the previous results from  [21]. For this dataset, we introduced a new RBM training scheme, a convolutional WGAN and a convolutional VAE. Both WGAN and new RBM models substantially improved the quality of AGs in terms of all summary statistics. For the 65,535-SNP dataset, we used a similar WGAN architecture (which is essentially deeper in comparison to the architecture for the 10,000-SNP dataset) and a conditional RBM (CRBM) protocol. AGs generated by these new models preserve population structure, allele frequency distribution and haplotypic integrity of real genomes reasonably well with little or no privacy leakage from the training data.
Generative models trained with similar-sized genomic data have been reported in the literature but the main goal of these studies was characterization of population structure via dimensionality reduction and the generated genomes did not possess good haplotypic integrity  [27, 40]. There have been other studies focusing on demographic parameter estimation  [41]  and data generation  [8] [9] [10] [11]  for population genetics but these only included training with smaller genomic segments. The upscaling to a larger sequence size in comparison to our previous study is an essential step for AGs to be utilised in real-life applications as publicly accessible alternatives for sensitive genomic samples, yet obstacles remain. Although we could generate substantially larger sequences by incorporating convolutions for the WGAN and a conditional approach for the RBM, training these large models requires computational time and fine-tuning in most cases.
To avoid loops, a sample was removed once reached in the chain. Expected frequency for chains of size 2 is 0. 25, size 3 is 0. 125, size 4 is 0. 0625 and size 5 is 0. 03125. when experimenting with reduced training sample sizes. For future applications, automated architecture and hyperparameter tuning methods can help significantly for finding the optimal combinations without extensive trial and error  [42, 43]. VAEs might be seen as better alternatives in terms of training stability compared to GANs since they do not suffer from the difficulty of balancing two networks in an adversarial manner, yet we could not obtain highquality AGs for the 65,535-SNP dataset. Presumably, considerably better outcomes can be achieved with further architectural, hyperparameter and loss function exploration, which is outside the scope of this study.
Another future research direction for the generative models could be the exploration of alternative stopping criteria for the training. Unlike image generation, where the generated outcomes can be assessed easily by visual inspection, assessment of generated genomic data is not trivial. For VAE and WGAN models, we decided to use PCA plots for initial inspection and as the stopping criterion since the highest variation in most genomic datasets is due to population structure which PCA captures well. This PCA match, inspected visually or measured through Wassertein distance, had been used previously as the stopping criterion  [8, 21]. Although an obvious candidate for the stopping criterion of WGAN is the convergence of the critic's loss to a value close to zero (since this loss provides an estimation for the Earth mover's distance between real and generated data), we observed that it does not always coincide with good PCA outcomes for the AGs.
SS The other approach we presented to incorporate large-scale data was the conditional training of RBMs. Instead of a single training of a large and deep model as in the case of WGAN, CRBM training includes multiple training runs of a small model. A main advantage of this method is that any sequence size can theoretically be generated as long as the positional conditionality is not broken over the target genome segment. Therefore, the only bottleneck in terms of computation is the time needed to train all the RBMs. Currently, the learned weights are specific to each machine and no parameters are shared between RBMs. This advantageously allows parallel training when needed, but produces a large number of parameters to be optimized since the number of weights and biases has to be multiplied by the number of machines.
Further studies could investigate whether parameter sharing between machines (i. e. , between regions) and/or within each machine (through convolutions) is advantageous or instead complicates training. In addition, many datasets also present a difficulty during training, because the equilibrating time for Markov chains diverges as the training goes on, causing issues for sampling (and re-sampling)  [44]. To overcome this, we relied on out-of-equilibrium training as described in Materials and Methods. While this approach improved training stability and also the quality of the generated samples with respect to other approaches, it has the disadvantage that the learned features cannot be easily interpreted. For the assessment of overfitting and underfitting, we utilised the AA TS score and haplotypic pairwise distances as in  [21].
An interesting finding was the large difference between two terms of the AA TS score (AA truth and AA syn ) for AGs generated by the CRBM even though the averaged AA TS score was good. Interestingly, this is the opposite of what we had observed for our previously published GAN model and possibly points to the type of generative aberration demonstrated in Fig 4D. In fact, it is known that the likelihood function used to infer the parameters of RBMs tends to create potentially spurious modes -that do not match any region of the dataset-, which might explain these patterns. For further investigation, we performed a nearest neighbour chain analysis since we would expect the nearest neighbours of the nearest neighbours for CRBM AGs to be mostly real genomes with this anomaly (Table  1 ). High frequency of ST, STT, STTT, STTTT and low frequency of SS, SSS, SSSS, SSSSS measurements provide additional evidence that what we observe for the CRBM AGs is possibly similar to the scenario seen in Fig 4D.
To the best of our knowledge, this type of anomaly for generative models has not been reported in literature and could have been missed by classical evaluation metrics. Such complex overfitting/underfitting phenomena require vigilance and further investigation in future generative studies. We furthermore performed membership inference attacks on AGs generated by WGAN and RBM models to assess possible privacy leakage. While the white-box attack on WGAN AGs and black-box attack on RBM AGs produced high accuracy for detecting a portion of the samples used in training, black-box attack on WGAN AGs was mainly not successful (S15 Fig). This indicates that even if the model architecture is available publicly and the adversary is in possession of some samples from the training dataset, it is not trivial to pinpoint the training samples via this attack without access to the model weights.
This work was funded by the  Agence Nationale de la Recherche  through grant  ANR-20-CE45-0010-01 RoDAPoG  (B. Y. , L. B. , A. S. , C. F. , G. C. , F. J. ); the  Comunidad de Madrid  and the  Complutense University of Madrid (Spain)  through the  Atraccion de Talento programs  (Refs. 2019-T1/ TIC-13298 ), the  Banco Santander  and the  UCM  (grant  PR44/21-29937 ), and the  Ministerio de Economı ´a y Competitividad, Agencia Estatal de Investigacion and Fondo Europeo de Desarrollo Regional (FEDER) (Spain and European Union)  through grant  PID2021-125506NA-I00  (A. D. );  Labex DigiCosme  (project  ANR-11-LABEX-0045-DIGICOSME ) operated by  ANR  as part of the program " Investissement d'Avenir"   Idex Paris-Saclay  ( ANR-11-IDEX-0003-02 ) (L. B. ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Following the main motivation of the study, we trained the new WGAN and CRBM models on 1000 Genomes data with 65,535 SNPs. The WGAN model for this data was deeper compared to the model used for the 10,000-SNP data (see Materials and methods). We again implemented a VAE similar to the WGAN architecture yet we could not train this model with satisfactory results (see Discussion). Both WGAN and CRBM generated AGs were able to capture the real population structure and PCA modes quite well (Fig 5A) along with allele frequencies (correlation coefficient for low frequency alleles being 0. 97 for both models; Fig 5B), yet WGAN AGs had substantially more fixed alleles which had low frequency in the real dataset (S8 Fig). Since computation of the full correlation matrix is very intensive due to large sequence size, we calculated an approximation of the LD decay based on a subset of SNP pairs (see Materials and methods).
Metabolite levels shape cellular physiology and disease susceptibility, yet the general principles governing metabolome evolution are largely unknown. Here, we introduce a measure of conservation of individual metabolite levels among related species. By analyzing multispecies tissue metabolome datasets in phylogenetically diverse mammals and fruit flies, we show that conservation varies extensively across metabolites. Three major functional properties, metabolite abundance, essentiality, and association with human diseases predict conservation, highlighting a striking parallel between the evolutionary forces driving metabolome and protein sequence conservation. Metabolic network simulations recapitulated these general patterns and revealed that abundant metabolites are highly conserved due to their strong coupling to key metabolic fluxes in the network.
Specifically, a previous study quantified the relative levels of ~150 metabolites in four major organs of 26 mammalian species, spanning an evolutionary period of ~200 My  (7). Our study revealed that the extent of conservation of individual metabolite levels is largely uniform across mammalian lineages, but varies extensively across metabolites. Such variation in conservation results from the differing amounts of functional constraints across metabolites, and is primarily determined by metabolite abundance (i. e. , absolute concentration), as evidenced by an independent dataset on absolute metabolite levels in mice  (17). Systems modeling revealed that highly abundant metabolites are more closely correlated with key fluxes in the metabolic network, explaining their strong conservation. Remarkably, evolutionary conservation is predictive of the diseaseinvolvements of metabolites, confirming that high conservation implies large fitness impacts in human.
Extensive Variation in Evolutionary Conservation of Metabolite Levels. To systematically investigate the evolutionary conservation of metabolite levels, we first analyzed a published dataset of mammalian metabolomes within a phylogenetic framework. The dataset contains the relative levels of 139 nonlipid metabolites in four organs (brain, kidney, heart, and liver) across 26 mammalian species, representing nine taxonomical orders (7) (Fig. 1A ). The dataset covers a wide range of central metabolic pathways, including amino acid, carbohydrate, energy, and cofactor metabolism, and represents the most quantitative and phylogenetically diverse multispecies comparison of metabolomes to date. Therefore, it is particularly well-suited to interrogate the general principles of metabolome evolution. The magnitude of metabolite-level fold-change differences among the 26 species varies widely across metabolites in all studied organs (SI Appendix, Fig.
S1 ). For example, in liver, cytidine level varies up to 529-fold among species, while malate shows highly similar levels in all species, with less than 2. 3-fold differences (Fig. 1A ). This pattern suggests extensive variation in the evolutionary conservation of metabolite levels among different metabolites. To more rigorously estimate the degree of evolutionary conservation, while also accounting for species phylogeny, we introduce a score that captures the extent of conservation of metabolite levels along the phylogeny for each metabolite and for each organ (Methods, Dataset S1A). This metabolite conservation score (MCS) is based on the widely used Brownian motion (BM) model of evolution, which provides a simple and informative measure of evolutionary rate for quantitative traits  (18).