dataesr/scientific-paragraphs-categorization huggingface dataset:
To test the validity of this principle in a different expression chassis and construct library, we repeated the analysis in Fig. 5  using a recent genotype-phenotype screen of promoter sequences in Saccharomyces cerevisiae  25. These data are comparable to the screen in Cambray et al. 23  in the sequence length (80nt) and its highly clustered coverage of genotypic space (Fig. 6A ). This clustered structure results from the design of the library itself, which is composed of 3929 variants of 199 natural promoters. A key difference between this new dataset and Cambray et al. 23  is the construct architecture; unlike the UTR sequences in Fig. 1B , promoter sequences account for regulatory effects but do not undergo transcription. Akin to our results in Fig. 5 , we aimed at testing the accuracy of machine learning regressors trained on datasets of constant size but increasing sequence diversity.
Since this dataset contains a small number of variants for each gene (on average 20 variants/gene, see inset of Fig. 6A ), we first randomly aggregated the variant clusters into twelve groups containing an average of 327 sequences/group. We subsequently trained five Random Forest models on N = 400 binary one-hot encoded sequences drawn from different groups. For example, as shown in the Fig. 6B , model 1 was trained on 200 sequences from two groups, whereas model 2 was trained on 100 variants from four groups. The training results (Fig. 6B ) show a strikingly similar pattern to those observed in our original dataset in Fig. 5 , thus strongly suggesting that sequence diversity can be exploited to train models with broader coverage and improved data efficiency.
Discussion Progress in high-throughput methods has led to large improvements in the size and coverage of genotype-phenotype screens, fuelling an increased interest in deep learning algorithms for phenotypic prediction  9, [12] [13] [14] 16, 17, 19, 34. Synthetic b ology offers a host of applications that would benefit from such predictors, e. g. for optimization of protein-producing strains  39  , selection of enzymatic genes in metabolic engineering  40  , or the design of biosensors  41. An often-overlooked limitation, however, is that deep learning models require huge amounts of data for training, and the sheer cost of the associated experimental work is a significant barrier for most laboratories. Recent sequence-to-expression models have focused primarily on datasets with tens to hundreds of thousands of training sequences (Supplementary Table  S1 ).
While large data requirements are to be expected for prediction from long sequences such as entire protein coding regions, synthetic biologists often work with much shorter sequences to control protein expression levels (e. g. promoters  3  , ribosomal binding sequences  4  , terminators  42  and others). From a machine learning standpoint, shorter sequences offer potential for training models with smaller datasets, which can lower the entry barriers for practitioners to adopt deep learning for strain optimization. Here, we examined a large panel of machine learning models, with particular emphasis on the relation between prediction accuracy and data efficiency. We used data from an experimental screen in which sequence features were manipulated using a Design of Experiments approach to perturb the translation efficiency of an sfGFP reporter in E.
coli  23. Thousands of local mutations were derived from more than fifty sequence seeds, yielding mutational series that enable deep focal coverage in distinct areas of the sequence space (Fig. 1B ). By suitable sampling of these data, we studied the impact of the size and diversity of training sequences on the quality of the resulting machine learning models. Our analysis revealed two key results that can help incentivize the adoption of machine and deep learning in strain engineering. First, in our dataset we found that the number of training sequences required for accurate prediction is much smaller than what has been shown in the literature so far  8, 12, 16, 17, 25. Traditional non-deep models can achieve good accuracy with as few as 1000-2000 sequences for training (Fig. 2B ). We moreover showed that deep learning models can further improve accuracy with the same amount of data.
For example, our convolutional neural networks achieved gains of up to 10% in median prediction scores across all mutational series when trained on the same 2000 sequences as the non-deep models (Fig. 3C ). Such performance improvement is a conservative lower bound, because we employed a fixed network architecture for all mutational series; further gains in accuracy can be obtained with custom architectures for different mutational series. Second, we found that sequence diversity can be exploited to increase data efficiency and enlarge the sequence space where models produce reliable predictions. Using two different datasets with a similar structure of their sequence coverage, the E. coli library from Cambray et al. 23  as well as a recently published library of S. cerevisiae promoters  25  , we showed that machine learning models can expand their predictions to entirely new regions of the sequence space by training on a few additional samples from that region (Figs.
5, 6 ). This means that controlled sequence diversity can improve the coverage of sequence-toexpression models without the need for more training data. In other words, instead of utilizing fully randomized libraries for training  8, [16] [17] [18]  , it may be beneficial to first design few isolated variants for coverage, and then increase the depth with many local variants in the vicinity of each seed. Our work strongly suggests that such balance between coverage and depth can be advantageous in small data scenarios, where fully randomized libraries would lead to datasets with faraway and isolated sequences that inherently require large datasets to achieve high accuracy. This principle is conceptually akin to the idea of "informed training sets" introduced by Wittmann and colleagues  43  in the context of protein design, which can provide important benefits in cases where data efficiency is a concern.
Our observations raise exciting prospects for Design of Experiments strategies for training predictors of protein expression that are both accurate and data-efficient. Data requirements above 1000 sequences are still too costly for most practical applications. Further work is thus required on DNA encodings that are maximally informative for training, as well as model architectures that can deliver high accuracy for small datasets. Both strategies have proven highly successful in protein engineering  44, 45  , yet their potential for DNA sequence design remains largely untapped. We found that seemingly superficial changes to DNA encodings, e. g. from binary one-hot to ordinal one-hot encodings (Fig. 2B ), can have substantial impact on predictive performance. Moreover, although biophysical properties such as the CAI or the stability of mRNA secondary structures are not good predictors by themselves  17  , we observed small but encouraging improvements when these were employed in conjunction with one-hot encodings, particularly for small datasets.
Data processing Data sources and visualization. The E. coli dataset presented by Cambray et al. 23  was obtained from the OpenScience Framework  47. After removing sequences with missing values for sfGFP fluorescence and growth rate, the dataset contains ~228,000 sequences. In all trained models, we employed the arithmetic mean of sfGFP fluorescence across replicates for the case of normal translational initiation  23. To visualize sequences in a two dimensional space (Fig. 1B ), we employed the UMAP algorithm 27 v0. 5. 1 on sequences featurized on counts of overlapping k-mers. We found that the UMAP projection improved for larger k, and chose k = 4 to achieve a good trade-off between computation time and quality of projection (Supplementary Fig. S1 ); k-mer counting was done with custom Python scripts.
In all cases, fluorescence measurements were normalized to the maximum sfGFP fluorescence across cells transformed with the same construct averaged over 4 experimental replicates of the whole library  23. Training, validation, and test data. In Supplementary Fig. S3A  we illustrate our strategy to partition the full dataset into sets for training, cross-validation and model testing. For each mutational series, we first perform a split retaining 10% of sequences as a fixed held-out set for model testing. We use the remaining sequences as a development set and perform a second split to obtain two partitions for each series. The first partition is for model training and comprises 3200 sequences from which we used varying fractions for training regressors in each series. The second partition was employed for hyperparameter optimization, containing ~400 sequences from each series (10% of the whole series) that we then merged into a large validation set comprising 22,400 sequences (56 series × 400 sequences per series) from all series.
Model training Non-deep machine learning models. DNA encodings (Table  1 ) were implemented with custom Python code, and all non-deep models were trained using the scikit-learn Python package. To determine model hyperparameters, we used a validation set for all combinations of encodings and regressors. As illustrated in Supplementary Fig. S3B , for each model we explored each the hyperparameter search space (Supplementary Table  S3 ) for all encodings using grid search with 10fold cross-validation on 90% of our validation set (~20,000 sequences), using mean squared error (MSE) as performance metric. This resulted in six hyperparameter configurations for each regressor (one for each encoding). For many regressors, we found that the same configuration was optimal for several encodings simultaneously, and we thus settled on most frequent configuration among the six encodings; in case of a tie between configurations, we settled for the one with the best MSE computed on the remaining 10% of our whole validation set.
Convolutional neural networks. CNNs were trained on Tesla K80 GPUs from Google Colaboratory. To design the CNN architectures, we use the Sequential class of the Keras package with the TensorFlow backend  48, 49. All CNNs were trained on binary one-hot encoded sequences with mean squared error as loss function, batch size of 64, learning rate 1 × 10 -3 , and using the Adam optimizer  50. Since Adam computes adaptive learning rates for each weight of the neural network, we found that the default options were adequate and did not specify a learning rate schedule. We set the maximum number of epochs to 100, and used 15 epochs without loss improvement over the validation set as early stopping criterion to prevent overfitting. Model hyperparameters were selected with Bayesian optimization implemented in the HyperOpt package  35.
Impact of sequence diversity Escherichia coli dataset. The models in Fig. 5  were trained on data of constant size and increasing sequence diversity. We successively aggregated fractions of mutational series to create new training sets with improved diversity. We employed the same CNN architecture and training strategy as in Fig. 3A  with the same hyperparameters (Supplementary Table  S6 ) for all 27 models. To ensure a comparison solely on the basis of diversity, we fixed the size of the training set to 5800 sequences. To increase diversity, for successive models we sampled training sequences from two additional series, as shown in Fig. 5. The specific series for the aggregates were randomly chosen; four training repeats with randomized selection of series can be found in Supplementary Figure  S12.
Saccharomyces cerevisiae dataset. We obtained the promoter dataset presented in Supplementary Fig. 4F  in Vaishnav et al. 25  from CodeOcean 52. The data contains 3929 yeast promoter sequences with YFP fluorescence readouts. To visualize the yeast sequences (Fig. 6A ), we employed the same strategy as in Fig. 1B  for the E. coli dataset, and used the UMAP algorithm for counts of overlapping 4-mers. Additional details can be found in the Supplementary Text. For the models in Fig. 6B , we first aggregated sequences from the clusters in Fig. 6A  into twelve groups. We then employed the same strategy as in Fig. 5 , and successively aggregated fractions of groups to create new training sets with improved diversity. We used the same Random Forest configuration (Supplementary Table  S6 ) for all 5 models.
2 =0. 25 ridge regressor 25% R 2 =0. 41 support vector regressor 25% R 2 =0. 69 ridge multilayer support random regressor perceptron vector regressor forest (MLP) (SVR) (RF) B 2 ) ≤0 0 0. 2 0. 4 0. 6 0. 8 1 0. 51 0. 67 0. 70 0. 79 binary one-hot uniform 0. 41 0. 40 0. 54 0. 54 0. 62 0. 63 0. 76 0. 77 4-mer ordinal ordinal one-hot 0. 62 0. 70 0. 74 0. 75 4-mer counts near-gaussian 0. 41 0. 62 0. 45 0. 71 0. 54 0. 73 0. 71 0. 74 biophysical properties mixed bimodal C ridge ) % 100 50 MLP SVR RF ( e right-skewed left-skewed c n e c s e r o u l f P F G s d e r u s a e m 100 50 100 0 50 0 R random forest 5% 10% 25% 50% 75% 25% 0 0 50 100 predicted sGFP fluorescence (%) Fig. 2 | Accuracy of non-deep machine learning models. A We trained models using datasets of variable size and with different strategies for DNA encoding.
We analysed resting state functional magnetic resonance imaging (rs-fMRI) data of twins. To disentangle strength and topography, we first performed cortical registration based on anatomical features, and then identified variability of topography by subsequent functional registration. We found that the cortical landscapes of genetic influence on variability of these two features diverge along an axis from primary to higher order association areas. In primary areas variability of connectivity strength exhibits high heritability compared to topography, while this relationship is reversed in parts belonging to association areas. Distinguishing the role of individual experience in shaping variation of connectivity strength from the role of heritability in shaping its topographical variability may provide crucial insights into the mechanisms enabling the flexible, yet stable nature of brain organization supporting the human cognitive repertoire.
Dataset and preprocessing We use the data of 231 participants labeled either as monozygotic or dizygotic twin taken from the HCP S1200 ICA-FIX denoised dataset. Only twins with 4 rsfMRI runs available are chosen. Although not intended on purpose, there were then only same-sex dizygotic twin pairs included in the study. Further details on exclusion criteria and data acquisition are described elsewhere  ( Van Essen et al. , 2012 , 2013 ; Dubois et al. , 2018 ; U ğurbil et al. , 2013 ). 112 subjects out of 231 are labeled as monozygotic twin. The mean age in this group is 30. 02 years and 74% are female. The group of monozygotic twins consists of 2 subjects denoted as "Asian/Nat. Hawaiian/Other Pacific ", 12 denoted as "Black or African Am ", 1 as "Unknown or Not Reported and 97 as "White ". The other 119 subjects labeled as dizygotic are 62% female an the mean age in this group is 29.
82 years. 4 are denoted as "Asian/Nat. Hawaiian/Other Pacific ", 21 as "Black or African Am ", 1 as "Unknown or Not Reported and 93 as "White ". Participants provided verbal consent in accordance with guidelines set by the Wu-Minn HCP Consortium. The preprocessing consists of the HCP-pipelines  ( Glasser et al. , 2013 ) , which include processing of the volume data and bringing the participant's surface into a standard space (fs_LR). For the ICA-FIX dataset independent component analysis is used to decompose the data set into "good " and "bad " components based on the volume data. Bad components are then removed from the surface data  ( Griffanti et al. , 2014 ; Salimi-Khorshidi et al. , 2014 ). Global signal regression and bandpass filtering (range 0. 01 Hz-0. 08 Hz) are applied in addition to the HCP-pipelines.
Embedding coordinates Ψ i of vertex i are derived from the right eigenvectors of L multiplied with  k /(1- k ), where  k denotes the eigenvalue of the k th eigenvector  k , as  1 (i) is the i th entry of the first eigenvector. Ψ 𝑖 = ( 𝜓 1 ( 𝑖 ) , 𝜓 2 ( 𝑖 ) , … , 𝜓 𝑁 ( 𝑖 ) ) 𝑇 Fig. In order to reach functional correspondence across participants the embedding of each twin Ψ S = ( 𝜓 1 S , 𝜓 2 S , …, 𝜓 N S ) is aligned to the em- bedding Ψ R = ( 𝜓 1 R , 𝜓 2 R , …, 𝜓 N R ) of a reference participant (ID: 101915) not part of the twin dataset by calculating a rotation matrix Q S,R , that has the form   ,  =    , with V and U constructed via singular value decomposition, 𝑈 Σ𝑉 𝑇 = ( Ψ 𝑅 ) 𝑇 Ψ 𝑆. A publically available implementation is used to obtain and align diffusion maps (mapalign: https://github.
(2015)  who used diffusion maps for multi-subject functional registration and found that using the first 3 components gave the best results. Box plots summarizing the distribution of eigenvalues across subjects for the first 7 components are shown in Supplementary Fig. 2. Deformation fields for discovery and robustness analysis are shown in Supplementary Fig. 3. After alignment a vertex of the twin is assigned to each vertex of the reference participant, using nearest neighbor to reach functional correspondence of vertices across all individuals in the twin dataset. W can be constructed in different ways from the signal correlation data. To investigate the stability of the gradients dependent on the construction of W , we also calculated gradients by (1) thresholding W at 0 and (2) taking the absolute values W instead of shifting the range.
The trait covariance of monozygotic Twin 1 with Twin 2 is A + C, whereas for dizygotic twin pairs it is 0. 5 A + C , since they share on average only half of their genes, whereas the genes of monozygotic twins are identical. For parameter estimation the expected trait population covariance matrix V p , as well as the expected covariances of Twin 1 with Twin 2 (mono-and dizygotic) are then equated with the corresponding values calculated from the dataset. The twin model is implemented using R's package OpenMx version 2. 11. 5  ( Neale et al. , 2016 ; Hunter, 2018 ).
5 ). Additionally, the genetic contributions to variability of connectivity strength before and after functional alignment and to spatial layout are correlated with each other using Pearson's correlation coefficient. Furthermore, the genetic contributions to variability of the spatial layout are correlated with variability in spatial layout, distance to primary area and cortex expansion. Uncorrected and corrected p -values are provided for the correlation coefficients and the t -test, whereas the uncorrected p -values for correlation coefficients are calculated with a spin test using 10,000 spins in Matlab  ( Alexander-Bloch et al. , 2018 ). The level of significance is set to 0. 05 and obtained p -values are adjusted through Holm-Bonferronicorrection. Holm-Bonferroni correction is done for each hemisphere separately and covers the t -test and correlation of heritability of SP with variability in spatial layout, distance to primary area and cortex expansion as well as correlations between genetic contributions to variability of connectivity strength before and after functional alignment and to spatial layout with each other.
Within each bin normalized activation probabilities are summed given one of 12 components related to function (i. e. language)  ( Yeo et al. , 2016 ). The 12 components (rows) are ordered based on the difference of activation probabilities between A FC and A PS summed over bins and weighted by the log of percentile. The sequence visualizes the gradient from high A SP and low A FC on top, towards low A SP and high A FC on the bottom. The 12 activation probability maps and the maps for A FC and A SP are in the same space, therefore activation probabilities of voxels lieing in one A SP /A FC -bin can be summed. Additionally we performed robustness analysis using 2 RL fMRI runs, which have also been available for each participant additionally to the 2 LR runs used for the discovery analysis. In addition to performing the same steps as described in the first paragraph of this section, we also correlated maps based on the discovery dataset with maps obtained based on the RL runs.
Acknowledgements We thank  Oscar Miranda-Dominguez  for sharing his version of the Gordon parcellation with us. This work was supported by the  Medical University of Vienna , the  Austrian Research Fund (FWF)  [grants P 35189, P 34198, and I  3925-B27 ] in collaboration with the  French National Research Agency (ANR) , the  Vienna Science and Technology Fund (WWTF)  [ LS20-065 ], the  European Research Council  Grant [866533-CORTIGRAD], the  National Natural Science Foundation of China  [Grant No. 81790652 , No. 81790650 ] and the  NAM Advanced Biomedical Imaging Program  [ FY2016 ] between  Nanyang Technological University, Singapore  and  Medical University of Vienna , Austria.
Data and code availability Third party code used in this study included Connectome Workbench v1. 3 26 ( https://github. com/Washington-University/workbench/ releases/tag/v1. 3. 2 ) and Freesurfer v5. 3 25 ( https://surfer. nmr. mgh. harvard. edu/fswiki/DownloadAndInstall5. 3 ) for preprocessing and MSM v2 [30][31][32] ( https://www. doc. ic. ac. uk/ ∼ecr05/MSM_HOCR_v2/ ) and mapalign ( https://github. com/satra/mapalign ) for functional alignment. Own code was developed in R 3. 4 for setting up the twin model using R's package OpenMx version 2. 11. 5  ( Bürger, 2000 ; Neale et al. , 2016 ; Hunter, 2018 )  ( https://openmx. ssri. psu. edu/ ), as well as in Python 3. 6 for calculation of connectivity matrices. Matlab 2018 was used for generation of figures and the spin test ( https://www. github.
After washings, secondary antibodies were incubated 45 min at RT and nuclear staining was performed using ProLong TM Glass Antifade Mountant with NucBlue TM (Thermo Fischer Scientific, #P36981). Negative controls were performed by omitting primary antibodies. Images were visualized using High Content Screening Yokogawa CQ1 microscope (Yokogawa, Tokyo, Japan), digitalized using sCMOS camera (Olympus, Hamburg, Germany) and analysed using ImageJ software (version 2. 1. 0/1. 53c). 2. 6. Total RNA/DNA Isolation and Real-Time Quantitative PCR Total RNA and DNA were isolated using Quick-DNA/RNA TM Miniprep kit (Zymo research, Mülhauser, Germany) according to manufacturer's instruction. First, mRNA were reverse-transcribed into cDNA using PrimeScript TM RT reagent kit (Takara Bio Europe, #RR037A) and analysed in real-time qPCR using SYBR ® Premix ExTaqII (Takara Bio Europe, # RR820A) on an AriaMx Realtime PCR system (Agilent Genomics, Santa Clara, CA, USA).
These apparently conflicting data may reflect non-canonical miRNA-target interactions, independently of the seed complementary sequence  [17] , in the proximity of the both sites 1126-1132 and 2651-2657. Finally, we confirmed the functional targeting of BNIP3L by miR-30a-5p by Western blot analysis of the protein content. Indeed, after transient transfection of miR-30a-5p in primary keratinocytes, we found that the protein levels of BNIP3L were decreased by about 40% (Figure  1E ,F), which corresponds to the level of mRNA decrease in the same conditions. Thus, miR-30a-5p is likely regulating the protein level of BNIP3L through mRNA degradation.
Acknowledgments: We acknowledge the contribution of  SFR Biosciences  ( UAR3444/CNRS ,  US8/Inserm ,  ENS de Lyon, UCBL)  facilities: ANIRA  IMMOS  and especially  Laurence Canaple  for the metabolic phenotyping, PLATIM and especially  Elodie Chatre  for the microscopy, for their precious help. We thank our valuable trainees Jules Granet and  Emma Fraillon  for their contribution in the setting of the protocols. We also thank  Damien Roussel  ( CNRS, UMR 5023-EHNA, Lyon ) for helpful discussions regarding the metabolic activity and  Corinne Leprince  ( INSERM, U1056-UDEAR, Toulouse ) for helpful discussions regarding the keratinocyte terminal differentiation. Alejandro Gonzalez-Torres is grateful for a postdoctoral fellowship from SECTEI.