dataesr/scientific-paragraphs-categorization huggingface dataset:
Since activations of units within RNNs depend not only on the incoming stimulus, but also on the "current" state of the network as influenced by past stimuli, they are capable of holding short-term events into memory. Adding the RNN module can thus be viewed as augmenting the encoding models with working memory. Investigating timescales of representations across brain regions by understanding the influence of contextual representations on language processing in the brain, as captured by LSTM language models for instance, has become a major research focus recently  (42). In these language encoding models for fMRI, past context has been shown to be beneficial in neural response prediction, surpassing word embedding models. However, models that explain neural responses under dynamic natural vision while exploiting the rich temporal context have not yet been rigorously explored with human fMRI datasets.
Concluding remarks Comprehensive descriptive models of the brain need comprehensive accounts of the stimulus. Using a novel group-level encoding framework, we showed that 'reliable' cortical responses to naturalistic stimuli can be accurately predicted across large areas of the cortex using multisensory information over longer time-scales. Since our models were trained on a large-scale, multi-subject and open-source dataset, we believe these results could provide an important point of reference against which encoding models for naturalistic stimuli can be assayed in the future. The continued interplay of artificial neural networks and neuroscience can pave the way for several exciting discoveries, bringing us one step closer to understanding the neural code of perception under realistic conditions.
Regularized linear regression: WordNet features Another popular approach in voxel-wise forward encoding beyond primary sensory cortices is the semantic category encoding model that is based on high-level semantic features  (38). This approach relies on labels that indicate the presence of semantic object and action categories in each movie frame. In this analysis, we employed WordNet labels that were provided as part of the HCP movie-watching data pipeline. The semantic labels were manually assigned by the Gallant lab team using the WordNet semantic taxonomy and subsequently converted to Word-Net synsets to build an 859-D semantic representational space (corresponding to 859 WordNet synset names). Following  (38) , we fitted l 2 regularized linear regression models (known as ridge regression) to find weights corresponding to different input features for every voxel.
Acknowledgments This work was supported by  NIH  grants  R01LM012719  (MS), R01AG053949 (MS), R21NS10463401 (AK),  R01NS10264601A1  (AK), the  NSF   NeuroNex grant   1707312  (MS), the  NSF   CAREER 1748377  grant (MS) and  Anna-Maria and Stephen Kellen Foundation Junior Faculty Fellowship (AK).
Data and Software availability All experiments in this study are based on the Human Connectome Project movie-watching database. The dataset is publicly available for download through the ConnectomeDB software (https://db. humanconnectome. org/). Throughout this study, we utilized 7T fMRI data from the 'Movie Task fMRI 1. 6mm/59k FIX-Denoised' package within HCP. The network implementation, analysis codes as well as trained model weights will be made available on the project Github page.
R NA polymerase II (RNAPII) transcribes many loci outside annotated protein-coding gene promoters  1, 2  to generate a diversity of RNAs, including for instance enhancer RNAs  3  and long noncoding RNAs (lncRNAs)  4. In fact, >70% of all nucleotides are thought to be transcribed at some point  1, 5, 6. Using the Cap Analysis of Gene Expression (CAGE) technology  7, 8  , the FANTOM5 consortium provided one of the most comprehensive maps of TSSs in several species  2. Integrating multiple collections of transcript models with FANTOM CAGE datasets, Hon et al. built a new annotation of the human genome (FANTOM CAGE-Associated Transcriptome, FANTOM CAT), with an atlas of 27,919 human lncRNAs, among them 19,175 potentially functional RNAs  4. Despite this annotation, many CAGE peaks remain unassigned to a specific gene and/or initiate at unconventional regions, outside promoters or enhancers, providing an unprecedented mean to further characterize noncoding transcription within the genome "dark matter"  9  and to decode part of the transcriptional "noise".
Results CAGE peaks are detected at STRs. We first intersected the coordinates of 1,048,124 CAGE peak summits  2  with that of 1,620,030 STRs called by HipSTR  29. We found that 89,948 CAGE peaks (~8. 6%) initiate at 84,555 STRs (Fig. 1a  and Supplementary Fig. 1 ). As a comparison, only 2. 3% of an equal number of randomly selected intervals with equivalent size intersected with CAGE peaks (Fisher's exact test P value < 2. 2e-16). Among CAGE peaks intersecting with STRs, 10,727 correspond to TSSs of FANTOM CAT transcripts  4  and 8823 to enhancer boundaries  3  (Supplementary Data 1). Note that the FANTOM CAT annotation was shown to be more accurate in 5' end transcript definitions compared to other catalogs (GENCODE  34  , Human BodyMap  35  , and miTranscriptome  36  ), because transcript models combine various independent sources (GENCODE release 19, Human BodyMap 2.
0, miTranscriptome, ENCODE and an RNAseq assembly from 70 FANTOM5 samples) and FANTOM CAT TSSs were validated with Roadmap Epigenome DHS and RAM-PAGE datasets  4. This transcription does not correspond to random noise because the fraction of STRs harboring a CAGE peak within each class differs depending on the STR class, without any link with their abundance (Fig. 1a, c ). Some STR classes with low abundance are indeed more often associated with a CAGE peak than more abundant STRs (Fig. 1a, c , compare for instance (CTTTTT) n or (AAAAG) n vs. (AT) n or (ATTT) n ). Likewise, the number of STRs associated with CAGE peaks cannot merely be explained by their length, as several STR classes have similar length distribution but very different fractions of CAGEassociated loci (compare for instance (AT) n and (GT) n in Fig.
Among these 2291 STRs, 904 (39%) are also associated with a CAGE peak. Thus, compared to the reproducibility of MinION sequencing in both libraries (only 2291 STRs in common out of 5472 (42%) or 7812 (29%)), CAGE and CTR-seq sequencing results are overall in agreement. In fact, STR classes associated with CAGE peaks correspond to those associated with CTR-seq reads (Fig. 3  compared to Fig. 1c ). The Spearman correlation ρ between the fractions of STRs associated with CAGE and MinION reads with and without polyA tailing equals 0. 88 and 0. 89 respectively. Besides, 301 out of 904 STRs associated with both CAGE peak and CTR-seq long read correspond to TSSs of FANTOM CAT transcripts and 54 to enhancer boundaries. Overall, CTR-seq confirms CAGE data and the existence of transcription initiating at STRs.
The lower whisker extends from the hinge to the smallest value at most 1. 5 × IQR of the hinge. Data beyond the end of the whiskers are plotted individually. Fig. 2  CAGE tags initiating at STRs are truly 5'-capped. G bias in ENCODE CAGE tags (bam files from nuclear fraction, polyA+) was assessed at FANTOM5 CAGE peaks assigned to genes (positive control) and CAGE peaks initiating at STRs. G bias at pre-microRNA 3' ends was also assessed as a negative control. Five libraries were analyzed corresponding to A549 (replicates 3 and 4), GM12878, HeLa-S3, and K562 cells. The number of intersecting tags in each case is indicated in the bracket. , for each STR associated with CAGE signal (Fig. 4b ). Briefly, this score corresponds to the difference between the CAGE signal on the (+) strand and that on the (-) strand divided by their sum (in HipSTR catalog, STRs are systematically defined on the (+) strand i.
1d. Note that, because we used mean signal across CAGE libraries, our model is cell-type agnostic. This choice was motivated by the observation that the CAGE signal at STRs in each library is very sparse, thereby strongly reducing the prediction accuracy of our model. As input, we used sequences spanning 50 bp around the 3' end of each STR. Model architecture and constructions of the different sets used for learning are detailed in the "Methods" section and in Supplementary Fig. 7. Source code is available at https://gite. lirmm. fr/ibc/deepSTR. The accuracy of our model was computed as Spearman correlation between the predicted and the observed CAGE signals on held-out test data (see "Methods"). The performance of this global model was overall high (Ρ ~0. 72), indicating that transcription initiation at STRs can indeed be predicted by sequence-level features.
ClinVar pathogenic variants are found at STRs with high transcription initiation level. Second, we evaluated the potential implication of transcription initiation at STRs in human diseases and used the ClinVar database, which lists medically important variants  46. We found that STRs harboring ClinVar variants, located in a window encompassing STR ± 50 bp (n = 34,578), are associated with high CAGE signal compared to STRs without variants (n = 3,068,280, Fig. 7a ), indicative of potential biological and clinical relevance for transcription initiation at STRs. Looking at the clinical significance of the variants, as defined in the ClinVar database, we indeed noticed that STRs associated with pathogenic variants exhibit stronger transcription initiation than STRs associated with other variants (Fig.
The findings that (A) n and (T) n represent distinct directional signals for nucleosome removal  54  are very well compatible with differences observed in flanking sequences (Fig. 5b ) and directional transcription (Fig. 4b ), both able to create asymmetry at (A) n and (T) n. Besides, we show that most CAGE tags initiating at STRs remain nuclear (Fig. 4a ). This observation suggests that, similar to other repeatinitiating RNAs  55, 56  , RNAs initiating at STRs could also play roles at the nuclear/chromatin levels, for instance in DNA topology  56, 57. Note that we also calculated the enrichment of STR classes in FANTOM CAT biotypes (Supplementary Data 3). The strongest enrichments correspond to (A) n , (AT) n , and (AAAT) n at enhancers, which are known to be GC-poor sequences compared to promoters for instance  50.
Methods Data and bioinformatic analyses. The bedtools window  69  was used to look for CAGE peaks (coordinates available at http://fantom. gsc. riken. jp/5/datafiles/ phase1. 3/extra/CAGE_peaks/hg19. cage_peak_coord_permissive. bed. gz) at STRs ± 5bp (catalog available at https://github. com/HipSTR-Tool/HipSTR-references/raw/ master/human/hg19. hipstr_reference. bed. gz) as follows: windowBed -w 5 -a hg19. hipstr_reference. bed -b hg19. cage_peak_coord_permissive. bed As a comparison, random intervals were generated using bedtools shuffle  69. shuffleBed -i hg19. hipstr_reference. bed -g hg19. chrom. sizes -excl hg19. hipstr_reference. bed -seed 927442958 > hg19. hipstr_reference. shuffled. bed Similar analyses were performed using mouse STR catalog (available at https:// github. com/HipSTR-Tool/HipSTR-references/blob/master/mouse/mm10.
stats. f_oneway, scipy. stats. mannwhitneyu, scipy. stats. kstest), as indicated. When indicated, P values were corrected for multiple testing using R p. adjust (method="fdr"). Evaluating mismatched G bias at Illumina 5' end CAGE reads. Comparison between Heliscope vs. Illumina CAGE sequencing was performed as in de Rie et al. 38. Briefly, ENCODE CAGE data were downloaded as bam files (using the following url [http://hgdownload. cse. ucsc. edu/goldenpath/hg19/encodeDCC/ wgEncodeRikenCage/] ('*NucleusPap*' files) and converted into bed files using samtools view  71  and UNIX awk: samtools view file. bam | awk '{FS="\t"}BEGIN{OFS="\t"}{if ($2=="0") print $3,$4-1,$4,$10,$13,"+"; else if($2=="16") print $3,$4-1,$4,$10,$13,"-"}' > file. bed The bedtools intersect  69  was further used to identify all CAGE tags mapping a given position.
Nanopore libraries were sequenced by MinION Mk1b with R9. 4 flowcell. Sequence data were generated by MinKNOW 1. 7. 14. Basecalling was processed by ÓAlbacore v2. 1. 0 basecaller software provided by Oxford Nanopore Technologies to generate fastq files from FAST5 files. To prepare clean reads from fastq files, adapter sequence was trimmed by Porechop v0. 2. 3. Data were deposited on DNA Data Bank of Japan Sequencing Read Archive (accession number: DRA010491). The mapping computational pipeline used a prototype of primer-chop available at https://gitlab. com/mcfrith/primer-chop. The precise methods and command lines are provided as Supplementary Methods. Data were first mapped on hg38 reference genome and liftovered to hg19 for analyses. Directionality score. We collected CAGE signal at each STR of the HipSTR catalog (see above).
5e ). Longer sequences were tested without improving the accuracy of the model (Supplementary Fig. 8 ). Note that only 89,189 STRs (out of 1,620,030, ~5. 5%) are longer than 50 bp and, only in these few cases, the sequence located upstream STR 3' end only corresponds to the STR itself. The parameters of the model were determined by brute force algorithms using a grid search approach. This approach makes a complete search over all hyperparameters (number of layers, number of neurons, activation functions, different learning rates, shape of convolutional kernels, number of convolutional filters, …). The grid search algorithm trains and tests all possible models with all combinations of parameters and returns the most accurate model. The model was implemented in PyTorch. The source code of the model, alongside scripts and Jupyter notebooks are available at https://gite.
lirmm. fr/ibc/deepSTR. In order to minimize overfitting, droupout is added to the fully connected layers (probability of droupout = 0. 30). The training pipeline is described in Supplementary Fig. 7 : we separate training, testing, and validation datasets prior to model training, and these sets are stored on disk. This allows us to carry out analyses on held-out data that has never been seen by the models. We stop the training once the loss function calculated on the validation set drops for five consecutive epochs (early stopping). Relatively good performances on mouse datasets (Fig. 6c ) show that the model generalizes well to unknown CAGE data. Our models were optimized to predict CAGE signal and cannot, as such, be applied to other types of data. However, the methodology used here is generic and could be applied to other types of data as long as one can associate a numeric signal to a specific genomic region.
tomutable() function. To keep sequences aligned, we only considered single nucleotide variants (SNVs). CNN models were then used to predict the CAGE signal of the initial and mutated sequences. The change was computed by the difference between the prediction obtained with the mutated sequence and that obtained with the reference sequence. To insert random variations (Fig. 7c, d ), we created a mutation position map, which follows a uniform distribution (each position has an equal probability of receiving a mutation). Then, we took sequences in the database and mutated them one by one at a position taken from the mutation map. All possible mutations at the chosen position have an equal probability of occurrence (Fig. 7d ).
Acknowledgements We thank  Cédric Notredame ,  Anthony Mathelier ,  Oriol Fornes Crespo ,  Philip Richmond ,  Jean-Christophe Andrau ,  Diego Garrido Martin ,  Dimitri D. Pervouchine ,  Roderic Guigo ,  Charles Plessy , and  Chung Hon  for their help in analyzing the data and for insightful suggestions. We also thank  Takahiro Arakawa  for the preparation and provision of cell culture samples. We are indebted to the researchers around the globe who generated experimental data and made them freely available. C. -H. L. is grateful to  Marc Piechaczyk  and  Edouard Bertrand  for their continued support. The work was supported by funding from  CNRS  ( International Associated Laboratory "miREGEN" ),  INSERM-ITMO Cancer project  " LIONS "  BIO2015-04 ,  Plan d'Investissement d'Avenir # ANR-11-BINF-0002   Institut de Biologie Computationnelle (young investigator grant  to C-H.
L. ) and  GEM   Flagship  project funded from  Labex NUMEV  ( ANR-10-LABX-0020 ). M. G. was supported by a  Conventions Industrielles de Formation par la Recherche (CIFRE)  PhD fellowship from SANOFI R&D. FANTOM5 was made possible by the following grants:  Research Grant for RIKEN Omics Science Center  from  MEXT  to Y. H. ; Grant of the Innovative Cell Biology by  Innovative Technology  ( Cell Innovation Program ) from the  MEXT  to Y. H. ;  Research Grant  from  MEXT  to the  RIKEN Center for Life Science Technologies ;  Research Grant  to  RIKEN Preventive Medicine and Diagnosis Innovation Program  from  MEXT  to Y. H. This work was further supported by a  Research Grant  from  MEXT  to the  RIKEN Center for Integrative Medical Sciences.
Data availability The data that support this study are available from the corresponding author upon reasonable request. CAGE peaks coordinates [http://fantom. gsc. riken. jp/5/datafiles/ phase1. 3 /extra/CAGE_peaks/hg19. cage_peak_coord_permissive. bed. gz]; human STR catalog [https://github. com/HipSTR-Tool/HipSTR-references/raw/master/human/hg19. hipstr_reference. bed. gz]; mouse STR catalog [https://github. com/HipSTR-Tool/HipSTRreferences/blob/master/mouse/mm10. hipstr_reference. bed. gz]; CAGE signals at human and mouse STRs, alongside fasta sequence files, are available on our git repository [https://gite. lirmm. fr/ibc/deepSTR]; FANTOM gene annotation [https://fantom. gsc. riken. jp/5/suppl/Hon_et_al_2016/data/assembly/lv3_robust/FANTOM_CAT. lv3_robust. bed. gz]; Coordinates of FANTOM CAT robust transcripts [http://fantom.
osc. txt. gz]; GTEx V7 data [https://storage. googleapis. com/gtex_analysis_v7/single_tissue_eqtl_data/GTEx_Analysis_v7_eQTL. tar. gz]; ClinVar vcf file [ftp://ftp. ncbi. nlm. nih. gov/pub/clinvar/]. CTR-seq data were deposited on DNA Data Bank of Japan Sequencing Read Archive (accession number: DRA010491 ). The mapping computational pipeline used a prototype of primer-chop available at https://gitlab. com/mcfrith/primer-chop. The precise methods and command lines are provided as Supplementary Methods.
Observation data Data from 14 sites with reported coordinates were collected from published literatures for model validation (Table S1 in the Supplement). Since a tropical humid climate is favourable for oil palm growth, most of the in situ measurements are located in Indonesia (six sites) and Malaysia (seven sites) except for one site in Benin (Fig. 1 ). The observation sites have high mean annual precipitation (MAP, 574. 2-3598. 8 mm yr -1 ) and high mean annual temperatures (MATs) between 24. 3 and 28. 8 • C throughout the year, which covers 97. 27 % and 85. 14 % of the range of MAP and MAT, respectively, in the global oil palm plantation area in 2010  (Cheng et al. , 2018)  (Fig. S1  in the Supplement). The MAT, MAP and clay fraction (CF) for the global oil palm plantation area were based on the climate data from the Climatic Research Unit National Centers for Environmental Prediction (CRUNCEP) gridded dataset  (Viovy, 2011)  and the Harmonized World Soil Database  (HWSD v1.
2, Nachtergaele et al. , 2010). The observation sites include six smaller plantations (< 50 ha, Sites 1 and 2 for smallholders and Sites 4, 5, 7 and 12 as research sites; Fig. 1 ) and seven industrial plantations of up to 23 625 ha. Site 12 and Site 14 were covered by very deep peat soil before oil palm cultivation, where the former natural vegetation was peat swamp forest. The natural vegetation at other sites was dominated by tropical rainforest and the clay fraction varied from 0 %-11 % (Fig. S1 ). LAI, gross primary productivity (GPP), net primary productivity (NPP), fruit bunches (yield) and biomass at different ages including young and mature oil palms were collected from these sites for model validation. Annual data of total biomass and yields were available for Site 3 and Site 12. The biomass data at Site 3 were calculated by an allometric equation using the measured diameter at breast height (DBH) and height of the stem  (Corley and Tinker, 2015) , while yield data at Site 12 were obtained from measurements of the harvested fruit bunch every time.
Therefore, our development of oil palm modelling started from ORCHIDEE-MICT v8. 4. 2. Processes related to the carbon cycle in ORCHIDEE include photosynthesis, respiration, carbon allocation, litterfall, plant phenology and decomposition  (Krinner et al. , 2005). We added a new PFT for oil palm starting from the default setting of the closest PFT: TBE trees. The major modification brought was for the carbon allocation, by including a new phytomer organ for oil palm, and a new fruit harvest module for fresh fruit bunch harvesting (Fig. 2 ). The new model called ORCHIDEE-MICT-OP (oil palm) is schematized in Fig. S2.
Photosynthesis parameters The photosynthesis module of ORCHIDEE-MICT is based on an extended version  (Yin and Struik, 2009)  of the Farquhar, von Caemmerer and Berry model  (FvCB model; Farquhar et al. , 1980). Leaf age class is introduced to take into account the fact that the photosynthetic capacity depends on leaf age  (Ishida et al. , 1999). The maximum rate of Rubisco activity (V cmax ) is defined by the prescribed V cmax25 and weighted leaf efficiency (e rel ; unitless: 0-1). The relative leaf efficiency (e rel ) is a function of relative leaf age (A rel ), where A rel is the ratio of the leaf age to the critical leaf age (the same as Age phycrit ), also known as leaf longevity (Fig. 4 , red line). The e rel change with A rel in the default ORCHIDEE-MICT version is shown in Fig. 4  (black dashed line), which increases from a low initial value to 1 (reaching the prescribed optimal V cmax25 ) for a given period and then decreases to a low level for the old leaves.
Productivity and fruit yield The simulated GPP, NPP and fruit yield in comparison with field measurements are shown in Fig. 7. Compared to the default ORCHIDEE-MICT version, NPP can be better reproduced by ORCHIDEE-MICT-OP (solid squares closer to the 1 : 1 line than the open square, Fig. 7a ) with a normalized mean bias error (NMBE, defined as the sum of biases divided by the sum of field values) of 12. 87 % and r 2 of 0. 9 across sites. Among the 14 sites with NPP observations, simulated NPP at Sites 1, 7 and 12 is comparable with observations with an NMBE of only 4. 0 % while simulated results from other sites are relatively higher than observations (NMBE of 28. 8 %). For GPP, there are only three observations availhttps://doi. org/10. 5194/gmd-14-4573-2021 Geosci. Model Dev. , 14, 4573-4592, 2021  For fruit yields, we collected six single-year observations at different sites for oil palm plantations aged from 10-15 years, except for one site where yield data cover ages 4 to 16.
The introduction of the phytomer structure and the sequential developing processes allows the reproduction of variable developmental stages for each phytomer including the initiation, fruit production, harvest and pruning in the model. Besides, the modification of the carbon allocation scheme improves the allocation of the assimilated carbon and partitioning of biomass pools (Fig. 9 ). Oil palm trees have specific physiological characteristics which are different from other tropical forests. The evolution of physiology with age is implemented by a new tree-age-specific parameterization scheme based on the tree age cohort module of ORCHIDEE MICT. Carbon assimilation is accelerated with increasing oil palm age. Carbon allocation to the phytomer shifts more resources to the fruit than the leaf and branches as fruits mature.
Acknowledgements. Wei Li  and  Philippe Ciais  acknowledge support by the  European Research Council  through Synergy Grant  ERC-2013-SyG-610028  " IMBALANCE-P". Financial support. This research has been supported by the  National Key Research and Development Program of China  (grant nos. 2019YFA0606601 ,  2019YFA0606604  and  2017YFA0604401 ). https://doi. org/10. 5194/gmd-14-4573-2021 Geosci. Model Dev. , 14,  4573-4592 , 2021.