dataesr/scientific-paragraphs-categorization huggingface dataset:
Lexical features We first introduce some lexical measures on words. • Word frequency: the frequency at which words appear in our dataset, known to be relevant for both recognition and recall  (Gregg, 1976)  • Age of acquisition: the average age at which words are learned (obtained from Kuperman, Stadthagen-Gonzalez, & Brysbaert, 2012), known to have different effects than word frequency  (Dewhurst, Hitch, & Barry, 1998; Morrison & Ellis, 1995)  • Phonological and orthographic neighborhood density (obtained from Marian,   Bartolotti, Chabal & Shook, 2012), also known to be relevant for word production  (Garlock et al. , 2001)  • The average number of phonemes and number of syllables for all pronuncia- tions of a word (obtained from the Carnegie Mellon University Pronouncing Dictionary, Weide, 1998), 7 as well as number of letters, as a proxy to word production cost • The average number of synonyms for all meanings of a word (obtained from WordNet, 2010) as an a priori indicator of how easy it would be to replace a word.
Substitution model We finally need a substitution detection model, for the quotation data we use presents a challenge: Quote-to-quote transformations and substitutions are not explicitly encoded in the dataset. More precisely, each set of quotations bears no explicit information about either the authoritative original quotation or the source quotation(s) each author relied on when creating a new post and reproducing (and possibly altering) that source. In other words, we face an inference problem where, given all quotations and their occurrence timestamps, we must estimate which was the originating quotation for each instance of each quotation. We therefore model the underlying quotation selection process by making a few additional assumptions. Given a particular occurrence of a quotation, the first issue is deciding whether that occurrence is a strict copy of an earlier occurrence, or a substitution from another quotation, or maybe a substitution or copy from quotes appearing outside the dataset, that is, from a source external to the data collection perimeter.
We also introduce two null hypotheses, H 0 and H 00 , to compare the actual variation of a word's feature to expected variations under unbiased transformations. H 0 models the situation where the arrival word w 0 is randomly chosen from the whole pool of words available in the dataset for that feature. 19 In this case, since /ðw 0 Þ becomes a constant value in the above averaging (by definition w 0 does not depend on w anymore), the baseline variation under H 0 may be rewritten as: m 0 / ðf Þ ¼ h/i H 00 models the situation where the arrival word w 0 is chosen among immediate synonyms of the start word w, that is, an arrival word chosen among semantically plausible yet still random words. In this case, m 00 / does depend on f: 20 m 00 / ðf Þ ¼ D /ðw 0 Þi w 0 2synðwÞ E wj/ðwÞ¼f f g This approach yields a fine-grained view of how word features evolve upon substitution, on average, with respect to (a) the original feature (vs.
little to no interaction between a disappearing word's features in determining the variations that that word will undergo when substituted. To make things concrete, here is an example substitution taking place in the dataset. Around mid-November 2008 several media websites reported the following quote from Burmese poet Saw Wai (arrested for one of his poems), Senior general Than Shwe is foolish with power. and a smaller number of media websites, and blogs, reported the following, Senior general Than Shwe is crazy with power. The word foolish is acquired at an average of 8. 94 years old, appears 675 times in dataset, has a Clustering coefficient of 8:2 Â 10 À3 and is seven letters long. The word it was replaced with, crazy, is acquired on average at 5. 22 years old, appears about 4. 1k times in the dataset, has a Clustering coefficient of 1:7 Â 10 À3 , and is five letters long.
Discussion We initially aimed to connect the field of cultural evolution with psycholinguistics by asking if cultural attractors appear in a corpus of online news-related quotes gradually transformed by low-level biases. The dataset we used imposed a few constraints on our analysis: First, it was necessary to infer source-destination links, an operation made more reliable when restricting the scope of transformations to very simple cases, which we did by focusing on single word substitutions. Second, contrary to laboratory experiments which produce data made of many repeated measures on a small number of cases (e. g. , a given list of words), we have a great number of different cases (one case per cluster in Fig. 9. Substitution susceptibility for in-quote feature quartiles: Susceptibility to substitution versus quartile of the feature distribution in the originating quote, with 95% asymptotic confidence intervals  (Goodman-based multinomial).
Software colophon Finally, this paper was developed using Python's scientific computing ecosystem  (Millman & Aivazis, 2011). In particular, we directly used NumPy and SciPy  (Walt, Colbert, & Varoquaux, 2011) , Matplotlib  (Hunter, 2007) , Pandas  (McKinney, 2010 ), scikit-learn (Pedregosa et al. , 2011) , NetworkX  (Hagberg, Schult, & Swart, 2008) , NLTK  (Bird et al. , 2009) , IPython  (P erez & Granger, 2007) , and many other libraries from the Python ecosystem. The software and analyses written for the paper are documented and published under a Free Software license. They can be found at https://github. com/wehlutyk/brainscopypaste. Notes 1. Attractors need not be points in fact; they can also be subareas. In that case, any transformation brings representations in the area closer to (or confined to) the target subarea.
2. As explained later on, source-destination links between quotes must be inferred from the dataset, an operation which is much more reliable if we restrict our analysis to substitutions. This also impedes us from observing the effect of accumulated transformations in the long term, limiting our results to a view of the individual evolutionary step. 3. Aside from our lack of control on the precise conditions of encoding and recall in our dataset, the analysis techniques mentioned above are better suited to data consisting of a high number of measures over a smaller number of lists (in which case it makes sense to ask, for example, what proportion of intrusions come from prior lists). As is explained further down, however, our dataset is shaped the opposite way: a great number of sentences, with only very few to no measures at all on each sentence.
11. Agglomerating into less than six groups merged groups (a) and (b), which we excluded to keep neighborhood densities in their own group; agglomerating into more than six groups separated age of acquisition from group (c), which we excluded given its high correlation values to the rest of group (c). We used scikitlearn's FeatureAgglomeration class for this procedure  (Pedregosa, Varoquaux, & Gramfort, 2011). 12. Note that feature values stem from different data sets which do not always encode the same words. Indeed, we have data on frequency for about 33. 5k words, on age of acquisition for 30. 1k words, on clustering coefficient for 5. 7k words, number of synonyms for 111. 2k words, and orthographic density for 17. 8k words. Quite often, then, not all features are available for a given word in our dataset; however, this is not problematic since the analysis is done on a per-feature basis, and not all words need to be encoded in all features.
18. Similarly to what we do for susceptibility, we avoid possible autocorrelation effects by averaging start-and arrival-word features of substitutions from the same cluster into a single aggregate substitution per cluster. 19. For instance, when considering the feature "Clustering coefficient," the arrival word is randomly chosen among words present in the dataset of FA norms. 20. The actual implementation has an additional level of averaging since WordNet, used to get word synonyms, defines several meanings for a single given word, which we have no means of disambiguating. Therefore: m 00 / ðf Þ ¼ D h/ðw 0 Þi w 0 2synðmÞ E m2meaningsðwÞ E wj/ðwÞ¼f f g 21. This reasoning is standard in the analysis of dynamical systems (where the same transformation is applied to the whole system over and over) and becomes obvious when one manually simulates a substitution on the graph by picking a start value, using the m / curve to obtain the corresponding arrival value, and comparing it to the start value: The arrival value is always closer to the intersection with y = x, meaning that that intersection is an attractor point for the substitution process.
Acknowledgments We are warmly grateful to  Ana Sofia Morais  for her precious feedback and advice on this research, and to  Jean-Pierre Nadal ,  Sharon Peperkamp , and  Nicolas Baumard  for insightful suggestions and comments. We additionally wish to thank three anonymous reviewers for their thorough evaluations, and  Telmo Menezes ,  Nicolas Claidi  ere, and  Jean-Philippe Cointet  for useful final observations. This work has also been partially supported by the  French National Agency of Research (ANR)  through the grant  Algopol  ( ANR-12-CORD-0018 ).
To date, few lncRNA studies have been reported for livestock species, apart from lncRNA studies in bovine  [16]  and trout  [17] , and the construction of multi-species databases such as NONCODE  [18, 19]  and the domestic-animal lncRNA database (ALDB)  [20, 21]. Research programs are in progress on several farm species, e. g. , in projects conducted within the framework of the Functional Annotation of Animal Genomes initiative  [14, 15]. Different methodologies have been described to discover and model lncRNAs. This generates some variability in the number of putative lncRNAs reported and stresses the importance of precisely defining the tools and thresholds for each analysis step. Regarding lncRNA modeling, the FEELnc program (FlExible Extraction of Long noncoding RNAs), developed by Wucher et al.
[22, 23] , distinguishes lncRNAs from mRNAs based on a machine-learning method that estimates a protein-coding score according to different criteria such as the RNA size, ORF coverage and multi k-mer usage. One main advantage of the FEELnc program is its ability to derive an automatically computed cut-off that maximizes the lncRNA prediction sensitivity and specificity. In addition, and contrary to other tools such as CPC  [24]  or CPAT  [25] , FEELnc provides a lncRNA classification based on their genomic position with respect to a pre-defined set of reference genes (usually protein-coding genes), which allows to distinguish intergenic from intragenic lncR-NAs and to sub-classify them according to their orientation with respect to a reference set of genes. Such a classification can be useful to formulate hypotheses about co-expression patterns observed between lncRNAs and their closest protein-coding genes.
65 billion per tissue) to capture weakly expressed lncRNAs and across a large number of biological replicates (16 birds per tissue) to obtain sufficient statistical power to assess correlations of expression levels between lncRNAs and their closest protein-coding RNAs. In coordination with the FAANG initiative (FAANG Bioinformatics and Data Analysis subcommittee), we used a pipeline based on STAR, Cufflinks and FEELnc to describe and characterize a catalogue of expressed putative lncRNAs. We used two protein-coding score cutoffs (including a stringent one for lncRNAs) to partition our transcript set into lncRNAs, protein-coding RNAs and ambiguous RNAs (i. e. , with intermediate proteincoding scores). We found approximately 2193 lncRNA genes (2979 transcripts), from which we extracted a reliable subset of 1670 genes (2412 transcripts) that were characterized by reproducible expression across the 16 replicates.
Chicken lncRNAs predicted by FEELnc and their structure and expression features For the liver and adipose tissue samples (16 replicates per tissue), we obtained on average 100 million stranded, paired-end reads. We compared the efficiencies of the recently published Stringtie and the classical Cufflinks programs to predict transcripts from our sequencing data, providing the Ensembl annotation as a guide and starting from the same BAM files generated by STAR. The Cufflinks/Cuffmerge pipeline processed our dataset of 32 samples in approximately 79 h and generated 39,504 transcripts for 22,413 genes. Stringtie took less than 3 h but produced approximately 4 times more predictions (150,659 transcripts for 108,098 genes), which included a majority of mono-exonic models (68 vs. 11% for Cufflinks).
The number and the structure of the transcript models found with Stringtie in our data were considerably larger than expected based on data from the literature  [6]. Thus, for this study, we used the more realistic models from Cufflinks/Cuffmerge. Finally, the STAR/Cufflinks/ Cuffmerge pipeline applied to our 32 samples resulted in a more than two-fold increase in number of transcripts compared to that reported in the Ensembl V84. 4 annotation on the reference GalGal4 genome, with 39,504 transcripts for 22,413 genes compared to the 17,954 transcripts for 15,508 genes in the Ensembl annotation. To date, no lncRNA has been annotated in the V84. 4 Ensembl chicken gene dataset. These 39,504 newly modeled transcripts were then submitted to the "FlExible Extraction of Long noncoding RNAs" (FEELnc) pipeline to identify putative lncRNAs (see the "Methods" section).
Fixing a specificity cut-off at 0. 97 and using the NONCODEV5 database as the noncoding transcript training set (see the "Methods" section), we identified 2979 putative lncRNA transcripts (for 2193 genes), 376 new mRNAs (for 279 genes), and 179 ambiguous RNAs (Fig. 1a ). When the training set of intergenic regions was used as the noncoding transcript training set (see the "Methods" section), we found 2588 lncRNA transcripts, with most of them (2539 lncRNAs) being common to the two final sets. Such a result shows the usefulness of FEELnc to predict lncRNAs in a species for which no lncRNA repertoire is available for training. We then compared our lncRNA set with the chicken lncRNAs available in the NONCODE and ALDB multispecies databases. We found that 14 and 25% of our chicken set was in common with the chicken NONCODE and ALDB datasets, respectively, using stringent criteria and 16 and 27% using more relaxed criteria (see "Methods" section).
Note that the ALDB dataset shares 25% of the chicken lncR-NAs with NONCODEV5 under the relaxed criteria. Such results highlight that lncRNA annotation strongly relies on the bioinformatics pipelines used for the gene modeling and lncRNA prediction but also on the RNA-Seq samples used in terms of sequencing depth, tissue analyzed and probably physiological status of the animals. To evaluate the relevance of our chicken lncRNA set, we analyzed the gene expression profiles of the three classes "putative lncRNA transcripts", "new mRNAs" and "ambiguous RNAs" and also compared the structural features of our lncRNAs with those of the mouse and human lncRNAs. As expected, the 2193 putative lncRNA genes are on average tenfold less expressed than the known or new protein-coding genes, and the ambiguous RNAs have an intermediate expression (Fig.
1b ). This is in accordance with previous findings in mammals that showed that lncRNAs are far less expressed than protein-coding genes  [6, [29] [30] [31]. Then, we characterized the structural features of these chicken putative lncRNA transcripts in comparison to the human and mouse lncRNAs available in Ensembl and compared them with the protein-coding RNAs available in Ensembl for these three species. Overall, the features observed for the chicken lncRNAs are consistent with those observed in mammals in the human and mouse ENCODE projects  [6]  (Fig. 1c ). First, regardless of the species analyzed, lncRNAs are spliced but with fewer exons than the protein-coding RNAs, with medians of 3 and at least 5, respectively. Second, the median exon length is similar for lncRNAs and protein-coding RNAs in chickens (127 ± 1 nt).
To obtain a more reliable set of expressed lncRNAs, we took advantage of the large number of replicates to remove genes with low signals. Rau et al. [32]  developed an R package (HTSfilter) for RNA-Seq data analysis to correctly filter out lowly-expressed genes and thereby increase the power of detection in the context of the differential expression of protein-coding genes. Unfortunately, this data-driven method (based on the Jaccard similarity index to calculate a filtering threshold) is not appropriate for lncRNAs because of their low expression level (see Additional file 1: Fig. S1 ). Therefore, we analyzed the reproducibility of the expression level across the 16 replicates of each tissue using the standard 0. 1 FPKM-UQ threshold (see the "Methods" section). Figure  1d  provides the numbers of long noncoding and protein-coding genes expressed according to the number For the chicken lncRNAs, the data were generated in this study, while for the human and mouse lncRNAs, the data are taken from Ensembl V83.
d Number of genes considered as expressed (FPKM-UQ ≥ 0. 1) (y-axis) according to the number of biological replicates (x-axis) in the liver (left) and adipose tissue (right) for lncRNAs and Ensembl protein-coding genes. On each plot are indicated the number of genes for which at least 10 samples have a FPKM-UQ ≥ 0. 1 (right number) and the number of genes for which a maximum of four samples have a FPKM-UQ ≥ 0. 1 (left number). e Classification by FEELnc of the 1670 reliable lncRNA genes for 2412 transcripts of biological replicates for each tissue. Long noncoding genes show quite good reproducibility of expression across samples, with 1249 of them having an FPKM-UQ higher than 0. 1 in at least 10 of the 16 samples in the liver, i. e. , 64% of all hepatic lncRNA genes with one read in one sample (Fig.
Using the FEELnc classifier module, we then analyzed the class distribution of the 1670 reliable FEELnc lncRNA genes compared to annotated protein-coding genes from Ensembl (Fig. 1e ). We found 1493 intergenic lncRNA genes (89%), which was the largest class as reported in humans by Derrien et al. [6] , compared to 177 intragenic lncRNA genes (11%). These 1670 lncRNA genes, which are characterized by a good reproducibility of expression level in at least one of the two tissues and corresponding to 2412 transcripts, were analyzed more deeply and are reported in Additional file 2: Table  S1.
Pre-processing steps on RNA-Seq data Three billion reads from the RNA sequencing were mapped onto the chicken Galgal4 reference genome using STAR (v2. 4. 0i)  [68] , and the PCR duplicates were removed for each RNA-Seq sample using the SAMtools rmdup tool (v0. 1. 19)   [69]. All the data were merged into one bam file with the merge tool (v1. 1) from the Samtools suite to create the input file used to model transcripts and genes. Gene modeling was performed with both Stringtie (v1. 0. 1)  [70]  and Cufflinks (v2. 2. 1)  [71] , using the Ensembl gene annotation file (release 82) as a reference. To compare the results, tests were conducted under the same conditions with 12 cores. The CPU was an Intel(R) Xeon(R) CPU E5-2670 v2 @ 2. 50 GHz. The counting step was performed by featureCounts (v1. 4.
Long noncoding RNA prediction lncRNA annotation was performed by the FEELnc program (FlExible Extraction of Long noncoding RNAs, v. 23/11/2015  [22, 23]. Briefly, FEELnc is an alignmentfree software that uses multi k-mer frequency data and relaxed open reading frame (ORF) annotation as the main computational features/predictors to discriminate protein-coding from non-coding RNAs. These features are then used in a machine-learning algorithm (random forest) to compute a coding potential score (CPS) that will discriminate between mRNAs and lncRNAs. In particular, the program can be self-trained with speciesspecific annotations and it automatically defines the coding potential threshold that maximizes the classification performance (i. e. , where the sensitivity equals the specificity). Once the FEELnc model is trained with the above predictors, it is then applied on a set of novel transcript models (e.
g. , from Cufflinks or Stringtie) reconstructed after transcriptome sequencing to predict their proteincoding capacity. The description of the FEELnc program is accessible at bioarxiv  [23]  in which extensive benchmarking of the program in comparison with six other programs is presented based on the GENCODE human and mouse gold-standard datasets. Basically, FEELnc has three modules: "FEELnc_filter", "FEELnc_codpot" and "FEELnc_classifier". Using the first module "FEELnc_filter", we filtered out all transcripts for which exons overlapped in the sense protein-coding exons or pseudogenes that are referenced in the chicken V78 Ensembl annotation. Note that the V78 Ensembl annotation is equivalent to the last V84. 4 annotation for the chicken, with 15,508 coding genes and 17,954 coding transcripts.
We also filtered out transcripts that were shorter than 200 bp according to the commonly accepted definition of long noncoding RNAs. The second module "FEELnc_codpot" separates putative long noncoding RNAs (lncRNAs) from protein-coding RNAs by first computing a coding potential core (CPS, ranging from 0 to 1) for each transcript and then computing a CPS cut-off that maximizes both the lncRNA sensitivity and specificity using a tenfold cross-validation according to the input training files. For the training set of protein-coding transcripts, we used the 15,508 known coding transcripts annotated by Ensembl. For the training set of long noncoding transcripts, we used both the 13,085 chicken putative transcripts from the NONCODEV5 database (v. 2016)  [18, 19]  and a set of 11,000 genomic intergenic regions automatically extracted by FEELnc.
Note that the lncRNA predictions of NONCODE are mainly based on the analysis of the Cufflinks gene models by the coding-non-coding index (CNCI) method  [35]. Here, the CPS calculation is based on ORF coverage, mRNA size and multi k-mer frequencies; for this latter criterion, we chose frequencies of 1-, 2-, 3-, 6-, 9-and 12-mers, and the optimal performance in terms of specificity for our training data was 0. 96. FEELnc allows the user to increase the performance metrics to obtain high-confidence predictions of lncRNAs/ mRNAs, although this option leads to the creation of an intermediate category of ambiguous coding/noncoding transcripts (TUCp). The third module "FEELnc_classifier" classifies each lncRNA with respect to its location and orientation compared to its closest annotated protein-coding genes.
The two main classes are (1) the genic lncRNA class, corresponding to lncRNA transcripts that overlap a protein-coding gene, and (2) the intergenic lncRNA class, with three subtypes that are the divergent, convergent and same-strand sub-classes, as detailed on the FEELnc website  [22]  and schematized in Fig. 1e. Each command line and input/output file used to run the different analyses are available in Additional file 6.
Comparison of our lncRNA set with the chicken lncRNAs from the NONCODE and ALDB databases The multi-species NONCODE  [18, 19]  and ALDB  [20, 21]  databases contain 9343 and 6132 chicken lncRNAs, respectively, that are either intergenic or overlap a gene in antisense orientation. The exon coordinates of our chicken lncRNA set were compared to those of both databases using the "bedtools intersect" tool v. 2. 25. 0  [73]. Two thresholds were used i. e. 100% (stringent criteria) and 50% (relaxed criteria), which refer to the percentage of the lncRNA exon lengths in our dataset that match those of the analyzed database set. Because of the non-perfect modeling of lncRNAs, we considered that a lncRNA was present in two sets if at least one exon was shared by these sets.