dataesr/scientific-paragraphs-categorization huggingface dataset:
The authors chose a very wide breadth of datasets as input and the resulting framework may prove rather generic since transcriptional regulation is largely cell and tissue-specific. A key characteristic of fixed transcriptional module frameworks is that they are intended for reuse, often over long periods of time. One of the major implications is that greater amounts of effort can be dedicated to the annotation and development of ad hoc visualization tools. This is obviously not possible when new module repertoires are constituted each time a new dataset needs to be analyzed. And indeed, considerable annotation and development efforts have been undertaken to support this latest generation of blood transcriptional modules. This permits us in turn to make available an ecosystem of custom tools and resources that were not available for transcriptional frameworks which have been released before.
This includes a compendium of functional profiling reports and reference transcriptional profiles that can be accessed via an interactive web application. In addition, we are releasing another web application for data certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020. ; https://doi. org/10. 1101/525709 doi: bioRxiv preprint browsing and multi-tiered visualization. And it should also be noted that this ecosystem is scalable and will continue to grow over time as more analyses are being conducted, custom scripts developed, and reference dataset profiles added to the annotation scaffold. It is illustrated by ongoing efforts that consist in compiling relevant aggregate-centric information for A28.
This is with the intent of improving over time understanding of the clinical relevance and biological significance of this transcriptional signature (accessible by clicking the information icon on the A28 interactive presentation: https://prezi. com/view/E34MhxE5uKoZLWZ3KXjG/ and video demonstration: https://youtu. be/0j-kcE1tlXA). A bioinformatics analysis toolkit that comprises a collection of custom R scripts will also follow the publication of this report. Notably, earlier attempts were made by our group to develop such a support infrastructure in the past, albeit on a smaller scale. But maintaining such resources over long periods of time proved a challenge and these by now have gone offline through a combination of hardware failures, hacking or discontinued institutional support. Such issues should be addressed here in part through adoption of zero cost infrastructure (e.
g. hosting of Shiny apps, Prezi presentations or YouTube videos is free as long as public access is maintained). Finally, the resource that we are making available here has some limitations which are also worth noting. First, our framework is specifically designed to analyze and interpret human blood transcriptome profiling data. Analyses of other tissues or in the context of other species would require the development and use of separate frameworks. Indeed, we also developed fixed modular frameworks for various mouse tissues  (28)  and in vitro culture systems, including human whole blood  (29)  and human dendritic cells  (30). Second, we generated our transcriptome datasets for module construction using Illumina BeadArrays, a technology that pre-dates RNA sequencing. Our choice to use this technology was based on data availability at our institute, and the decision certified by peer review) is the author/funder.
Such assays are costeffective, robust and scalable. Secondly, fixed transcriptional module repertoires may be leveraged for conducting meta-analysis. It was for instance employed as a basis to perform such an integrative analysis encompassing six independent RSV datasets encompassing 490 unique profiles [a preprint is available:  (32) ]. More recently, this resource has also been used in a different context to identify signatures associated with CTLA4 and combined CTLA4-PD1 blockade in peripheral blood of ovarian cancer patients [P249:  (33) ]. It may also be possible to extend the certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020. ; https://doi. org/10.
DECLARATION OF INTERESTS The authors declare no competing interests Each dataset was independently clustered via k-means clustering. C. Gene co-clustering events were recorded in a table, where the entries indicate the number of datasets in which co-clustering was observed for a given gene pair. D. The co-clustering table served as the input to a weighted co-clustering graph (see also Supplementary Figure  1 ), where the nodes represent genes and the edges represent co-clustering events. E. The largest, most highly weighted sub-networks among a large network (here constituting 15,132 nodes) were identified mathematically and assigned a module ID. The genes constituting this module were removed from the selection pool and the process was repeated, resulting in the selection of 382 modules constituted by 14,168 transcripts.
Changes measured at the "aggregate-level" are represented by spots to the left of the grid next to the denomination for the corresponding aggregate. The colors and intensities of the spots are based on the average across each given row of modules. A module annotation grid is provided where a color key indicates the functional associations attributed to some of the modules on the grid (top right). Positions on the annotation grid occupied by modules for which no consensus annotation was attributed are colored white. Positions on the gird for which no modules have been assigned are colored grey. B-D. FIGURE LEGENDS Fingerprint grid plots for additional reference datasets. A gene list, along with the ontology, pathway, literature term enrichment and transcriptional certified by peer review) is the author/funder.
All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020. ; https://doi. org/10. 1101/525709 doi: bioRxiv preprint 29 profiling data for reference transcriptome datasets (circulating leukocyte populations, hematopoiesis) is provided for each module. Zoom in and out functionalities for close-up examinations of the text and figures embedded in the presentation is possible. Web links providing access to modules within a given aggregate are listed in Table  3. For a demonstration video, please visit: https://youtu. be/8ajESET2mqI. certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020.
; https://doi. org/10. 1101/525709 doi: bioRxiv preprint The columns and rows were arranged by hierarchical clustering. Such heatmaps can be accessed and exported for all 16 datasets and 38 module aggregates using the web application: https://drinchai. shinyapps. io/dc_gen3_module_analysis/# (under the "MODULES x STUDIES" tab). were used to construct the modular repertoires. These networks factor in differences in coexpression across different "states" of the biological system. For the blood transcriptome, these states would be different diseases or physiological phenotypes. Under Scenario A, the genes are co-expressed in all three disease states, so the weight attributed to the edges of the network is three. Under Scenarios B and C, co-clustering only occurs in two or one of the disease states, resulting in weights of 2 and 1, respectively.
Microarray analysis and data preprocessing Gene expression profiles from whole blood samples generated using Illumina HumanHT-12 v3. 0 or Illumina HumanHT-12 v4. 0 expression BeadChips were obtained from 16 groups of patients and controls selected as above. Thus, 16 datasets were used as the input (Table  1 ). The expression data for each dataset were preprocessed and independently clustered. First, the probes were discarded if they were not present (detection P < 0. 01) in at least 10 samples or in at least 10% of the samples, whichever was greater. Then, the sample data for each dataset were normalized using the BeadStudio average normalization algorithm. Once normalized, the signal was floored such that all signals <10 were set to 10. Then, the fold change was calculated relative to the median signal for that probe across all samples.
Module construction algorithm Sets of coordinately regulated genes, or transcriptional modules, were extracted from the patient's whole blood microarray datasets. Each of the preprocessed microarray datasets were clustered in parallel using Euclidean distance and the Hartigan's K-Means clustering algorithm. The 'ideal' number of clusters (k) for each dataset was determined within a range of k=1-100 by means of the jump statistic  (35). Taking the 16 sets of clusters as the input (Table  2 ), a weighted co-cluster graph was constructed  (Chaussabel, Quinn et al. 2008 , Chaussabel and Baldwin 2014). To select modules, an iterative algorithm was used to extract the sets of probes that are most frequently clustered together in the same datasets, proceeding from the most stringent requirements to the least, as previously described  (Chaussabel, Quinn et al.
2008 ). This iteration differed from the previous implementation of this algorithm in that the k value was calculated independently for each dataset cluster and the size of the core sub-networks was smaller (10 probes). The algorithm also differed from previous implementations to ensure that the core sub-networks co-clustered in the same datasets. Further details and an example of the code are included in the supplemental methods (Supplementary File 1). The resulting 382 module set constitutes the third generation of modular blood transcriptome repertoire constructed since the development of the first generation published in 2008 (4), and the second generation published in 2013  (3).
Module annotation Gene Ontology / Pathway enrichment: Module gene lists were investigated using Database for Annotation Visualization and Integrated Discovery (DAVID) version 6. 7  (12). This database uses a modified Fisher exact test to identify specific biological/functional categories that are overrepresented in gene sets in comparison with a reference set. The top matched DAVID annotation cluster (using default settings), the top matched canonical pathway from Kyoto certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. encyclopedia of genes and genomes (KEGG), the top matched pathway from BioCarta, and the top matched Gene Ontology biologic process (GO_BP) and molecular function (GO_MF) terms were identified for each module. Each module was also investigated for significant overlap with two other established blood transcriptome module repertoires  (3, 9).
Gene Set Annotation (GSAn): To further annotate the modules, a new alternative to statistical enrichment analysis tool called GSAn was applied  (13). Statistical enrichment methods may have limitations  (36) (37) (38) , as these methods tend to focus on the subpart of the most studied genes and to provide gene set annotation results that cover a limited number of the well-annotated genes. To address these issues, GSAn offers: (i) an original method that combines semantic similarity measures and data mining approaches to perform a unified and synthetic annotation for a gene set of interest, and (ii) a visualization approach to facilitate the interactive exploration of the gene set annotation results according to the hierarchical structure of Gene Ontology  (13). The tool is available online: https://gsan.
labri. fr/. A page listing analysis results for all 382 generation 3 blood transcriptome modules can be accessed here: https://ayllonbe. github. io/modulesV3/index. html. Pathway enrichment analyses: These analyses were conducted using Ingenuity Pathway Analysis (Qiagen, Valencia, CA). Literature profiling: Literature Lab™ (LitLab; from Acumenta Biotech, Boston, MA) was used to associate genes within a particular module to terms used in PubMed abstracts  (Febbo, Mulligan et al. 2007 ). Association scores reflecting the strength of the associations were used to calculate the "Product Scores". The top three terms that showed the strongest association and certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020.
; https://doi. org/10. 1101/525709 doi: bioRxiv preprint highest "Product Scores" were used to create the functional annotation. A similar approach using LitLab has been previously reported  (Alsina, Israelsson et al. 2014 ). The steps taken to annotate all 382 modules is described here in brief. All statistical analyses were performed using Microsoft Excel (2010) with Visual Basic for Applications (VBA), Linux-based command line in Mac OS, and R statistical software. After completing the analysis, the summary result page was exported to an xls file. Using the UNIX command line, the exported files were converted to csv files with the filename appended in the last column of each row and vertically appended. The "merged" file was used to populate the table that included all the available LitLab terms.
Fingerprint grid plot visualization Modules were arranged on a grid based on their similarities in patterns of activity across the 16 input datasets, each of them corresponding to a different pathological or physiological state. First, the modules were partitioned using k-means clustering, which resulted in the constitution of 38 clusters. Given the possibility of collapsing values of the modules constituting each cluster in a single "aggregate" value, the term "module aggregate" was used to designate each cluster (A1 to A38). Of these 38 k-means clusters, 27 comprised >1 module. The modules were then arranged certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission. The copyright holder for this preprint (which was not this version posted April 28, 2020.
Module Fingerprint analysis and visualization The modular analysis was performed by using 14,168 transcripts. Fold change was computed using gene expression data prior log2 transformation. For group comparisons, paired t-test was performed on log2-transformed data (Fold change (FC) cut off = 1. 5; FDR cut off = 0. 1). For individual-patients analysis, each sample was compared to mean of control samples in each dataset. Cut off comprised a absolute FC > 1. 5 and a difference in gene expression level > 10. A module was considered to be "responsive" when the proportion of differentially expressed transcripts (as defined above) was greater than 15%. Data visualization were performed using "ComplexHeatmap"  (36).
Generation of Circular Plots Circular plots were generated to represent module expression at the gene-level (Figure  7 ). Fold change in case vs control group in each dataset were shown. Probes that passed FDR < 0. 1 and FC > 1. 5 were present in gradated red colors and those with FC < -1. 5 were represented in gradated blue colors. The position of the genes on each of the circular plots is fixed.
ACKNOWLEDGEMENTS The authors would like to thank  Quynh-Anh Nguyen ,  Kimberly O'Brien ,  Dimitry Popov ,  Michael Mason , and  Cate Speake  for technical assistance. We would also like to acknowledge  Insight Editing London  for assistance in editing the manuscript prior to submission. This project has been funded in part with Federal funds from the  National Institutes of Health  under contract number  U01AI082110. Authors affiliated with Sidra Medicine, a member of the  Qatar foundation for Education, Science and Community Development , are fully supported by institutional funding. RJW is supported by the  Wellcome Trust  ( 203135 ,  104803 ),  NIH  ( U01AI115940 ) and the  Francis Crick Institute , which receives funding from the  Wellcome Trust.
Availability of data and material Transcriptome profiling data are deposited in the Gene Expression Omnibus, https://www. ncbi. nlm. nih. gov/geo/, under the accession GSE100150. lib301 lib315 lib224 lib225 lib328 lib335 lib308 lib322 lib355 lib222 lib279 lib341 lib223 lib286 lib348 lib265 lib259 lib272 lib226 lib288 lib295 lib309 lib242 lib248 lib323 lib329 lib302 lib316 lib349 lib232 lib273 lib336 lib237 lib280 lib342 lib253 lib260 lib266 lib227 lib310 lib289 lib296 lib243 lib249 lib324 lib330 lib303 lib317 lib350 lib233 lib274 lib337 lib238 lib281 lib343 lib254 lib261 lib267 lib228 lib290 lib297 lib311 lib244 lib250 lib325 lib331 lib304 lib318 lib351 lib234 lib275 lib338 lib239 lib344 lib282 lib255 lib262 lib268 lib229 lib291 lib298 lib312 lib245 lib251 lib326 lib332 lib305 lib319 lib352 lib235 lib276 lib339 lib240 lib283 lib345 lib256 lib263 lib269 lib230 lib292 lib299 lib313 lib246 lib252 lib327 lib333 lib306 lib320 lib353 lib277 lib236 lib340 lib241 lib284 lib346 lib257 lib264 lib270 lib231 lib293 lib300 lib314 lib247 lib334 lib307 lib321 lib354 lib278 lib294 lib301 lib315 lib224 lib225 lib328 lib335 lib308 lib322 lib355 lib222 lib279 lib341 lib223 lib286 lib348 lib265 lib259 lib272 lib226 lib288 lib295 lib309 lib242 lib248 lib323 lib329 lib302 lib316 lib349 lib232 lib273 lib336 lib237 lib280 lib342 lib253 lib260 lib266 lib227 lib310 lib289 lib296 lib243 lib249 lib324 lib330 lib303 lib317 lib350 lib233 lib274 lib337 lib238 lib281 lib343 lib254 lib261 lib267 lib228 lib290 lib297 lib311 lib244 lib250 lib325 lib331 lib304 lib318 lib351 lib234 lib275 lib338 lib239 lib344 lib282 lib255 lib262 lib268 lib229 lib291 lib298 lib312 lib245 lib251 lib326 lib332 lib305 lib319 lib352 lib235 lib276 lib339 lib240 lib283 lib345 lib256 lib263 lib269 lib230 lib292 lib299 lib313 lib246 lib252 lib327 lib333 lib306 lib320 lib353 lib277 lib236 lib340 lib241 lib284 lib346 lib257 lib264 lib270 lib231 lib293 lib300 lib314 lib247 lib334 lib307 lib321 lib354 lib278 lib285 lib347.
In this paper, we investigate the feasibility of directly representing DNA reads as continuous vectors instead, and replacing some discrete operations by continuous calculus in this embedding. To illustrate this idea, we focus on an important application in metagenomics, where one sequences the DNA present in an environmental sample to characterize the microbes it contains  [21, 4]. An important problem in metagenomics is taxonomic binning, where each of the billions of sequenced reads must be assigned to a species, given a database of genomes characteristic of each species considered  [16]. Standard computational approaches for taxonomic binning try to align each read to a reference sequence database with sequence alignment tools like BLAST  [5]  or short read mapping tools such as BWA  [14]  or BOWTIE  [10].
However, the computational cost of these techniques becomes prohibitive with current large sequence datasets. Alternatively, compositional approaches rephrase the problem as a multiclass classification problem, and employ machine learning methods such as a naive Bayes (NB) classifier  [23, 19]  or a support vector machine (SVM)  [17, 20, 22]  after representing each read by the vector of k-mer 1  counts it contains. Interestingly,  [22]  showed that compositional approaches can be competitive in accuracy with alignment-based methods, while maintaining a computational advantage, by using large-scale machine learning approaches. However, a good accuracy is only achieved with k-mers of length at least k = 12, corresponding to representing each read as a sparse vector in N = 4 k dimensions. This representation raises computational challenges both at training time (  [22]  push VowPal Wabbit to its limit) and at test time (a N × T matrix of weights must be stored to model T species).
In this work, we propose to extend state-of-the-art compositional approaches by embedding the set of DNA reads to R d , with d N. For that purpose, we still extract the k-mer composition of each read, but replace the N -dimensional one-hot encoding of each k-mer by a d-dimensional encoding, optimized to solve the task. This approach is similar to, e. g. , the fastText model for natural language sequences of  [7, 3]  or word2vec  [18] , with a different notion of words to embed, and a direct optimization of the classification error to learn the representation. This can reduce the memory requirements to store the model and accelerate classification time when d < T , since the N × T matrix of weights is replaced by a N × d matrix of embeddings, and a d × T matrix of weights. After presenting in more detail the model and its optimization, we experimentally study the speed/performance trade-off on metagenomics experiments by varying the embedding dimension, and demonstrate the potential of the approach which outperforms state-of-the-art compositional approaches.
Implementation We implemented the model (3) by modifying the fastText open-source library  [7, 8] , which involves a similar model with k-mer embedding for natural language. There are some important differences between DNA reads and standard NLP applications, though: (i) One is that the concept of a "word", space-delimited groups of letters, does not exist in DNA sequence data. Hence we resort to a distributed representation of overlapping k-mers only, and not to words as in word2vec or fastText. (ii) Second is the number of training examples to be seen by the model is very large: up to 5 × 10 9 if we were to achieve full coverage of the large database below, for example. (iii) Third, the vocabulary is different, as it is of known size (4 k ) and is densely represented for relatively small values of k.
For greater values of k than those considered in this paper, k-mers become rare and individual long k-mers can become discriminative. This is in fact used by some other compositional algorithms such as Kraken  [24]. For these reasons we rewrote part of the fastText software to extract overlapping k-mers rather than words. As appropriate for (iii), the k-mer embeddings are stored in a fixed-size table of dimension (4 k , d), each row corresponding to the vector of a different k-mer. For a given k- mer b = b 1. b k ∈ A k , the index of its corresponding row in M is ind(b) = k-1 i=0 4 i h(b i ), where h(A) = 0, h(C) = 1, h(G) = 2 and h(T ) = 3. This allows to quickly find ind(x i+1:i+k ) from ind(i : i + k -1) as explained for example in  [22]. To address (ii) we generate random fragments on the fly directly from the full genomes, rather than reading text samples line by line as in fastText.
Regularization with noise As a form of regularization, we also add random mutations in the training fragments. When reading the fragment from the reference genome, nucleotide by nucleotide, we introduce a chance r of replacing the nucleotide by a random one (equally distributed on {A, C, G, T }). This is akin to the dataset augmentations commonly used in image classification tasks, and in our case promotes the nearby embeddings of k-mers similar in terms of Hamming distance.
Data We test our model on two benchmarks proposed by  [22] : a small one, useful mostly for parameter tuning, and a large one. Both benchmarks involve a training database of genomes organized by species, and a validation set of genomes coming from the same species as the training database, but from different strains. Reads are randomly sampled, with or without noise, from the validation set of genomes, and the goal is to predict, for each read, from which species it comes from. The small database contains 356 complete genomes, belonging to 51 species of bacteria; its validation set is composed of 52 genomes, belonging to the same 51 species. The large database contains 2,961 genomes belonging to 774 species, which is closer to real-life situations. The validation set is composed of 193 genomes, each from a separate species.