dataesr/scientific-paragraphs-categorization huggingface dataset:
Quality improvement of crowdsourced gene expression signatures. To improve the quality of the gene expression signatures derived from thousands of GEO studies, we first checked for batch effects. To achieve this, we obtained the 'scan date' from the raw microarray data files as an indicator of a potential source for batch effects. We then estimated the magnitude of such batch effect using principal variation component analysis  26, 27. We estimate that batch effects on average account for B18. 7% of the variance in the gene expression dataset collections, whereas the perturbation versus control on average accounts for B16. 7% of the variance (Supplementary Fig. 2a ). To correct for these batch effects, we applied the surrogate variable analysis (SVA)  28  algorithm and generated new signatures using both the CD and limma methods to call the DEGs.
Overall, we find that the MSigDB signatures overlap significantly with matched crowd-generated signatures, with only a few exceptions (Supplementary Fig. 3d , Supplementary Table  1 ). The discrepancies were due to a figure from He et al. 30  that only reported genes related to the cell-cycle as opposed to all DEGs; the Sagiv et al. 31  study reported DEGs in both siRNA knockdown and mAb treatment, whereas the DEGs in our database were derived from knockdown versus control only; and the gene sets curated from Soucek et al. 32  by MSigDB do not match the original figure from that paper. However, overall, our analysis shows strong agreement between the matched signatures in both databases. Assessment of signature associations within each collection. We next asked whether signature similarity within and across the three collections can recover prior knowledge and discover novel connections.
4c, d ), while others can be explored using the interactive clustergram or packed circles plot on the CREEDS web portal. In the first cluster that we chose to highlight, imatinib, a small molecule that is known to be a tyrosine kinase inhibitor  36  , has signatures that were generated from multiple cell lines, including K562 leukaemia cell line (GSE1922), chronic myelogenous leukaemia (CML) CD34 þ cells (GSE12211) and three other CML cell lines (KU-812, KCL-22, JURL-MK1) (GSE24493), which cluster together with knockdown signatures of NRAS in melanoma cell lines (GSE12445) (Fig. 4b ). This strongly suggests that NRAS is targeted by imatinib. Although NRAS is currently not considered a direct target of imatinib, a recent study showed that melanoma patients with NRAS mutations are resistant to imatinib therapy  37.
This raises the possibility that the wild-type form of NRAS is at least a key downstream effector of imatinib. In the second cluster that we chose to highlight, multiple myelodysplastic syndrome (MDS) signatures from CD34 þ cells (GSE4619, GSE19429) and ERBB2 overexpression signature from MCF10A cells (GSE14990) cluster together (Fig. 4c ), suggesting that the up-regulation of ERBB2 may have a role in MDS. Indeed, it was shown that ERBB2 amplification is present in 35% of a cohort of MDS patients 38. In the third example, endometrial cancer signatures (GSE17025) are shown to cluster with estradiol signatures derived from MCF7 cells from multiple independent studies (GSE4668, GSE11352, GSE53394), as well as MIR34A overexpression signature from HCT116 cells (GSE7754), PPARG overexpression signature from NIH-3T3 cells (GSE2192), and IGF1 stimulation signature from MCF7 cells (GSE7561) (Fig.
The fourth cluster contains a YY1 knockout (GSE39009) signature produced in mice soleus, and an autosomal muscular dystrophy signature from a mouse model sourced from the diaphragm (GSE3252). This association suggests that YY1 may be disrupted in muscular dystrophy tissues. Literature supports that almost all facioscapulohumeral muscular dystrophy patients carry deletions of repetitive elements (D4Z4) that contain binding sites for YY1  50, 51. All of the aforementioned examples are just a small portion of the signature connections our integrative analysis offers. These examples illustrate how novel associations between diseases, genes and drugs can be discovered through a crowdsourcing project. Identifying drug mimickers. To further demonstrate the utility of the crowdsourced gene expression signatures of drug perturbations, we queried these signatures against the database of drug or other small molecule compound signatures derived from the LINCS L1000 dataset.
We then recorded the ranks of the matched drugs out of 430,000 LINCS L1000 signatures and found that  many crowdsourced drug perturbation signatures are significantly highly ranked (Rank sum P value o4. 8e À 24 ) (Fig. 5a, b , Table  1 ). a b c d NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Argyrin A|GSE8565 Imatinib|GSE24493 BPDE|GSE19510 NR4A2|GSE58475 SOX7|GSE10809 EPZ004777|GSE29828 Imatinib|GSE12211 NFKB1|GSE20667 EPZ004777|GSE29828 KDM4C|GSE41040 KDM4C|GSE41040 CCND1|GSE8866 GSK3A|GSE35200 Argyrin A|GSE8565 Deferasirox|GSE11670 Azacitidine|GSE29077 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 NRAS|GSE12445 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Imatinib|GSE1922 Argyrin A|GSE8565 Imatinib|GSE24493 BPDE|GSE19510 NR4A2|GSE58475 SOX7|GSE10809 EPZ004777|GSE29828 imatinib|GSE12211 NFKB1|GSE20667 EPZ004777|GSE29828 KDM4C|GSE41040 KDM4C|GSE41040 CCND1|GSE8866 GSK3A|GSE35200 Argyrin A|GSE8565 Deferasirox|GSE11670 Deferasirox|GSE11670 Azacitidine|GSE29077 KLF1|GSE46600 BNIP3L|GSE7020 Anemia|GSE4619 Similarly, the results can also be reproduced when querying the drug perturbation signatures against 46,000 signatures from the Connectivity Map dataset 52 (Supplementary Fig.
5 ). We additionally queried the gene perturbation signatures against 109,000 shRNA knockdown and over-expression profiles from the LINCS L1000 data and found similar consistency (Fig. 5c, d ). These results suggest that some drugs induce similar transcriptional changes in small-scale studies, when compared with results from large-scale studies such as LINCS L1000 and the original Connectivity Map. This means that we can identify potential mimickers using the LINCS L1000 dataset for drugs whose signatures are highly similar between the LINCS L1000 dataset and the GEO studies. Interestingly, we found that dexamethasone signatures in the LINCS L1000 dataset were ranked in the top 10 using dexamethasone-induced gene expression signatures from three independent GEO studies: GSE34313, GSE7683 and GSE54608 (Supplementary Table  5 ).
Web portal to visualize and query the signatures database. To provide easier access to the three collections of the gene expression signatures for knowledge reuse and exploration, we developed a web portal (Supplementary Fig. 6 ). This portal visualizes all of the signatures in a packed circles layout in which similar signatures are closer to each other. Furthermore, the portal has interactive heatmaps of hierarchically clustered matrices of all signatures. The web portal is available at: http://amp. pharm. mssm. edu/creeds. The portal also has a search engine that enables users to search by text or by providing lists of up and down DEGs. Since DEGs for the gene expression profiles in the CREEDS database were computed with the CD method, which is not a standard method, we tested whether signatures computed via other methods would produce similar results.
We found that most signatures computed by fold change or limma are ranked similarly (Supplementary Fig. 7 ). However, some signatures were not ranked as expected. The CD is a multivariate method, whereas fold change and limma are univariate; a gene can be identified as significantly differentially expressed by a  univariate method but may not contribute to the joint expression changes of large sets of genes. Finally, to scale up the three collections of signatures, we developed machine learning classifiers that use the manually curated signatures as a training set. The classification task was divided into two parts: (1) classify whether a GEO dataset is likely to contain gene, disease or drug signatures, and (2) label the samples as control and perturbation. The features for the classifiers were extracted from the text associated with the each GEO study in our manually curated collection as well as from all currently available studies on GEO where genome-wide expression was assessed by microarrays to profile human, mouse or rat cells and tissues.
Discussion Gene expression profiling is arguably the most common type of omic data. The resource we developed for this project can be combined with transcriptomics profiling projects such as Genotype-Tissue Expression  53  , the Cancer Genome Atlas  54  , the Cancer Cell-Line Encyclopaedia  55  , and the Library of Integrated Network-based Cellular Signatures (LINCS). Here we show, for example, how combining drug perturbation signatures collected from GEO with the LINCS L1000 data can be used to identify potential novel drug mimickers. The manually extracted and cleaned signatures were proven to be useful as a training set that enabled us to scale up the three collections of signatures using machine learning. However, we are aware that the quality of the automatically generated signatures is not as good as the signatures created by the human annotators.
Participation was strictly voluntary, and was not required for completion of any parts of the course. Participants were not provided with a list of predefined gene expression profiles; instead, they were encouraged to find diverse, yet relevant, gene expression studies from GEO. Briefly, contributors first had to locate relevant GEO studies fitting into one of the three themes, and then select the perturbation and control samples (GSMs) from GEO series (GSE) or GEO datasets (GDS). Only gene expression studies from selected species of mammals (human, mouse and rat) were considered valid. Participants were also asked to submit additional metadata about the cell or tissue type, and gene, disease or drug used in each experiment and associate these with common published identifiers. Standard names of genes, diseases, and drugs were provided as autocomplete options in the submission forms, created from controlled vocabularies: HGNC for genes  59  , disease names from the Disease Ontology 60 and drug names from DrugBank 61.
To incentivize participants, a real-time leaderboard was developed to display the number of submissions from each user, and modest prizes were promised to the top ten contributors (custom T-shirt and headphones). Additionally, co-authorship on the published research resulting from these crowdsourcing tasks was promised to contributors of a minimum of 15 valid entries. Sanitization of the crowdsourced gene expression signatures. Multiple steps of quality control filters were applied to improve the collection of the gene expression signatures extracted by the crowd. We first performed integrity checks using the association between GEO studies (GSE or GDS) and samples within these studies (GSMs) by re-processing all the collected gene expression signatures based on the metadata supplied by the curators.
As a result, B20% of all the submissions were removed from the final collections. Evaluation of batch effects. To obtain batch information from each study, we retrieved the 'scan date' from the raw microarray CEL files and assumed that the experiments were performed on the same dates that were listed within the experimental batch. We then quantified the batch effect using principal variation component analysis  26, 27  , which attributes the variation in the gene expression data to known sources such as batches and experimental conditions. Batch effects were corrected using the surrogate variable analysis (SVA) algorithm  28  implemented in R 62 with default parameters. Construction of expected DEGs from prior knowledge. To generate lists of expected DEGs for the three collections of signatures for benchmarking, we used: (1) the known direct physical interactors of the protein product of a gene from a consolidated protein-protein interaction network we assembled for a previous study 63 ; (2) a consolidated collection of manually-curated disease-gene associations from the DISEASES resource  64  ; and (3) known drug targets from DrugBank v4.
Classifiers from the scikit-learn 71 package were tested including: random forest  72  , extra trees  73  , support vector classifier and the XGBoost implementation of gradient boosting machines  74. Hyperparameters of the classifiers were optimized using grid search. Classifying control versus treatment samples based on text. We formulate the problem of classifying GEO samples as a binary classification problem. This means that we aim to learn from text-derived features whether a sample is part of the control or treatment group. Features were extracted from the following text fields associated with each GEO sample: title, description, characteristics and source name. These text elements were tokenized and converted to binary vectors representing the presence or absence of tokens for each sample.
The classifier we used for solving this problem is a Bagging 75 of 20 multinomial Bernoulli Naı ¨ve Bayesian  69  classifiers after probability calibration with isotonic regression  76. To measure the performance of the classifier, 10-fold cross-validation was applied to calculate area under the ROC curve, area under the precision-recall curve, Matthew's correlation coefficient and F1 score. Development of the CREEDS web portal. A web portal was developed for visualizing and querying the collections of the gene expression signatures. Relationships between all signatures are visualized using the D3. js pack layout and D3. js clustergrammer. Clustergrammer is a visualization tool we developed starting with the open-source code example for the matrix co-occurrence visualization on the D3. js website.
All data and metadata of the signatures are stored in a MongoDB database. The portal uses the Python Flask framework. Signed Jaccard index was implemented to query signatures in which users input up or down gene lists into two separate text boxes. The text signature search option queries the metadata text of all signatures in the database. RESTful application programming interface (API) endpoints were also developed to enable users to programmatically query and search the CREEDS database. Automatic extraction of gene expression signatures from GEO. To automatically extract gene expression signatures from GEO, we first applied the gradient boosting machines classifier (described above) to predict the categories of all GEO studies (n ¼ 31,905) performed in human, mouse or rat using microarrays.
To determine whether a cluster of samples is a control group or a perturbation group, we chose the average probability P40. 7 and Po0. 3 from the Naive Bayesian-based classifier as control group and treatment group, respectively. Next, we enumerated every pair of valid control groups and perturbation groups within each study as metadata for valid predicted gene expression signatures. To properly label the terms associated with each predicted signature, we used the API of BeCAS 78 to tag biological entities from the text associated with each study, as well as the text associated with the samples, including: genes, cell or tissue, disease, and drug or other small molecule chemical; and then recorded these term counts for a final decision of which terms we should use to label each signature. To process the gene expression data of the predicted gene expression signatures, we first used SVA  28  to correct the batch effect as described above, and then applied the CD algorithm  21  to compute differential expression.
Figure5| Distributions of the ranks of matched perturbations between signatures from CREEDS and the LINCS L1000 dataset. The highest ranks (a,c), and all ranks (b,d) of matched drugs (a,b) and matched genes (c,d) are presented. Drug perturbation signatures from CREEDS were queried against B30,000 significant drug perturbation signatures from the LINCS L1000 dataset; whereas gene perturbation signatures from CREEDS were queried against B110,000 gene perturbation signatures from the LINCS L1000 dataset.
Data availability. All extracted and processed signatures with their accession numbers and other metadata are freely available for download from the CREEDS portal at: http://amp. pharm. mssm. edu/creeds. The CREEDS portal also provides the data through API. Users can search the data by submitting their own signatures for analysis. The site also provides two modes of visualization of all signatures. Accession codes for top hits for drug signatures extracted from GEO queried against drug perturbations can be found in Table  1.
, which contains 8 collections of gene sets. The collection C2 has curated gene sets extracted manually from tables and figures within publications. We compared the Single-gene perturbations GEO Identify relevant studies normal states Disease versus Gene expression profiles Single-drug perturbations Extract metadata Use GEO2 Enrichr To compute Visualization Query Download Metadata expression signatures CREEDS BD2K-LINCS-DCIC Crowdsourcing portal <form> </form> CREEDS database Signatures database Sanitize and Combine Cleaned signatures reprocess database signatures Predicted signatures database Natural language processing, machine learning Metadata of predicted signatures Text classification models Extract text GEO Process signatures Figure 1 | Workflow of the crowdsourcing project. Participants identify relevant studies from GEO and then extract gene expression signatures using GEO2Enrichr.
PCR Amplification and Next-Generation Sequencing For 16S rRNA gene amplification, soil DNA was amplified by PCR using the primers UnivBactF 9 (5 ′ -GAGTTTGATYMTGGCTC-3 ′ ) and BSR534/18 (5 ′ -ATTACCGCGGCTGCTGGC-3 ′ ), which amplified the V1-V2 hypervariable regions, as described in  Yergeau et al. (2012). Plant tissue DNA was amplified by PCR, targeting the 16S rRNA gene using the primers 520F (5 ′ -AGCAGCCGCGGTAAT-3 ′ ) and 799R2 (5 ′ -CAGGGTATCTAATCCTGTT-3 ′ ) targeting the V2-V3 hypervariable regions and excluding the chloroplast 16S rRNA gene, as described in  Edwards et al. (2007). To account for use of different primers and for downstream comparison of samples, a set of samples which were amplified with both primer sets were compared (see supplementary material). A co-inertia analysis was performed, comparing both datasets generated from different primers, and revealed a strong correlation between these datasets and in similar bacterial communities.
Hence, we determined that downstream analyses comparing datasets generated with these different primer sets were valid. For the ITS region, soil and plant tissue DNA were amplified using the ITS1F (5 ′ -CTTGGTCATTTAGAGGAAGTAA-3 ′ ) and ITS2 (5 ′ -GCTGCGTTCTTCATCGATGC-3 ′ ) primer set as in  Ghannoum et al. (2010). To distinguish between samples, unique multiplex identifier (MID) tags from the extended MID set recommended by  Roche (2009)  were integrated in the primers, as described in  Sanschagrin and Yergeau (2014). PCR reactions were performed in 20 µL volumes containing 12. 5 µL HotStart Taq Plus Master mix (Qiagen, Germantown, MD, USA), 0. 5-2 µL of template DNA, 0. 4 µL of bovine serum albumin from 20 mg/mL stock and 0. 625 µL of each forward and reverse primer from 20. 0 µM stocks. For the 16S rRNA primers, cycling conditions were as follows: 5 min at 95 • C, 25 cycles of 30 s at 95 • C, 30 s at 55 • C, and 45 s at 72 • C, and a final elongation step of 10 min at 72 • C.
Amplification of 520F /799R2 amplicons used identical cycling conditions but for 30 cycles and with a 58 • C annealing temperature. The amplification of a plant tissue sample could only be achieved when raising the number of cycles to 35. To evaluate the biases that this might cause, another sample that was previously amplified with 30 cycles was also amplified with 35 cycles and sequenced. The ordination coordinates and the community composition of the two datasets were highly similar (see supplementary material), suggesting that cycle number would have a negligible influence in downstream analyses. For the ITS primers, cycling conditions were as follows: 5 min at 95 • C, 30 cycles of 60 s at 94 • C, 60 s at 45 • C, and 60 s at 72 • C, and a final elongation step of 10 min at 72 • C. Amplicons were verified on 2% agarose gels and then gel purified using the PureLink R Quick Gel Extraction Kit (Invitrogen, Life Technologies, Carlsbad, CA, USA) and quantified using the Quant-iT PicoGreen dsDNA assay kit (Invitrogen, Life Technologies).
Sequence Data Treatment Sequences were analyzed using the procedure described in  Tremblay et al. (2015). Since the bacterial primers used for endophytic and soil communities targeted different regions, they were processed separately as the alignments generated during the procedure would have only overlapped over ∼15 bp. Soil and endophyte OTU tables were merged for further taxonomic classification and subsequent downstream analyses. OTU tables were rarefied to 1000 reads for bacterial datasets and 306 reads for ITS fungal datasets.
Statistical Analyses All statistical analyses were performed in R v. 2. 15. 2 (R Foundation for Statistical Computing; available at http://www. R-project. org). The effect of compartment, contamination and cultivar type on the bacterial and fungal community composition was tested. First, the square roots of UniFrac matrices were used for principal coordinate analyses (PCoA), which were carried out using the "pcoa" function of the "ape" package  (Paradis et al. , 2004). This unconstrained analysis was used in order to explore the patterns of bacterial and fungal OTU composition across samples. Following this, a distance-based redundancy analyses (dbRDA;  Legendre and Anderson, 1999a)  with the "rda" function and the "anova. cca" of the "vegan" package  (Oksanen et al. , 2008)  was carried out in order to test the significance of these descriptors by multivariate statistical hypothesis testing.
This statistical analysis was used to test the significance of the individual descriptors (plant compartment, contamination level, and cultivar type) on multispecies response variables (bacterial/fungal communities) in a multifactorial analysisof-variance model as described in  Legendre and Anderson (1999b). A partial dbRDA was used for testing the effect of compartment in order to control for the identity of the trees. The reported adjusted redundancy statistics R 2 were obtained using the "RsquareAdj" function of the "vegan" package. The relative abundance of specific classified OTUs was represented by samples in a heatmap using the "pheatmap" function in the "pheatmap" package. Venn diagrams were created using the "venn. diagram, " "draw. triple. venn, " and "draw. quad. venn" functions in the "VennDiagram" package to visualize the number of OTUs that were shared between compartments, contamination, and cultivar type.
Venn diagrams of soil and endophyte bacterial samples were produced separately because unique OTUs were formed for each dataset upon clustering. Co-inertia analysis  (Dolédec and Chessel, 1994)  was used to determine whether bacterial and fungal communities were similarly affected by contamination and cultivar type. The OTU abundance data was first normalized with a hellinger transformation with the "decostand" function of the "vegan" package. Co-inertia analyses were then performed separately for each compartment with the "dudi. pca" and "coinertia" functions in the "ADE4" package  (Dray and Dufour, 2007). The significance was tested by permutation using the "randtest" function in the "ADE4" package. Similarly, co-inertia analysis was used to determine the covariance between rhizosphere microbial communities and endophyte root and stem microbial communities.