dataesr/scientific-paragraphs-categorization huggingface dataset:
[22]  analyzed 383 consecutive patients treated with ORC (120) or RARC (263) between 2001 and 2014 at a single institution. With a median follow up of 30 months (for ORC) and 23 months (for RARC), they reported similar recurrence rates with an increasing risk of experiencing extrapelvic lymph node recurrence and peritoneal carcinomatoses for RARC patients. Our analyses did not include the type of recurrence limiting our ability to test this aspect; but we found a similar overall recurrence risk for patients treated with RARC when compared to ORC. Hu et al. [25] , using the SEER database compared 439 patients treated with RARC and 7308 treated with ORC. These authors observed an increasing trend in RARC utilization over the study period and with a median follow up of 44 months, they found no survival differences between the two techniques.
After antibody staining, cells were washed. Cells were stained intra-cellularly using Foxp3 Permeabilzation/Fixation Kit according to manufacturer's notice (eBiosciences, ThermoFisher, Waltham, MA, USA). Cells were washed prior to cell acquisition. LSR Fortessa (BD Biosciences, Franklin Lakes, NJ, USA) was used for flow cytometry acquisition, with Diva6 software (BD Biosciences, Franklin Lakes, NJ, USA), and analyzed with FlowJo 10 software (FlowJo LLC, Ashland, OR, USA). For visual purposes, only 8000 events maximum are shown in each FACS dot plot. Cells were purified with a FACSAria III sorter (BD Biosciences, Franklin Lakes, NJ, USA). Cells were recovered in PBS for cell injection, or OPTIMEM (Gibco, Thermo Fisher, Waltham, MA, USA) plus 10% FCS for cell culture.
Enhancer Prediction Algorithms Several types of enhancer predicting algorithms have been developed for integrating multiple types of data, such as TF motifs, ChIP-seq, DNase-seq, ATAC-seq, and P300 binding data sets for enhancer prediction by using clustering and machine learning approaches. These algorithms include supervised tools that rely on high-confidence positive and negative training sets (e. g. , known-and non-enhancers) to build models that can maximize the differentiation between enhancer and nonenhancer sets, and non-supervised tools that are used to identify hidden and unknown patterns directly from data. Examples of supervised algorithms include CSI-ANN  (Firpi et al. , 2010) , ChromaGenSVM  (Fernandez and Miranda-Saavedra, 2012) , RFECS  (Rajagopal et al. , 2013) , EnhancerFinder  (Erwin et al.
, 2014) , DEEP  (Kleftogiannis et al. , 2015) , DELTA  (Lu Y. et al. , 2015) , PEDLA  (Liu F. et al. , 2016) , REPTILE  (He et al. , 2017) , eHMM  (Zehnder et al. , 2019) , and DBN  (Bu et al. , 2017). Other unsupervised algorithms such as Segway  (Hoffman et al. , 2012)  and ChromHMM  (Ernst and Kellis, 2012)  integrate multiple types of epigenome data to define chromatin segmentations that can be used to assign functional roles for various parts of chromatin. One of the main problems of all these prediction programs is that we still lack a detailed understanding of the underlying regulatory code in the non-coding genome. Despite all the advances made over the last decade, we are yet to pinpoint a feature that can identify enhancers (and their activity) in all cell types. As most programs rely on previously generated training sets or on static features such as DNA sequence motifs which by their own do not necessarily predict enhancers in each instance, it is more than logical that despite the large amount of efforts that are undertaken, enhancer prediction programs are far from perfect.
Another area where further progress needs to be achieved is the prediction of enhancer-promoter interactions, that can be used to assign NCREs to their target genes. Several tools for this are currently available, such as ELMER 2 and InTAD. ELMER 2 computes the correlation between the enhancer and target genes by combining both DNA methylation and gene expression data derived from the same data set  (Silva et al. , 2018) , but is limited by the fact that correlations are restricted to the closest neighboring gene, which does not necessarily present the real biological relevant target gene  (Kvon et al. , 2014). InTAD is a tool to detect genes located upstream and downstream of the enhancer in the same TAD boundary and it can support different types of data as input. The TAD information comes from available Hi-C datasets  (Okonechnikov et al.
, 2019) , but the currently available ones have a low resolutions and, up to date, have include a limited number of cell types. Therefore, to fully identify enhancer-gene interactions, it will be important to come up with novel experimental procedures that will enable us to directly test the biological relevance of enhancer-promoter predictions. Finally, a number of programs are developed that aim to predict the functional relevance and possible pathogenicity of variants in NCREs. These include, amongst others, RegulomeDB  (Boyle et al. , 2012) , HaploReg  (Ward and Kellis, 2012) , CADD  (Kircher et al. , 2014) , GWAVA  (Ritchie et al. , 2014) , GenoCanyon  (Lu Q. et al. , 2015) , Genomiser  (Smedley et al. , 2016), and INFERNO (Amlie-Wolf et al. , 2018). In addition, a number of databases have been generated, including HEDD  (Wang Z.
Available Enhancer Databases As the available information on the non-coding genome is increasing rapidly over the last decade, more and more resources are available online that can help to localize NCREs and interpreting their functional roles. In the next part, we summarize a selection of databases and resources that are currently available (Table  2 ) and can be used to find NCREs of relevance for brain development. One of the first resources of experimentally tested NCREs was the VISTA enhancer database  (Visel et al. , 2007). Based on comparative genomics, a large selection of putative NCREs from mouse and human was selected and tested in transgenic mouse embryo assay, to determine their in vivo enhancer activity, as determined by LacZ expression. The database provides detailed information on the genomic localization of the tested sequences, likely associated genes, and images of transgenic mouse embryos identifying the localization of enhancer driven LacZ expression.
Based on the 2/25/2019 update, this database contains 542 tested enhancers active in human forebrain, hindbrain, and midbrain. In addition, the VISTA tool portal can be used as a comparative tool and users can submit their own sequences to conduct comparison against multiple species  (Visel et al. , 2007) , thereby possibly identifying conserved functional NCREs. EnhancerAtlas 2. 0 is a database that has collected putative NCREs based on publicly available data obtained from ChIPseq for different histone modification, TFs, EP300, and POLII, CAGE and eRNA expression, interaction studies by ChIA-PET (a method that combines 3C with chromatin immunoprecipitation) and chromatin accessibility as determined by FAIRE and DNase-seq. Each putative enhancer is supported by at least three independent high-throughput data sets although the database does not contain any direct functional validations.
It contains more than 4,506,217 putative enhancers from 8,573 datasets of 179 human tissue/cells, which through an interactive website can be easily accessed. The 49,925 human fetal brain and 17,103 cerebellum enhancers were predicted using DNa-seq, CAGE, and H3K4me1, and H3K27ac deposition  (Gao et al. , 2016). The FANTOM5 database, is the latest version of the FANTOM project, that aims to generate an atlas of mammalian regulatory elements, transcriptomes, and long-noncoding RNAs. NCREs are predicted from sequencing data from cap analysis of gene expression (CAGE) along with RNA-Seq data from multiple tissue and cell types from different developmental time points  (Andersson et al. , 2014). In total, the database contains more than 43,000 putative enhancers, of which 639 are expressed in brain and 376 were found in neuronal stem cells.
et al. , 2018). In addition, 96,375 enhancers were shown to interact with protein coding genes during fetal brain development and during in vitro differentiation of brain organoids  (Amiri et al. , 2018). Of the latter, 46,735 enhancers were active only in fetal cortex. Several super-enhancer databases have been generated such as dbSUPER  (Khan and Zhang, 2016) , SEA  (Wei et al. , 2016) , and SEdb  (Jiang et al. , 2019)  providing annotation, genomic   (Visel et al. , 2007) , dbSUPER (n = 6002)  (Khan and Zhang, 2016) , FANTOM5 (n = 639)  (Andersson et al. , 2014) , PsychENCODE (n = 46731; the 46735 enhancers mentioned in the text are in genome build hg38 and the difference of four loci is due to liftover to hg19)  (Amiri et al. , 2018)  and EnhancerAtlas 2. 0 (n = 49925)  (Gao et al. , 2016).
(B) Venn diagram showing the overlap between ChIP-seq peaks for H3K27ac from adult brain, as identified in three studies: Sun (n = 56503)  (Sun et al. , 2016) , Vermunt (n = 83553)  (Vermunt et al. , 2014) , and Wang (n = 124437)  (Wang D. et al. , 2018). The intersection between different sources (in the same order as above) was performed using bedops and bedtools. In both graphs, the minimum overlap of a single nucleotide is required. coordinates and length of super-enhancers, and their possible associated genes. Among those, dbSUPER is one of the most popular databases with 82,234 super-enhancers from multiple human and mouse cell types. In this database, there are 6,002 and 1,114 super-enhancers detected by H3K27ac enrichment from seven human and three mouse brain tissues and cell types, respectively  (Khan and Zhang, 2016).
Finally, GeneHancer is a database of human enhancers and their inferred target genes  (Fishilevich et al. , 2017). Integrating enhancer predictions from ENCODE, Ensembl, FANTOM and VISTA yielded more than 280,000 candidate regions that where assigned to their target genes based on co-expression correlation, expression of quantitative trait loci and capture Hi-C. Although all of these databases can easily be accessed and are user friendly, it is important to realize when using them that it is still difficult to judge which of the sources provides the user with those sequences that are indeed most likely to be of functional biological relevance. To illustrate this, it is interesting to compare the overlap between predicted enhancers from the various resources. When we compare putative brain enhancers from VISTA, EnhancerAtlas 2.
Doan and colleagues performed a similar targeted sequencing approach assessing so-called human accelerated regions (HARs)  (Doan et al. , 2016). HARs are conserved regions with elevated divergence in humans and this might reflect potential roles in the evolution of human-specific traits. This study provides evidence that HARs can function as regulatory elements for dosage-sensitive genes expressed in the central nervous system. Using data from a large cohort study investigating 2100 sibling cases of ASD, they found that de novo CNV's affecting HARs, or HAR-containing genes, could be implicated in up to 1. 9% of ASD cases in simplex families. They then analyzed consanguineous ASD cases using WGS from 30 affected and 5 unaffected individuals and designed a custom capture array to sequence HARs in another 188 affected and 172 unaffected individuals.
Single-cell RNA-sequencing has become a widely used, powerful approach for studying cell populations. However, these methods often generate multiplet artifacts, where two or more cells receive the same barcode, resulting in a hybrid transcriptome. In most experiments, multiplets account for several percent of transcriptomes and can confound downstream data analysis. Here, we present Single-Cell Remover of Doublets (Scrublet), a framework for predicting the impact of multiplets in a given analysis and identifying problematic multiplets. Scrublet avoids the need for expert knowledge or cell clustering by simulating multiplets from the data and building a nearest neighbor classifier. To demonstrate the utility of this approach, we test Scrublet on several datasets that include independent knowledge of cell multiplets.
, 2018). Both of these methods have drawbacks. As we will show later, the former method often performs poorly because it assumes that cells contain similar amounts of RNA, when in reality samples with diverse cell types or cells in different cell cycle stages are expected to have a wide range in the number of transcripts per cell. The latter method requires expert knowledge and careful annotation of the data. Below, we propose a computational approach, Single-Cell Remover of Doublets (Scrublet), for identifying multiplets and apply the method to several datasets that include some measure of "ground truth" labels for cell multiplets. Briefly, our method involves two steps. First, doublets (multiplets of just two cells) are simulated from the data by combining random pairs of observed transcriptomes.
With these assumptions, we show that doublets simulated from empirical datasets can be used to construct a "target-decoy"  (Elias and Gygi, 2010)  k-nearest neighbor (KNN) classifier capable of identifying doublets, as shown schematically in Figure  1C  (Bayesian classifier derived in STAR Methods). When the total fraction of transcriptomes expected to be doublets is known a priori, the classifier outputs a posterior likelihood of an observed transcriptome being a doublet. The prior fraction is often hard to estimate, however. Therefore, Scrublet uses a threshold likelihood based on the observation that classifier scores are largely bimodal for doublets simulated from empirical data. A low-scoring mode corresponds to doublets indistinguishable from singlets, and thus such doublets are "embedded," while a high-scoring mode corresponds to doublets departing from singlet states and thus can be considered neotypic.
4. A binary label for each cell identifying neotypic doublets. If the expected fraction of transcriptomes that correspond to doublets in a dataset is ρ, then the fraction of all transcriptomes classified as neotypic doublets should equal ρ ⋅ ϕ D , while a remaining fraction ρ ⋅ (1 -ϕ D ) of the observed transcriptomes will correspond to undetected "embedded" doublets. Our implementation of the Scrublet classifier (github. com/AllonKleinLab/scrublet) can incorporate arbitrary functions for preprocessing and embedding of single-cell data. We also provide a standalone default set of functions to construct the cell state manifold by applying principal-component analysis (PCA) to the observed transcriptomes and simulated doublets. The implementation avoids the need to cluster data or predefine cell state marker genes, and it is suitable for routine use, with classification of datasets of tens of thousands of cells requiring only a few minutes (Figure  S1 ).
RESULTS The results are organized into five sections. First, we test Scrublet on simulated datasets in order to assess its performance and limitations under simplified conditions where there is perfect knowledge of singlet and doublet identity. We then apply Scrublet to three experimental datasets, each of which provides some form of independent ground truth for doublet identity. Finally, we apply Scrublet to our own recently published hematopoiesis dataset, which presents a complex continuum of well-characterized cell states and where doublets can be identified through prior knowledge.
Performance on Simulated Data Using pedagogical tests on simulated data, our goal is to demonstrate that (1) it is possible to use the proposed approach as a classification scheme; and (2) the detectable doublet fraction, ϕ D , can be used to estimate the sensitivity of the classifier, i. e. , the fraction of true doublets that one might be able to identify using this approach alone. Using the Splatter package  (Zappia et al. , 2017) , we simulated single-cell data in the form of distinct cell clusters or as a continuum of cell states (Figure  2A ). Though these simulations may not accurately recapitulate actual scRNA-seq data, they serve as an initial demonstration of Scrublet's strengths and limitations. Applications to real data in later sections more usefully test Scrublet's performance.
Performance on Dataset #1: Human-Mouse Cell Mixture We tested the Scrublet on a publicly available dataset consisting of a mixture of human (HEK293T) and mouse (NIH3T3) cells (Figure  3A ). This dataset, though not representative of most single-cell experiments, provides a useful test case because the differences between human and mouse genomic sequence provide an independent way to detect doublets  (Klein et al. , 2015; Macosko et al. , 2015). We defined a partial ground truth on doublet identity according to whether a cell barcode associates with transcripts from both species (a doublet) or just one species (Figure  3B ). Because doublets arising from the encapsulation of two human or two mouse cells cannot be identified as such, we expected our doublet detector to correctly predict all ground truth labeled doublets, since they arise from distinct human and mouse cell types and should thus be neotypic.
After hiding species labels and restricting to orthologous genes (Figure  3C ), Scrublet estimated the detectable doublet fraction at ϕ D = 54%, close to the 50% expected for crossspecies doublets given equal input of mouse and human cells (Figure  3D ). Furthermore, the detector accurately identified human-mouse doublets with a receiver-operator characteristic (ROC) area under the curve (AUC) of 0. 99 (recall of 98% of human-mouse doublets with precision of 96%) (Figures  3E, 3F , and S2A), and the overall Scrublet-predicted doublet fraction agreed well with the expected fraction (Table  2 ). False positives and negatives tended to lie in "boundary states. " Specifically, false negatives (mixed-species doublets incorrectly called as singlets by Scrublet) had lower degrees of species mixing than true positives, while false positives had below-average total transcript counts and lay on the border between the doublet and singlet clusters in PCA space (Figure  S2B ).
Performance on Dataset #2: Peripheral Blood Cells from Multiple Individuals To test the doublet detector in a more typical experimental context, we evaluated its performance using a published dataset generated from a mixture of eight genotyped human donors' mature blood cells (peripheral blood mononuclear cells, PBMCs)  (Kang et al. , 2018). The authors identified ground truth multiplets as cell barcodes associated with reads containing polymorphisms from more than one individual (Figure  4A ). Given that the data represent a similar number of cells from each individual, roughly 7 out of 8 doublets should occur between individuals and can be identified using this approach. Thus, in the ground truth data, we expect at least 12. 5% of true doublets to be undetected. To make use of this orthogonal method for multiplet detection, we compared Scrublet predictions to the ground truth doublets and also generated a "ground truth score" by calculating the fraction of each cell's neighbors that were mixed genotype doublets  (Figures 4B and 4C).
Furthermore, the recall of true doublets was accurately predicted by f D , the detectable doublet fraction as measured by the simulated doublet distribution (Figure  4F ). At the selected doublet score threshold, the recall of 49% was in good agreement with the f D of 55%, and this held true across a range of thresholds. This suggests that even though many doublets go undetected, the fraction of identifiable doublets can be accurately estimated. Though the precision was just 66%, this can be explained in part by the imperfect nature of the ground truth labels, since doublets formed by cells from the same individual are undetectable based on genotype alone. Finally, the predicted overall doublet fraction was only slightly higher than the experimentally estimated value (Table  2 ). As with the previous dataset, we compared the doublet detector performance to alternative strategies: (1) identifying cells coexpressing curated marker genes of distinct cell types and (2) identifying cells with high total transcript counts.
For the former method, we created a list of highly specific marker genes of each cell type in this dataset and then calculated the amount of co-expression of marker genes from different cell types (Figure  4G ) to define a "marker co-expression score" (STAR Methods). Of the 773 true doublets correctly identified by Scrublet, 68% also had a high degree of marker gene co-expression. Overall, the "marker co-expression score" did not perform as well as Scrublet (AUC 0. 77 versus 0. 88) and required significant manual annotation. For the method relying on high total transcript counts, we found that true doublets did tend to have higher total transcript counts than singlets (AUC = 0. 81) (Figures  4H  and 4I ). Because total counts appeared to be informative and did not require any manual annotation, we created a hybrid predictor by linear combination of each cell's Scrublet score with its locally normalized total counts (STAR Methods).
Performance on Dataset #3: Peripheral Blood Cells at Multiple Concentrations In a third test, we turned to a dataset that offers a less direct independent strategy for detecting neotypic doublets: namely, a single sample of PBMCs split and barcoded at two different cell concentrations, yielding either 4,352 ("PBMC-4k") or 8,391 ("PBMC-8k") transcriptomes. We reasoned that multiplet-specific cell states should be identifiable as clusters whose relative abundance increases with increasing input cell concentration because, in fully dissociated samples, a doubling of cell concentration doubles the probability of randomly encapsulating two cells into the same droplet. In the PBMC data, states composed uniquely of doublets should double in relative abundance, with cell states that are predominantly singlets decreasing only incrementally (Figure  5A ).
As expected, the doublet detector identified roughly twice as many doublets in the PBMC-8k sample (4. 1%) as in PBMC-4k (1. 8%) (Figures 5B, 5C, and S4A-S4D), and as with the previous datasets, Scrublet's estimate for the overall doublet fraction was in close agreement with the expected value (Table  2 ). Furthermore, when we compared the PBMC-8k cells to their most similar PBMC-4k counterparts, the predicted doublet states were present at a higher relative abundance, while singlet states changed little or decreased (Figure  5D ). This test again suggests that the doublet detector correctly identifies neotypic doublets. We used this dataset to demonstrate that removal of doublets can improve the quality of downstream analyses, such as identification of genes expressed in a cell type-specific manner.
Prediction of Doublets in a Cell State Continuum The above examples demonstrate Scrublet's ability to correctly identify neotypic doublets from datasets consisting of distinct cell types. In a final example, we applied it to a continuum of cell states by analyzing transcriptomes of KIT+ hematopoietic progenitors from the mouse bone marrow  (Tusi et al. , 2018)  (Figure  6A ). These cells form a continuum from multipotent progenitors to unilineage committed cells. Several groups of doublets were readily distinguishable (Figures 6B, 6C, and S5) and formed "bridges" between different committed progenitor types. Here, we lack a ground truth for confirming the identity of the doublets, but since such bridges are inconsistent with our current understanding of hematopoiesis, it is likely that our doublet detector is correct in identifying them.