dataesr/scientific-paragraphs-categorization huggingface dataset:
Gene cluster characterization We used the MMseqs2 taxonomy module (commit: b43de8b7559a3b45c8e5e9e02cb3023dd339231a) in combination with the UniProtKB (release of January 2018) (UniProt  Consortium, 2018)  to retrieve the taxonomic ids of all genes in a gene cluster. The taxonomy module implements the 2bLCA  (Hingamp et al. , 2013)  to compute the lowest common ancestor of query sequence. We used the following parameters "-e 1e-05 -covmode 0 -c 0. 6" for the search. To retrieve the taxonomic lineages, we used the R package CHNOSZ  (Dick, 2008). We used eggNOG-mapper  (Huerta-Cepas et al. , 2017)  and the EggNog5 database  (Huerta-Cepas et al. , 2019)  to provide functional annotations for each gene in a gene cluster. We refined the functional annotations by selecting the orthologous group within the lowest taxonomic level predicted by EggNog-mapper.
We measured the intra-cluster taxonomic and functional admixture by applying the entropy. empirical() function from the entropy R package  (Hausser and Strimmer, 2008). This function estimates the Shannon entropy based on the different taxonomic and functional annotation frequencies. For each cluster, we also retrieved the cluster consensus taxonomic and functional annotation using a quorum majority voting approach. In addition to the taxonomic and functional annotations, we evaluated the clusters' level of darkness and disorder using the Dark Proteome Database (DPD)  (Perdigão et al. , 2017)  as reference. We searched the cluster genes against the DPD, applying the MMseqs2 search program  (Steinegger and Soding, 2017)  with "-e 1e-20 --cov-mode 0 -c 0. 6". For each cluster, we then retrieved the mean and the median level of darkness, based on the gene DPD annotations.
High-quality clusters We defined a subset of high-quality clusters based on the completeness of the cluster genes and their representatives. We identified the minimum required percentage of complete genes per cluster by a broken-stick model  (Macarthur, 1957)  applied to the percentage distribution. Then, we selected the GCs found above the threshold and with a complete representative. A set of non-redundant domain architectures We estimated the number of potential domain architectures present in the Known GCs considering the large proportion of fragmented genes in the metagenomic dataset and that could inflate the number of potential domain architectures. To identify fragments of larger domain architecture, we considered their topological order in the genes. To reduce the number of comparisons, we calculated the pairwise string cosine distance (q-gram = 3) between domain architectures and discarded the pairs that were too divergent (cosine distance ≥ 0.
Validation of gene cluster communities We tested the biological significance of the GCCs using the phylogeny of proteorhodopsin  (Boeuf et al. , 2015)  (PR). We used the proteorhodopsin HMM profiles  (Olson et al. , 2018)  to screen the marine metagenomic datasets using hmmsearch (version 3. 1b2)  (Finn et al. , 2011). We kept the hits with a coverage > 0. 4 and e-value <= 1e-5. We removed identical duplicates from the sequences assigned to PR with CD-HIT  (Li and Godzik, 2006) (v4. 6 ) and cleaned from sequences with less than 100 amino acids. To place the identified PR sequences into the MicRhode  (Boeuf et al. , 2015)  PR tree first, we optimized the initial tree parameters and branch lengths with RAxML (v8. 2. 12)  (Stamatakis, 2014). We used PaPaRA (v2. 5)  (Berger and Stamatakis, 2012)  to incrementally align the query PR sequences against the MicRhode PR reference alignment and pplacer  (Matsen et al.
Rate of genomic and metagenomic gene clusters accumulation We calculated the cumulative number of known and unknown GCs as a function of the number of metagenomes and genomes. For each metagenome count, we generated 1000 random sets, and we calculated the number of GCs and GCCs recovered. For this analysis, we used 1,246 HMP metagenomes and 358 marine metagenomes (242 from TARA and 116 from Malaspina). We repeated the same procedure for the genomic dataset. We removed the singletons from the metagenomic dataset with an abundance smaller than the mode abundance of the singletons that got reclassified as good-quality clusters after integrating the GTDB data to minimize the impact of potential spurious singletons. To complement those analyses, we evaluated the coverage of our dataset by searching seven different state-of-the-art databases against our set of metagenomic GC HMM profiles (Appendix 9).
, 2008 ) R package. We created 100 random datasets picking up one random sample from each group. For each of the 100 random datasets, we created 100 random abundance matrices using the nullmodel function of the quasiswap count method  (Miklós and Podani, 2004 ). Then we calculated the observed NB and obtained the 2. 5% and 97. 5% quantiles based on the randomized sets. We compared the observed and quantile values for each gene cluster and defined it to have a Narrow distribution when the observed was smaller than the 2. 5% quantile and to have a Broad distribution when it was larger than the 97. 5% quantile. Otherwise, we classified the cluster as Non-significant  (Salazar et al. , 2015). We used a majority voting approach to get a consensus distribution classification based on the ten random datasets.
Evaluation of the OM-RGC v2 uncharacterized fraction We integrated the 46,775,154 genes from the second version of the TARA Ocean Microbial Reference Gene Catalog (OM-RGC v2)  (Salazar et al. , 2019)  into our cluster database using the same procedure as for the genomic data. We evaluated the uncharacterized fraction and the genes classified into the eggNOG (Huerta-Cepas et al. , 2019) category S within the context of our database. Augmenting RB-TnSeq experimental data We searched the 129,477 bacterial genes associated with mutant phenotypes from Price et al. (Price et al. , 2018)  against our gene cluster profiles. We kept the hits with e-value ≤ 1e-20 and a query coverage > 60%. Then we filtered the results to keep the hits within 90% of the Log(beste-value), and we used a majority vote function to retrieve the consensus category for each hit.
Acknowledgements The authors thankfully acknowledge the computer resources at  MareNostrum  and the technical support provided by  Barcelona Supercomputing Center  ( RES-AECT-2014-2-0085 ), the  BMBFfunded de. NBI Cloud within the German Network for Bioinformatics Infrastructure (de. NBI)  ( 031A537B ,  031A533A ,  031A538A ,  031A533B ,  031A535A ,  031A537C ,  031A534A ,  031A532B ), the  University of Oxford Advanced Research Computing  (http://dx. doi. org/10. 5281/zenodo. 22558)and the  MARBITS bioinformatics core at ICM-CSIC. CV was supported by the  Max Planck Society. AFG received funding from the  European Union 's  Horizon 2020 research and innovation program Blue Growth: Unlocking the potential of Seas and Oceans  under grant agreement no. 634486  (project acronym  INMARE ). AM was supported by the  Biotechnology and Biological Sciences Research Council  [ BB/M011755/1 ,  BB/R015228/1 ] and RDF by the  European Molecular Biology Laboratory  core funds.
Availability of data and materials The code used for the analyses in the manuscript is available at https://github. com/functionaldark-side/functional-dark-side. github. io/tree/master/scripts. The code to recreate the figures is available at https://github. com/functional-dark-side/vanni_et_al-figures. Detailed descriptions of the different methods and results of this manuscript are available at https://dark. metagenomics. eu. The workflow AGNOSTOS is available at https://github. com/functional-dark-side/agnostos-wf (the version used for the manuscript can be found here https://doi. org/10. 5281/zenodo. 4557847) and its database can be downloaded from https://doi. org/10. 6084/m9. figshare. 12459056.
Methods We searched the gene clusters (GCs) without any Pfam annotated gene against two functional databases, the UniRef90, from UniProt (The UniProt  Consortium, 2017) , and the NCBI nr database (NCBI Resource Coordinators, 2018). We screened the two databases using the cluster consensus sequences, obtained by applying the hhconsensus program of the HH-SUITE  (Steinegger et al. , 2019)  on the clusters multiple sequence alignments (MSAs) generated with the FAMSA program  (Deorowicz et al. , 2016). We performed two nested searches using the MMSeqs2  (Steinegger & Söding, 2017)  program and following a similar workflow as the ''2bLCA'' described in Hinghamp et al. (Hingamp et al. , 2013). The search-workflow consisted of five steps: First, we searched the consensus sequences against the functional database, with -e 1e-05 --cov-mode 2 -c 0.
6. Second, we extracted the high scoring pairs (HSP) of the best hits and we searched them again using the same parameters. Third, we merged the top hits from the first with the second search results. Fourth, we filtered out the second search hits with a bigger e-value than the first search top hits. And fifth, we selected the hits that were found in 60% of the log10(best-e-value). We first applied this search-workflow to screen the UniRef90 database (release 2017_11) (The UniProt  Consortium, 2017). We classified the GCs as GU if their consensus sequences were found annotated to proteins labeled with any of the terms commonly used to define proteins of unknown function in public databases (Supplementary file 1G). We classified, instead, as KWP, the clusters with consensus annotated to functionally characterized proteins.
Secondly, we applied the same search-workflow to search the consensus sequences with no homologs in the UniRef90 database, against the NCBI nr database (release 2017_12) (NCBI Resource Coordinators, 2018). We used the same criteria to classify a GC as GU or KWP. Ultimately, we classified as EU the GCs whose consensus sequences did not align with any of the NCBI nr entries. We processed the Pfam annotated GCs to retrieve a GC consensus domain architecture (DA). We classified as GU the GCs with a consensus DA composed only of Pfam domain of unknown function (DUFs) and as K the rest. The methods for this step are described in Methods -Remote homology classification of gene clusters. We refined the classified GCs to account for remote homologies. A detailed description of this process can be found in Methods -Gene cluster remote homology refinement.
Methods We searched seven different state-of-the-art databases against our dataset of cluster HMM profiles. The different profile searches were all performed using the MMSeqs2 (version 8. fac81) search program  (Steinegger & Söding, 2017) , setting an e-value threshold of 1e-20 and a query coverage threshold of 60% (-e 1e-20 --cov-mode 2 -c 0. 6). We kept the hits within 90% of the log10(best-e-value). Then we applied a majority vote function to retrieve the consensus functional category (K, KWP, GU or EU) for each search hit. In the end, the results were sorted by the lowest e-value and the largest query and target coverage to keep only the best hits. We applied the described method to the following datasets: the Families of Unknown Functions (FUnkFams) (61,970 genes)  (Wyman et al. , 2018) , the Pacific Ocean Virome (POV) (4,238,638 genes) (Hurwitz & Sullivan, 2013) and the Tara Ocean Virome (TOV) (6,642,187 genes)  (Brum et al.
, 2015). The Genome Taxonomy Database (GTDB)  (93, 723, 190  archaeal and bacterial genes)  (Parks et al. , 2018). The MGnify proteins from the EBI metagenomics database (release 2018_09)  (Mitchell et al. , 2018) (843,535,611 genes). The manually curated collection of 957 MAGs from TARA metagenomes  (Delmont et al. , 2017 ) (TARA MAGs) (2,288,202 genes), and the one made of 92 MAGs, from the fecal microbiota transplantation study (FMT MAGs) of  Lee et al. (Lee et al. , 2017) (188,983 genes). And also, the collection of unannotated genes with mutant phenotypes identified in  Price et al. 2018 (Price et al. , 2018 ) (37,684 mutant genes).
Methods We searched the metagenomic EU GCs HMM profiles, obtained from the cluster MSA using the hhmake program of the HH-SUITE  (Steinegger et al. , 2019) , against the set of 957 highquality MAGs binned from the TARA Ocean prokaryotic dataset  (Delmont et al. , 2017). We performed the sequence-profile search using the MMSeqs2 search program  (Steinegger & Söding, 2017) , using -e 1e-20 --cov-mode 2 -c 0. 6. We filtered the results to keep the hits within 90% of the log10(best-e-value). We applied a majority vote function to retrieve the consensus category for each hit. Then, we sorted the results by the smallest e-value and the largest query and target coverage to keep only the best hits. We then filtered the search results focusing on the broadly distributed EU GCs and GCCs. We retrieved MAG contigs containing the EU GCs and GCCs from the Anvi'o MAG profiles using the program anviexport-gene-calls from Anvi'o v4  (Eren et al.
, 2015). We functionally annotated the contigs searching their genes against the Pfam database (v. 31. 0) (Finn et al. , 2016) , using the hmmsearch program from the HMMER package (version: 3. 1b2)  (Finn et al. , 2011) , and complementing the search using Prokka (Seemann, 2014) in metagenomic mode. We then selected the contig with the lowest percentage of hypothetical proteins, and we extracted a region of 1kb surrounding the genes mapping to the EU GCCs.
, 2017) Dataset Reference Samples Sites Reads GOS Rusch et al. (Rusch et 80 70 12,672,518 al. , 2007) (B) Number of predicted genes per completeness category. Total "00" "10" "01" "11" 322,248,552 118,717,690 106,031,163 102,966,482 75,694,123 Note: "00"=complete, both start and stop codon identified. "01"=right boundary incomplete. "10"=left boundary incomplete. "11"=both left and right edges incomplete.
Data analysis is increasingly performed over data assembled from uncontrolled sources, facing inconsistency in knowledge-representation conventions. The typical practice is to create ''clean'' data for analysis, matching entities and merging variants to overcome differences in knowledge representation. Despite progress in data management techniques to automate this process, it still needs labor-intensive supervision from the analyst. In this paper, we evaluate the benefit of advanced statistical tools to address directly many analytic tasks across data sources without such entity-matching cleaning. Reframing analytical questions as machine-learning tasks enables to replace exact matching of entities by continuous descriptions -vectorial embeddings-that expose similarities between entries. But are analyses with less cleaning trustworthy? We answer this question with a thorough benchmark on questions typical of socio-economic studies across 14 employee databases: we compare the approaches based on machine learning to manual data cleaning (entity matching).
For a given job, how does salary evolve with experience? 2. For a given job, what is the 0. 75-quantile of salaries? 3. What is the typical male-female pay gap? Answering these questions from data assembled across different employers must overcome the lack of correspondences in job titles: as illustrated on Fig. 1 , the same job title appears with multiple variants, such as senior research associate and sr research assoc. This problem is typically addressed via entity matching procedures, available in increasingly sophisticated data-integration softwares such as Wrangler  [13] , Tamr  [14] , OpenRefine  [15]. Despite these tools, entity matching remains a difficult task as it often involves domain expertise or faces the lack of clear correspondences in entities across sources. Is such matching necessary to the validity of the analysis, or can more complex statistical pipelines do without? In this paper, we show that applying advanced statistical techniques directly to non-normalized data can avoid labour-intensive data curation for many analytical questions.
The issue is that both similarity and threshold must be tailored to domain specificities and the resulting matches must often be manually reviewed. The process is thus labor intensive. Automating the match of entities is challenging because it is an unsupervised-learning problem  [27] , unless there are known matches for supervision  [28] -  [30]. Such matches must typically be constructed manually by a user, though active learning can reduce human intervention  [18] ,  [31]. Dedicated data-integration softwares, such as http://openrefine. org/OpenRefine, facilitate the process with a user interface. The software suggests potential matches and enables users to tune parameters and match entries in a semiautomated way. Automation tools can use techniques capturing natural language semantics, which shares with entity matching the challenge of relating multiple forms that denote the same things.
The underlying problem is that of statistical estimation: to compute the quantity that represents best the complete population from the instances at hand. If the entity matching is valid, matching & averaging estimates are unbiased from a statistical point of view. But they may exhibit high variance, as the data often exhibits a small number of representatives from a given category. A paradox of statistics is that the most accurate way of estimating the mean of a population from a small sample may not be the sample average, but biasing estimates with other sources of information (see Stein's paradox  [35] ). For instance, estimating the typical salary of an associate professor can leverage similar populations: professor, lecturer. Drawing information across similar entities is related to the notion of semantic queries in databases  [36] , as opposed to exact value matching, used in matching & averaging.
The outcome y i can take two potential values depending on W i : y i (0) = y i (W i = 0) or y i (1) = y i (W i = 1) (1 for a man, 0 for a woman), though for each individual only one of these is observed in the data. The analytical quantity of interest is the typical gender pay gap, known as the average treatment effect τ = E[y(1)y(0)]: the average difference in outcome for the same individual under scenario W i = 1 and W i = 0, i. e. that differ only by their sex  [46] ,  [47]. In general, it does not suffice to subtract the average salary of men from that of women: E[y|W = 1] -E[y|W = 0] = τ. Indeed, the populations of men and women may not be directly comparable in the database. For instance, women may need to interrupt their careers during maternity leave, causing them to have less work experience and thus lower salaries.
A. EXPERIMENTAL DETAILS 1) MEASURING ESTIMATION ERROR How to compare estimators of a quantity such as conditional expectation of salary given job title? Even without entitymatching noise, the data at hand is limited and its mean is 3 The code and data to reproduce our experiments is available on Code Ocean: https://codeocean. com/capsule/6435573/tree an imperfect estimate of the unknown population quantity y. We adapt a classic procedure of machine learning: we leave out a test fraction of the databases, and use the rest of the data to derive estimates ŷtrain. Applying an averaging-based estimator on the test data provides another estimate ŷtest , that is unbiased though noisy. Importantly, as it has been estimated from different data than ŷtrain , its estimation error is independent. We can thus use the difference between ŷtrain and ŷtest over multiple splits -a cross-validation loop-to quantify the estimation error of the procedure that we use to compute ŷtrain.
B. QUALITATIVE RESULTS: DISPERSION ACROSS VARIANTS The curves of salary as a function of experience represented on Fig. 2  are computed either with a matching-based or a learning-based approach. Machine-learning estimates leverage job similarities and have low dispersion across variants of project manager or administrative assistant. This robustness reduces the need for manual matching: taking the model output on any variant provides reliable estimates that are representative of the whole population. It is more convenient and reliable for an analyst to query the model for ''project manager'' or ''administrative assistant'' (thick magenta curves), than to search the database for all variants and average them. Beyond the dispersion across variants, matching & averaging curves appear more noisy; in particular they fail to capture well the evolution of salary with experience.
C. QUANTITATIVE RESULTS: CROSS-VALIDATED ERRORS To go beyond the face validity of Fig. 2 , we use crossvalidation, as detailed in subsection IV-A, to quantify which approach best estimates the population quantities. The 14 databases are randomly split into two sets of 7 databases: one to compute estimates for salary, quantile, and propensityscore; and the other to measure their error, reported in Table  3. Results show that for all three quantities embeddings notably reduce the error compared to exact matching and perform best when combined with learning. Adding manual matching on top of embeddings improves further, but the benefit is smaller than that brought by embeddings & learning. The residual error is due to variance in individual salary that is not explained by the attributes of the employees present in the databases, such as the appreciation of the manager.
D. ESTIMATION OF COUNTERFACTUALS How do the differences in estimation errors reported in Table  3  impact complex end-user analytical questions? We investigate their impact on estimation of salary gap across sex. Fig. 4  gives average treatment effects computed with statistical methods -IPS and AIPS  [52] -based on embedding & learning approaches, as well as manual matching and fuzzy-matching estimates (see appendix). To force the need for analysis across the databases, we create a sex imbalance by dropping randomly a fraction of either men or women in each database, with 50/50 probability. As a result, the estimation relies on employees of opposite sex with matching job titles across databases. Machine learning methods have much less variance than matching and averaging methods, but both approaches lead to estimates across databases (large sex imbalance) that do not depart for values obtained within databases (no sex imbalance).
Merging the two will lead to loss of information. From an ontological point of view, the solution would be to create a new category, posh salad dressing. But maintaining a complete and consistent ontology, catering for all the edge cases, requires manual work each time new data is integrated. Should the necessity to merge entities be considered as a bug of analytic pipelines, rather than a feature? New tools that do not require exact matches can give more reliable analyses in the face of ambiguity, as illustrated by the estimation of salary gap across databases with sex imbalance (Fig. 4 ). Manually curating entity matching brings to the data a consistency that is good practice in production settings. Yet, as illustrated in our empirical study, favoring more advanced statistics down the line facilitates valid analysis.
B. SUPERVISION FACILITATES INTEGRATING DATA WITH AMBIGUITIES Representing uncertainty in relational systems helps tackling ambiguities  [36] ,  [55] ,  [56]  or curating data  [57]. However, extending relational data management to a general probabilistic framework is intrinsically hard. Indeed, unlike with the relational algebra, queries in a probabilistic database can suffer non-polynomial complexity  [58]. Approximate probabilities  [59] ,  [60]  or fuzzy logic and similarities  [61]  have better tractability. Yet how to weight similarities to best capture ambiguities is often a challenge in itself. Using supervised learning to answer a given statistical question alleviates the need for probabilistic models. In particular, many recent success rely on discriminative modeling using empirical risk minimization, as with deep learning  [62].