dataesr/scientific-paragraphs-categorization huggingface dataset:
The best-performing model from each team is picked for the top five teams, regardless of whether that model is submitted as model 1. All team methods are in gray while BLAST methods are in red and BLAST computational methods are in blue, see Table  3  for the description of the baselines shown in Additional file 1: Figure  S3. For more information regarding NK and LK designations, please refer to the Additional file 1 and the CAFA2 paper  [26]. After collecting these benchmarks, we performed two major deletions from the benchmark data. Upon inspecting the taxonomic distribution of the benchmarks, we noticed a large number of new experimental annotations from Candida albicans. After consulting with UniProt-GOA, we determined these annotations have already existed in the Candida Genome Database long before 2018 but were only recently migrated to GOA.
After the removal, the most frequent annotations became of depth 5 (Additional file 1: Figure  S15A ). In BPO, the most frequent annotations are of depth 5 or more, indicating a healthy increase of specific GO terms being added to our annotation database. In CCO, however, most new annotations in our benchmark set are of depths 3, 4, and 5 (Additional file 1: Figure  S15 ). This difference could partially explain why the same computational methods perform very differently in different ontologies and benchmark sets. We have also calculated the total Fig. 11  CAFA participation has been growing. Each principal investigator is allowed to head multiple teams, but each member can only belong to one team. Each team can submit up to three models Additional analyses were performed to assess the characteristics of the benchmark set, including the overall information content of the terms being annotated.
The two assessment modes from CAFA2 were also used in CAFA3. In the partial mode, predictions were evaluated only on those benchmarks for which a model made at least one prediction. The full evaluation mode evaluates all benchmark proteins, and methods were penalized for not making predictions. Evaluation results in Figs. 3  and 4  are made using the full evaluation mode. Evaluation results using the partial mode are shown in Additional file 1: Figure  S2. Two baseline models were also computed for these evaluations. The Naïve method assigns the term frequency as the prediction score for any protein, regardless of any protein-specific properties. BLAST was based on the results using the Basic Local Alignment Search Tool (BLAST) software against the training database  [52]. A term will be predicted as the highest local alignment sequence identity among all BLAST hits annotated from the training database.
These annotations not only represent new biological discoveries, but simultaneously serve to provide benchmark data for rigorous method evaluation. CAFA3 and CAFA-π feature the latest advances in the CAFA series to create advanced and accurate methods for protein function prediction. We use the repeated nature of the CAFA project to identify certain trends via historical assessments. The analysis revealed that the performance of CAFA methods improved dramatically between CAFA1 and CAFA2. However, the proteincentric results for CAFA3 are mixed when compared to historical methods. Though the best-performing CAFA3 method outperformed the top CAFA2 methods (Fig. 1 ), this was not consistently true for other rankings. Among all 3 CAFA challenges, CAFA2 and CAFA3 methods inhabit the top 12 places in MFO and BPO.
Between CAFA2 and CAFA3, the performance increase is more subtle. Based on the annotations of methods (Additional file 1), many of the top-ranking methods are improved versions of the methods that have been evaluated in CAFA2. Interestingly, the top-performing CAFA3 method, which consistently outperformed the methods from all past CAFAs in the major categories (GOLabeler  [23] ), utilized 5 component classifiers trained from different features; those included GO term frequency, sequence alignment, amino acid trigram, domains, motifs, and biophysical properties. It performs best in the Molecular Function Ontology, where sequence features perform best. Another method which did not participate in CAFA3 yet seems to perform well under CAFA parameters is NetGO  [58] , which utilizes the information from STRING, a network association database  [59]  in addition to sequence information.
Taken together, the strong predictive performance of mRNA co-expression data (Figs. 9 and 15) leads us to hypothesize that including more varied sources of data can lead to additional large improvements in protein function prediction. We are looking forward to testing this hypothesis in future CAFA challenges. It should be noted that CAFA uses both F max and S min. 3  for description of the baselines F max 's strength lies in its interpretability, as it is simply the maximum F 1 given for each model. At the same time, precision/recall-based assessment does not capture the hierarchical nature of ontologies or the differences in information content between different GO terms. For that reason, we also use the S min score which incorporates information content, but is somewhat less interpretable than F max and less robust to the problems of incomplete annotation  [60, 61].
For CAFA evaluations, we have included both Naïve and sequence-based (BLAST) baseline methods. For the evaluation of P. aeruginosa screen results, we were also able to include a gene expression baseline from a previously published compendium  [33]. Intriguingly, the expressionbased predictions outperformed the existing methods for this task. In future CAFA efforts, we will include this type of baseline expression-based method across evaluations to continue to assess the extent to which this data modality informs gene function prediction. The results from the CAFA3 effort suggest that gene expression may be particularly important for successfully predicting term-centric biological process annotations. The primary takeaways from CAFA3 are as follows: (1) genome-wide screens complement annotation-based efforts to provide a richer picture of protein function prediction; (2) the best-performing method was a new method, instead of a light retooling of an existing approach; (3) gene expression, and more broadly, systems data may provide key information to unlocking biological process predictions, and (4) performance of the best methods has continued to improve.
Review history The review history is available as Additional file 2. Sea", co-financed by the  Croatian Government  and the  European Regional Development Fund  ( KK. 01. 1. 1. 01. 0002 ). The work of SK was funded by  ATT Tieto käyttöön  grant and  Academy of Finland. JB and HM acknowledge the support of the  University of Turku , the  Academy of Finland  and  CSC -IT Center for Science Ltd. TB and SM were funded by the  NIH  awards  UL1 TR002319  and  U24 TR002306. The work of CZ and ZW was funded by the  National Institutes of Health   R15GM120650  to ZW and start-up funding from the  University of Miami  to ZW. The work of PWR was supported by the  National Cancer Institute of the National Institutes of Health  under Award Number  U01CA198942. PR acknowledges  NSF  grant  DBI-1458477. PT acknowledges the support from  Helsinki Institute for Life Sciences.
The work of AJM was funded by the  Academy of Finland  (No. 292589 ). The work of FZ and WT was funded by the  National Natural Science Foundation of China  ( 31671367 ,  31471245 ,  91631301 ) and the  National Key Research and Development Program of China  ( 2016YFC1000505 ,  2017YFC0908402 ]. CS acknowledges the support by the  Italian Ministry of Education, University and Research (MIUR)   PRIN 2017  project  2017483NH8. SZ is supported by the  National Natural Science Foundation of China  (No. 61872094  and No. 61572139 ) and  Shanghai Municipal Science and Technology Major Project  (No. 2017SHZDZX01 ). PLF and RLH were supported by the  National Institutes of Health   NIH R35-GM128637  and  R00-GM097033. JG, DTJ, CW, DC, and RF were supported by the  UK Biotechnology and Biological Sciences Research Council  ( BB/N019431/1 ,  BB/L020505/1 , and  BB/L002817/1 ) and Elsevier.
The work of IK and AB was funded by  Montana State University  and  NSF   Advances in Biological Informatics program  through grant number  0965768. BR, TG, and JR are supported by the  Bavarian Ministry for Education  through funding to the  TUM. The work of RB, VG, MB, and DCEK was supported by the  Simons Foundation ,  NIH NINDS  grant number  1R21NS103831-01  and  NSF  award number  DMR-1420073. CJJ acknowledges the funding from a  University of Illinois at Chicago (UIC) Cancer Center  award, a  UIC College of Liberal Arts and Sciences Faculty Award , and a  UIC International Development Award. The work of ML was funded by  Yad Hanadiv  (grant number  9660 /2019 ). The work of OL and IN was funded by the  National Institute of General Medical Science of the National Institute of Health  through  GM066099  and  GM079656.
Funding The work of IF was funded, in part, by the  National Science Foundation  award  DBI-1458359. The work of  CSG  and AJL was funded, in part, by the  National Science Foundation  award  DBI-1458390  and  GBMF 4552  from the  Gordon and Betty Moore Foundation. The work of DAH and KAL was funded, in part, by the  National Science Foundation  award  DBI-1458390 ,  National Institutes of Health   NIGMS P20 GM113132 , and the  Cystic Fibrosis Foundation   CFRDP STANTO19R0. The work of AP, HY, AR, and MT was funded by  BBSRC  grants  BB/K004131/1 ,  BB/F00964X/1  and  BB/M025047/1 ,  Consejo Nacional de Ciencia y Tecnología Paraguay (CONACyT)  grants  14-INV-088  and  PINV15-315 , and  NSF Advances in BioInformatics  grant  1660648. The work of JC was partially supported by an  NIH  grant ( R01GM093123 ) and two  NSF  grants ( DBI 1759934  and  IIS1763246 ).
Availability of data and materials Data repository: A data repository providing all additional data, analyses, and all anonymous prediction results for all methods are available at https:// figshare. com/articles/Supplementary_data/8135393/3 [63]. Code: The assessment software used in this paper is available under GNU-GPLv3 license on GitHub in both Matlab  [64]  (https://doi. org/10. 5281/zenodo. 3403452) and Python  [65] (http://doi. org/10. 5281/zenodo. 3401694).
Acknowledgements The authors thank  Caroline Filippi  for help with histology;  Mathieu Pasquet ,  Ludovic Petit ,  Susanne Reichinnek  and  Robert Martinez  for technical assistance;  David Dupret ,  Pierre-Pascal Lenck-Santini ,  Vincent Hok ,  Francesca Sargolini ,  Michae ¨l Zugaro  and members of the  Epsztein lab  for useful discussions;  Bryan Souza  and  Adriano Tort lab  for comments on a preprint version of this article; the animal facility, administrative and imaging platforms of INMED for support. This study was supported by  INSERM , a rising star grant from of the A* MIDEX  project (n˚ANR- 11-IDEX-0001-02 ) funded by the  «Investissements d'Avenir» French Government program  (to JE), by the  European Research Council  under the European Community's  Seventh Framework Program  ( ERC-2013-StG-338141_Intraspace  to JE and  ERC-2013-CoG-615699_NeuroKinematics  to DR) and by the  'Agence National de la Recherche ' (ANRJCJC to JK).
Data acquisition and pre-processing The position of the animal in the virtual maze was digitalized by the virtual reality controlling computer (Phenosys) and then sent to a digital-analog card (0-4. 5V, National Instrument Board NI USB-6008) connected to the external board (I/O Board, Open Ephys) of a 256 channels acquisition board (Open Ephys). Neurophysiological signals were acquired continuously on a 256-channels recording system (Open Ephys, Intan Technologies, RHD2132 amplifier board with RHD2000 USB interface board) at 25,000 Hz. Spike sorting was performed semi-automatically using KlustaKwik  (Rossant et al. , 2016 ; https://github. com/klusta-team/klustakwik). Clusters were then manually refined using cluster quality assessment, auto-and cross-correlograms, clusters waveforms and similarity matrix (Klustaviewa, Rossant et al.
HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Large expert-curated database for benchmarking document similarity detection in biomedical literature search.
For example, the PubMed (http://www. ncbi. nlm. nih. gov/ pubmed) database operated by the US National Center for Biotechnology Information at the US National Library of Medicine (NLM), has seen a sustained annual growth rate of nearly 5% for the past 4 years; from 25. 2 million records in 2015  (57) , 26. 4 million records in 2016  (58) , 27. 5 million records in 2017  (59)  and is now approaching 29 million records in 2018 as of the latest report  (66). The overall success of this classic approach depends on finding optimal keyword combinations as it is key for ensuring the precision and completeness (i. e. sensitivity) of the search results. The aforementioned scenario is an illustration of an 'informational' search goal  (8, 64) , essentially an exploratory request for discovering the most relevant resources to 'find out' about a particular topic.
Aside from these content-based features, seed articles may also provide additional metadata including the list of authors, publication venue and date, citation information and potentially manually curated categorical information such as assigned Medical Subject Headings (MeSH) descriptors  (52). Most importantly, all of these features are extracted and processed automatically without requiring any user input or intervention. Thus, for an article-based search, the search engine will accept as input either the selection of an existing document from its database or a user uploaded document in its entirety, in order to generate the list of results. Article-based relevance searches, often referred to as research paper recommender systems, is an active area of research with more than 100 different approaches published since 1998  (7).
Representative examples of commonly used methods used for calculating the similarity between text documents are Okapi Best Matching 25 (BM25)  (39, 40) , Term Frequency-Inverse Document Frequency (TF-IDF)  (65)  and PubMed-Related Articles (PMRA)  (49). These and other decade-old methods remain the backbone behind real-world recommender systems such as PubMed (http://www. ncbi. nlm. nih. gov/pubmed), ResearchGate (http://www. researchgate. net) and CiteULike (http://www. citeulike. org). Numerous newly developed methods are not yet translated into practice because it is unclear whether they have improved over existing methods  (7). Improvements in paper recommender systems can be measured by 'user studies', 'online evaluations' or 'offline evaluations'. User studies aim to quantify user experience by interacting directly with explicit testers of the system; online evaluations are a more passive approach wherein user activity is silently monitored and recorded for assessment, and offline evaluations are akin to traditional benchmarking processes where algorithmic performance measures are evaluated on appropriate datasets.
Previous work has focused on building gold standards of query-to-document relevance in biomedical literature. Queries are generally defined as some topic comprised as a natural language statement, a question or a set of keywords conveying a specific   32 ) collection, the first TREC Genomics track  (33, 73)  (the only applicible biomedical-focused track within TREC), the BioASQ (72) competition and the bioCADDIE (13) dataset. Although these datasets are suitable for benchmarking traditional information retrieval, they do not consider the similarity between document pairs as a whole. Re-purposing query-to-document datasets to the document-to-document similarity problem has been attempted; for example, adaptation of the 2004/2005 TREC Genomics data  (34, 35)  has been evaluated by previous studies  (11, 49, 74).
However, a pair of documents may both be related to a certain topic but only loosely related to each other as complete units. Constructing a gold standard dataset of sufficient size for offline evaluations is not a trivial endeavor. In the context of query-to-document relevance, a recent study concluded that 'there is unfortunately no existing dataset that meets the need for a machine-learning-based retrieval system for PubMed, and it is not possible to manually curate a large-scale relevance dataset'  (24). We propose here that relevance labels between document pairs in the documentto-document context can be 'crowd-sourced'. This ideology has a number of successful applications; for example, computational resources are crowd-sourced by the BIOINC (1) distributed computing system for the Folding@Home (47) and Rosetta@Home (28) projects, and human problem solving resources are crowd-sourced for 'distributed thinking systems' as in the Foldit  (14, 28)  project.
Here, we have established the RElevant LIterature SearcH (RELISH) consortium of 1500+ scientists from 84 countries around the world. The consortium annotated over 180,000 document-document pairs indexed by PubMed, nearly 400 times larger than the single other human annotated data collection we could find with just 460 annotations by 90 authors  (50). Analysis of the collected data indicates diversity and consistency of annotations across different levels of research experience as well as its usefulness in benchmarking different documentdocument comparison methods. Futhermore, these data could be utilized in future work to train deep machine learning models that may substantially increase relevancy of recommendations compared to existing methods.
APSE: document corpus The seed article corpus used by the APSE server was extracted from the biomedical literature database hosted by PubMed. The corpus database was constructed using the 2017 baseline downloaded from PubMed's public FTP server (ftp://ftp. ncbi. nlm. nih. gov/pubmed), followed by incremental synchronization with daily update releases. Only articles with available title and abstract metadata were included in the corpus, no other selection criteria were imposed. As of 3 July 2018, this collection contained 18,345,070 articles, from which participants selected seed articles. The candidate article corpus, from which the APSE server generated recommendations that were presented to participants, was a subset of the seed article corpus. This subset included only recent articles that were published within the past decade.
The primary justification for this was to increase participant motivation with potential discovery of recent work within their field, considering the decline of participation willingness with increasing publication age as speculated previously  (77). As of 3 July 2018, this candidate article subset contained 8,730,584 articles. All raw article texts were pre-processed into usable indexing elements (tokens) by the following pipeline. We first applied Lucene's 'UAX29URLEmailTokenizer', which follows Unicode Text Segmentation rules  (16)  to divide the text (while also tagging detected URLs or email addresses). Next, possessives (trailing apostrophe + s) were removed from each token, and any single character, numeric, URL or email tokens were removed. Stop-word tokens were filtered according to a list of 132 official MEDLINE stop words retrieved from a previous source  (6).
RELISH: participant recruitment To achieve the goal of a large dataset facilitating future deep machine learning-based method development, we established the RELISH consortium of biomedical researchers who have annotated studies potentially related to one or more seed articles that they had authored or were interested in. To maximize the quality and efficiency of annotation, we encouraged participants to use articles that they had authored as the seed articles. All participants contributed voluntarily. Only submissions with annotations completed for all 60 recommended candidate articles per seed article were accepted. Our recruitment strategy involved two major phases: the first was internal referrals, personal invitations, social media posts and a correspondence letter  (9) ; the second was directly contacting authors of papers published in 2018.
Infrastructure The database search infrastructure was powered by Elasticsearch [https://github. com/elastic/elasticsearch; using Apache Lucene (https://github. com/apache/lucene-solr)]. The web interface was designed using React (https://github. com/facebook/react), web services were built using Apache (https://github. com/apache/httpd) and Python (https:// github. com/channelcat/sanic) and data were organized using MongoDB (https://github. com/mongodb/mongo).
Annotations covered diverse research areas Figure  3  shows a word-cloud distribution of MeSH descriptor frequencies for all annotated documents, normalized by baseline frequencies in all PubMed documents. There seems to be a slight over-representation of publications on highthroughput 'Omics' technologies (e. g. genomics, genomewide association study, nucleotide sequencing, proteomics). It is possible that researchers in the field of bioinformatics/ computational biology were particularly eager to participate in the study of annotating documents as they are probably the ones seeing the most benefit in having a large annotated dataset of high quality. It may also be due to the first participant recruitment phase via internal referrals and personal invitations were somewhat enriched with scientists working in the bioinformatics/computational biology areas.
Consistency according to method performance Our definition of relevance, partial relevance and irrelevance is subject to different interpretations by different individuals. The perceived efficacy of document recommendation systems depends on the expected agreement between the subjective opinions of different individuals. To examine if such agreement exists, we compared the performance of the three baseline methods on annotations made by scientists of different experience levels, annotations of the same articles by different individuals, annotations across different research fields and durations of time spent annotating. To carry out these comparisons, seed articles for the benchmark datasets were selected from RELISH-DB  according to the following conditions. First, we removed 44 seed articles that PMRA provided fewer than 20 article recommendations for; this was to ensure a fair comparison with the other baseline methods that will always have 20 recommendations per seed article.
Next, 186 seed articles without any positive or negative samples were removed as assessment of discrimination is impossible when all candidate articles belong to either the positive or negative class. Finally, we set aside 154 duplicate annotations of the same seed article by different participants ('D154'), and 400 randomly selected seed articles as an independent test set ('T400'), to be analyzed separately. This led to a total of 2233 seed articles ('ALL2233') available for evaluation. From this set we derived an 'NR' (non-redundant) set, comprised of a single seed article from each unique participant to avoid potential bias against specific annotators. In this dataset, for participants with multiple seed article contributions, a seed article was selected at Downloaded from https://academic.