dataesr/scientific-paragraphs-categorization huggingface dataset:
oup. com/database/article-abstract/doi/10. 1093/database/baz085/5608006 by guest on 18 December 2019 Table  2. Overall evaluation set performance results for the PMRA, BM25 and TF-IDF methods. 'ALL' includes all seed articles from all participants, and 'NR' includes only one seed article from each participant, corresponding to the 'ALL2233' and 'NR1220' sets, respectively. Performance was measured using MCC, AUC (ROC), MRR and P@N, given as (mean ± stdev). random. This 'NR' set had 1220 seed articles ('NR1220') in total, less than the total number of unique participants due to aforementioned exclusions.
Data Records A complete dump of collected annotation data as at 27 July 2018 was deposited to a figshare repository. The dataset is released without copyright, available at https:// figshare. com/projects/RELISH-DB/60095, under the CC0 license. All data records were stripped of personally identifiable information and then converted to JSON format  (41). Record fields include a unique identifier ('uid'), Pub-MedID of the seed article ('pmi'), annotator experience level ('experience'), whether the annotator was an anonymous or registered user ('is_anonymous') and annotator response ('response') containing the lists of candidate article PubMedIDs corresponding to the assigned degree of relevance (i. e. one of 'relevant', 'partial' (somewhat-relevant) or 'irrelevant').
Usage Notes To make RELISH collections useful for future method developers, we have established a data server at https:// relishdb. ict. griffith. edu. au. This data server has three modules for data annotation, data retrieval and method testing. Data annotation: data annotation functions exactly as the APSE server. As shown in Figure  1 , users can search for publications related to any article of interest, with the option to voluntarily annotate article relevance. New submissions will be automatically saved for inclusion in the next database version. Data retrieval: we have made available several pre-built datasets (the evaluation sets within this work) on our data server. These include the complete versions of the 'ALL2233' and 'NR1220' sets, as well as copies of these sets broken down by annotator experience level.
In addition, the three single-annotator sets ('A1', 'A2' and 'A3') are also available. Furthermore, we also provided an option to allow users to generate a dataset according to various pre-defined parameters (dataset size, experience level, single seed article or multiple seed article annotators and cut-offs for the number of candidate articles labelled as relevant or irrelevant per seed article). As shown in Figure  11 , pre-compiled datasets can be inspected by starting at  (1). Alternatively, user-defined datasets may be generated by starting at  (2). After generating a new dataset, or clicking on an existing one, the dataset view page will be displayed as shown in stage  (3). Here, dataset details including size, number of positive and negative pairs, and any custom parameters that were set will be given.
For each dataset, we make available the respective subsets of article metadata, annotation data and an example result file in the format the automatic evaluation will accept, as shown in stage  (4). The metadata includes the PubMedID, title and abstract for each article involved in the dataset. Because we are using publicly licensed PubMed metadata, we allow for direct download of this metadata in JSON format  (41). Relevance data are provided in TREC relevance format  (56). Method developers can divide downloaded datasets into training and independent test sets according to their own specific needs. Method testing: to facilitate comparison and avoid overtraining, we set aside a blind-test set for critical assessment of method performance. This selection of 400 randomly selected NR seed articles have their annotation data withheld from the rest of the database.
The dataset has two variants ('BT1' where partially relevant articles are considered as 'positive' samples, whereas in 'BT2' partially relevant articles are considered as 'negative' samples). Baseline method performances for this set are shown in Table  8. Overall, the results are consistent with what was observed for the complete dataset. We envision that method developers can download pre-built or user-defined datasets for training and independent tests. Subsequently, they can download 'BT1' or 'BT2' to provide additional tests by submitting the result file in TREC result format  (61)  as shown in stage (5) of Figure  11. Evaluations will be automatically executed and a list of recorded evaluations for the dataset will be shown in stage  (6). After uploading a result file for a new evaluation, or clicking on an existing evaluation in the list, the evaluation results will be displayed: firstly, a summary of metrics over the dataset as a whole is given by stage  (7)  and secondly, the metrics are broken down into a per seed article query basis for inspection in stage  (8).
Moreover, the conversion process between an information need to an effective query is often a potential source of difficulty  (19, 37). While search engines often offer 'advanced' operators for more elaborate queries, the   6 ) uploaded evaluations-results of uploaded result file evaluations for the respective dataset will be presented here;  (7)  overall evaluation view-provides a performance summary of the dataset as a whole; (8) detailed evaluation view-performance details are broken down on a per-query basis. Table  8. Performance results of the blind test set of 400 randomly selected articles for the PMRA, BM25 and TF-IDF methods. 'BT1' considered partially relevant articles as 'positive', and 'BT2' considered partially relevant articles as 'negative'. Metrics include MCC, AUC (ROC), MRR and P@N, given as (mean ± stdev).
It is the developers of these types of recommendation systems we anticipate to benefit the most from our database. This work is not the first to propose the use of a seed article (manuscript) as a query instead of keywords  (10, 27, 37, 46, 55, 71). As stated in  (31) , the following scenarios illustrate potential applications of a system built upon this ideology: editors or reviewers wishing to explore a subject matter of which is not their specialty (e. g. to find other potential peer reviewers), researchers simply wishing to find related studies to a paper of interest (e. g. for the purpose of locating citations or venues for their work), researchers who are pursuing a new area of which they have little existing knowledge (e. g. a student following up on a paper given by their supervisor) and researchers wishing to keep up with latest developments in their field based on their previous publications.
Finally, we must stress that the basis of our use case here is a complementary alternative to keyword search, it is not intended as a replacement. An accurate keyword-less recommendation system is sure to benefit both expert and nonexpert users, especially in this era of information overload. In addition, this is more than a database exclusively for seed article-based search. It can be used to train keywords or sentence-based search by examining if keywords or the title from one article would automatically lead to the other articles. That is, a method can be trained for title/sentencebased and title/abstract based search on this benchmark.
Discussion Our work represents a community-wide effort to establish a database of document relevance that is suitable for machine learning. It should be noted that our APSE system facilitating the annotation process was not intended as a standalone search engine. Rather, it was developed and employed exclusively for data collection. Nevertheless, using three different methods appeared to provide a more complete search compared to systems powered by a single technique such as TF-IDF in JANE  (67)  or PMRA in PubMed  (49). This is consistent with comments from many users who found interesting articles previously unknown to them. More than 1500 scientists around the world participated in the RELISH consortium and annotated the relevance for over 180,000 document pairs that represent diverse research fields, covering 76% of all PubMed MeSH descriptors without clear bias (Figure  3 ).
These annotations are of high quality as more than 90% of the article pairs were annotated by original authors and 63% by scientists with 5 or more years research experience post-PhD. While the majority annotated a single seed article, a few dedicated researchers annotated more than 50 seed articles, and some independently annotated the same seed articles. Together, the resulting dataset provides the largest manually annotated benchmark for detecting document similarity in biomedical literature search. A number of potential systematic biases were considered during the construction of this database. Potentially affected aspects of our methodology primarily include how the documents presented to annotators were selected and how annotators themselves were selected. We discuss here the biases we were able to identify and explain the rationale behind how they were dealt with.
This task requires highly specialized experts in respective sub-fields to make that determination. In other words, the more sub-field experience of an annotator, the more accurate annotations will be made. Although non-PhD researchers are underrepresented in their participation of database construction, it will not affect the overall accuracy of annotations as we have demonstrated that there is no bias according to the experience level of annotators and across different fields of researchers. In fact, using expert annotations will ensure the production of relevant articles for all non-PhD researchers, clinicians and the general public in their searches. If a subjective bias exists, one would expect that the performance of the three baseline methods (PMRA, BM25 and TF-IDF) would have certain systematic trends.
Furthermore, high consistency in the dataset was also suggested by the relatively narrow distribution of thresholds that were used to maximize the MCC for each seed article (Figure  9 ). This indicated that relevance can be defined across different experience levels or research fields. The results collectively indicate that despite the presence of any subjective bias, a consensus exists regarding what is actually relevant literature for the majority of researchers, despite differences in research areas and experience levels. High-quality data collected by the RELISH consortium offered an unprecedented opportunity to compare the three baseline methods (PMRA, BM25 and TF-IDF). Somewhat surprisingly, all three methods performed similarly, with TF-IDF presenting a slight edge. Our finding is somewhat different from previous studies  (22, 51, 68)  where the classic TF-IDF was often considered inferior to methods developed afterwards.
This result highlights the importance of having a large benchmark dataset. Moreover, all three methods evaluated here performed only moderately well with an average AUC of around 0. 7 and average MCC of around 0. 5. In addition, different methods did capture different sets of relevant articles (Figure  10 ). This further indicates the necessity of large benchmark evaluation datasets, significant room exists for improving literature search with future method development. Encouraging method development was the principal motivation behind the assembly of this database. A recent survey of research paper recommendation systems revealed the urgent need for a common evaluation framework (4), as current method evaluation results are rarely reproducible (3). This work was inspired by large human-annotated databases like ImageNet  (17) , where labelled images have promoted method development and helped to revolutionize object detection and image classification  (45).
Our database should allow novel methods to be developed based on modern deep machine learning techniques to address the major deficiencies of current methods. Additionally, the benchmark datasets as a result of our database will enable objective comparison between old and new methods, increasing the uptake of newer methods and their translation into practice. To assist unbiased evaluation, we have made the RELISH database freely accessible to the academic community (released under the CC0 license). Furthermore, we set aside 400 × 60 document-document pairs for blind prediction using the automatic evaluation by our database server at https://relishdb. ict. griffith. edu. au. We expect that dissemination of this manuscript will attract more scientists to contribute more annotations. A major struggle for us was participant recruitment, the first phase of our recruitment strategy (internal referrals, personal invitations, social media posts and a correspondence letter) ultimately failed to attract enough contributions to establish a meaningful dataset.
Filipp 93, 352 , Hannes Klump 94 , Lukasz Kurgan 95 , Simon S. Smith 96 , Olivier Terrier 97 , Neil Tuttle 98 , David B. Ascher 99 , Sarath C. Janga 100 , Leon N. Schulte 101 , Daniel Becker 102 , Christopher Browngardt 103 , Downloaded from https://academic. oup. com/database/article-abstract/doi/10. 1093/database/baz085/5608006 by guest on 18 December 2019.
Heavy use of cannabis (marijuana) has been associated with decreased semen quality, which may reflect disruption of the endocannabinoid system (ECS) in the male reproductive tract by exogenous cannabinoids. components of ecS have been previously described in human spermatozoa and in the rodent testis but there is little information on the ECS expression within the human testis. In this study we characterised the main components of the ecS by immunohistochemistry (iHc) on archived testis tissue samples from 15 patients, and by in silico analysis of existing transcriptome datasets from testicular cell populations. The presence of 2-arachidonoylglycerol (2-AG) in the human testis was confirmed by matrix-assisted laser desorption ionization imaging analysis. Endocannabinoidsynthesising enzymes; diacylglycerol lipase (DAGL) and N-acyl-phosphatidylethanolamine-specific phospholipase D (NAPE-PLD), were detected in germ cells and somatic cells, respectively.
In this reference-providing study, we used immunohistochemistry (IHC) on human testis samples from tissue archives, and in silico analysis of the ECS transcripts and IHC data in existing datasets. For validation of the antibodies, we examined samples of human extra-testicular tissues, including epididymis. We also performed quantitative RT-PCR (qPCR) validation of the expression of ECS-receptors and their non-canonical isoforms. In addition, we investigated the presence of endocannabinoids AEA and 2-AG in testis tissue by matrix assisted laser desorption ionization (MALDI) imaging analysis. Taken together the data indicate that the ECS is present in the human testis and is likely to be directly involved in testicular function, particularly in regulation of spermatogenesis.
Results Localisation of the ECS components in the adult testis at the protein level. We investigated by IHC the spatial organisation of the ECS components necessary for the ECS machinery to function. The IHC results in the testis are summarised in Table  1 , separately for each cell type, and representative images are shown in Figs  1 2 3. The antibodies, listed in Table  2 , were selected and validated as described in Materials & Methods and in Supplementary Information. Additional validation of all antibodies included testing in other tissues, including epididymis and efferent ducts (the tissues of importance for testis function) and several other tissue types, with known expression levels of ECS. These data are summarised in Supplementary Table  S1 , with representative examples shown in Supplementary Fig.
Note that the RNA-seq data for spermatocytes did not distinguish any specific stages, and that the expression of the receptor RNA isoforms was studied by qPCR. Abbreviations: Spg: spermatogonia, Spc: spermatocytes, P-Spc: pachytene spermatocytes, Spt: spermatids, Peritub: peritubular cells, BV: blood vessels. N: nuclear staining, C: Cytoplasmic staining, *localised intra-nuclear reaction. The IHC scoring is explained in the Methods section, '-' denotes no staining, 'n. a. ' -cell type not available for transcriptional analysis. RNA-seq data are classified as strongly/ moderately expressed (++++/+++/++), detectable (+) or undetectable (-). cells. Additionally, ABHD2 transcripts were readily detected in all cell types, with highest expression level in round spermatids. The expression levels of CNR1 and CNR2, however, were below detection threshold in all cell types (maximum expression of 0.
Importantly, the purification and sequencing of PCR fragments confirmed the specificity of the expression signals for all four receptor isoforms. A summary of the data extracted from the RNA-seq human dataset is shown for each available cell type in Table  1 , together with the protein data. Qualitative detection of endocannabinoid 2-AG in human testis tissue. After detecting the presence of EC-synthesising enzymes, we then investigated whether the most common endocannabinoids, AEA and 2-AG are produced within the testis. By MALDI imaging methodology, we first analysed differentially distributed lipids in main testicular compartments. The PC (36:2) lipid (PC = phosphatidylcholine) detected as the sodium adduct at m/z 824. 5566 was found to be associated with interstitial cells, while the PC (38:5) lipid detected as the sodium adduct at m/z 846.
Hence, we hypothesise that MGLL in Sertoli cells may be needed to metabolise 2-AG produced in post-meiotic germ cells and shed within residual bodies. The proposed constant synthesis of 2-AG within sperm cell membranes to protect from premature sperm capacitation  21  would also require presence of a diacylglycerol lipase, and its transcription during earlier stages of spermatogenesis. Indeed, the analysis of our RNA dataset revealed the presence DAGLA and DAGLB transcripts in both germ cells and somatic cells. The protein data, however, indicated the presence of DAGLA protein mainly in cell nuclei, and no suitable antibody was available for DAGLB. The public database, Human Protein Atlas 29 reported the presence of DAGLA protein mainly in Leydig cells, hence more work is needed to firmly establish the cell type producing 2-AG within the testis.
NAPEPLD transcripts were present in these somatic cell types but were also detected in round spermatids. One of the antibodies screened in the preliminary phase of the study detected also NAPE-PLD in round spermatids, but only in the nuclei. This enzyme was not studied previously in the human testis tissue, only in ejaculated spermatozoa, where NAPE-PLD protein was described in the post-acrosomal region and mid-piece  2, 18. Data available in public databases are also somewhat contradictory concerning this enzyme, so further studies concerning AEA synthesis in the testis are needed. By contrast, the existing data agree on the robust expression of the AEA-metabolising enzyme, FAAH, in late spermatocytes and in spermatids. This is consistent with a hypothesis that the level of AEA or another natural FAAH substrate must be kept very low during meiosis and post-meiotic maturation of germ cells.
The antibodies were selected based on the previous performance in IHC of other tissues reported in the literature. Most of the commercial antibodies underwent extensive validation of their specificity by the suppliers, including Western blotting, and in all Sigma antibodies also IHC testing in a range of tissues, available in the public database Human Protein Atlas (www. proteinatlas. org), which also reports the presence of transcripts  29. Several different CNR1 antibodies were tested before selecting the most robust one (Abcam) for the final study (Supplementary Table  S2 ). Three of the four tested CNR1 antibodies demonstrated unusual nuclear localisation of staining in the testis. The selected antibody was additionally validated by Western blotting with and without a blocking peptide, and pre-absorption step in the IHC protocol (Supplementary Figures  S2  and S3 ).
The antibody in 1:400 dilution was thoroughly mixed with a 10-fold excess of the CNR1 blocking peptide and then applied on tissue sections. The rest of the staining protocol was as described above. The IHC with the peptide competition was repeated in three independent experiments, and a representative example is shown in Supplementary Fig. S3. IHC evaluation and scoring: Three investigators evaluated systematically all IHC slides. The sections were first examined manually on a Nikon Microphot-FXA microscope, then were scanned on a slide scanner, NanoZoomer 2. 0 HT (Hamamatsu Photonics, Herrsching am Ammersee, Germany) and analysed using the software NDPview version 1. 2. 36 (Hamamatsu Photonics). The observed IHC staining pattern and intensity was classified according to a pre-defined arbitrary scoring system: +++, strong staining in all cells of a given type in the sample; ++, staining in nearly all cells of a given type; ++/+, strong staining prevalent, but some weakly stained cells also visible; ++/-, strong staining present but negative cells also present; +/++/-, heterogeneous pattern with a mixture of strongly positive, weakly stained, and negative cells; +/++, majority of cells weakly stained, but focal strong staining present; +, weak staining overall; +/-, weak staining in limited areas; -/+, weak staining in single cells; -, no staining.
RNA-sequencing data processing and transcript quantification. Human RNA-seq data from testicular cells  26  were pre-processed as previously described  43, 44. Briefly, paired-end reads were first mapped on the human genome (hg19 release) using TopHat (2. 0. 10)  45. The RefSeq transcript annotation (GTF format)  46  was downloaded from the UCSC genome browser (https://genome. ucsc. edu/; 6 February 2017) and the abundance of corresponding genes in each sample was quantified using the Cufflinks suite  45  in fragments per kilobase of exon model per million reads mapped (FPKM). All data are available at the ReproGenomics Viewer 47 ; (http://rgv. genouest. org/app/#/). Gene expression data for the ECS components are provided in Supplementary Table  S3. Quantitative pcR analysis. cDNAs were synthesized from the same RNA samples as those used for RNA-seq 26 , using the iScript ™ Reverse Transcription Supermix for RT-qPCR (Bio-Rad) according to manu- facturer's instructions.
Each gene was assessed in each sample in technical duplicates on 2 ng of cDNA using iTaq ™ Universal SYBR ® Green Supermix (Bio-Rad) and a CFX384 Touch ™ Real-Time PCR Detection System (Bio-Rad) with default program (95 °C for 3 min followed by 40 cycles of 95 °C for 10 s, 60 or 64 °C for 30 s) including a melting curve step (65 °C to 95 °C with a 0. 5 °C increment and a hold time of 5 s before reading plate). Previously published primers of GAPDH and RPLP0 were used for normalisation purposes  48  and results were analysed with the Bio-Rad CFX Manager ™ software. Resulting PCR products were purified with the NucleoSpin ® Gel 317 and PCR Clean-up kit (Macherey-Nagel) prior to sequencing using forward and reverse qPCR primers (Table  3  and Supplementary Figs  S4  and S5 ).
Statistical analysis. Normal distribution of the qPCR data was first assessed for each isoform with a Shapiro-Wilk test as well as with a D' Agostino's K-squared test. When normality was met (CB1 and CB1A), homogeneity of variances was further verified with the Brown-Forsythe test, prior to evaluating statistical differences with parametric tests, i. e. one-way ANOVA for global analysis of variances and unpaired Student's t-test for pairwise comparisons. When normality was not met (CB1B and CB2A), non-parametric Kruskal-Wallis and Mann-Whitney U tests were used for global analysis of variances and for pairwise comparisons, respectively.