dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements Authors gratefully acknowledge the research participants who contributed to this project and project partners who have supported research operations. We acknowledge with respect the Traditional Custodians of the lands and waterways where we live and work, the Wurrundjeri people of the Kulin Nations and Clans of Dja  Dja Wurrung country in Victoria ; the Gadigal and Bidjigal people of the  Eora Nation in New South Wales ; the people of Ngunnawal in the  Australian Capital Territory ; and the  Kaurna people of South Australia.
Photometry and Physical Properties of Observed Galaxies We used the COSMOS2015 photometric catalog for the ∼2 deg 2 COSMOS field, which includes photometry of more than 0. 5 million galaxies in over 30 bands as well as their estimated stellar mass and photometric redshift values  (Laigle et al. 2016). The photometric redshifts and stellar masses were estimated using the SED fitting code LEPHARE 26   (Arnouts et al. 2002; Ilbert et al. 2006)  to fit multiwavelength observations of galaxies. The fitting process is based on the  Chabrier (2003)  initial mass function, two metallicities (solar and half-solar), emission line templates from  Ilbert et al. (2009) , attenuation curves from  Calzetti et al. (2000; Arnouts et al. 2013) , an exponentially declining and delayed star formation history, and stellar population synthesis models from  Bruzual & Charlot (2003).
2015; Santini et al. 2015). The median value of the average of all of these PDFs was then selected as the best stellar mass estimate for a given galaxy. The stellar mass measurements in the above photometric catalogs were based on the photometric redshifts of the galaxies. We require spectroscopic redshifts for our sample spectroscopic galaxy pairs and controls, which could be different from the photometric redshift of a given galaxy. Therefore, we re-measure the stellar masses of our pairs and controls using the SED fitting code Multiwavelength Analysis of Galaxy Physical Properties (MAGPHYS; da  Cunha et al. 2008 ) to fit the above-mentioned photometric observations of galaxies and their spectroscopic redshifts as described in detail in Section 4. 1. For the galaxies that have similar photometric and spectroscopic redshifts, our newly estimated stellar masses are consistent with the original stellar masses from the photometric catalogs.
Measurement of SFR We use the SED fitting tool MAGPHYS (da Cunha et al. 2008) for fitting the model SEDs to the photometric data points (far-UV-FIR band flux and flux-error values) to measure the SFR and stellar mass of galaxies. We choose MAG-PHYS for the SED fitting process as it self-consistently fits observations from UV to FIR based on an energy balance argument. It considers a combination of hot and cold dust grains as well as PAHs to estimate attenuation. MAGPHYS uses the  Bruzual & Charlot (2003)  stellar population libraries. For estimating the SFR of the spectroscopic pairs and their corresponding control galaxies, we use their spectroscopic redshifts. For the visually identified interactions and merger galaxies and their corresponding control galaxies, we use their photometric redshifts.
Support for this work was provided by NASA through grants HST-GO-13657. 010-A and HST-AR-14298. 004-A awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc. , under NASA contract NAS 5-26555. Support was also provided by NASA through grant NNX16AB36G as part of the Astrophysics Data Analysis Program. This work was also supported by start-up funds and the Dean's Research Initiation Grant fund from the Rochester Institute of Technology's College of Science. Spectral energy distribution fitting was performed using the computational resources and support from Research Computing Services at the Rochester Institute of Technology (Rochester Institute of Technology 2019). E. S. thanks the LSSTC Data Science Fellowship Program, which is funded by LSSTC, NSF Cybertraining grant No.
This work was supported by a  NASA   Keck PI Data Award , administered by the  NASA Exoplanet Science Institute. Some of the data presented herein were obtained at the  W. M. Keck Observatory , which is operated as a scientific partnership among the  California Institute of Technology , the  University of California , and the  National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the  W. M. Keck Foundation. Based in part on observations obtained at the  international Gemini Observatory  and processed using the Gemini IRAF package, a program of  NOIRLab , which is managed by the  Association of Universities for Research in Astronomy (AURA)  under a cooperative agreement with the  National Science Foundation  on behalf of the Gemini Observatory partnership: the  National Science Foundation  ( United States ),  National Research Council (Canada) ,  Agencia Nacional de Investigación y Desarrollo (Chile) ,  Ministerio de Ciencia, Tecnología e Innovación (Argentina) ,  Ministério da Ciência, Tecnologia, Inovações e Comunicações (Brazil) , and  Korea Astronomy and Space Science Institute (Republic of Korea).
The possibility of restricted strain diversity and the transient nature of infection with T. evansi in different transmission vectors, implies that the mechanisms for infection used by the parasite would be universal for all trypanosome spp. Parasites were irradiated at different doses ranging from 100 -250Gy so that they were temporarily unable to multiply but still capable of synthesizing many of the proteins that could potentially help the parasite establish disease in mice. Irradiated T. evansi parasites were subsequently observed in vitro before inoculation and challenge of mice. Doses that produced living and non-infectious parasites were then studied using a whole transcript gene microarray and compared to that of infectious parasites. Data was analyzed to identify genes and processes that are responsible for radiation damage repair and may possibly be involved in enabling the parasites to establish an infection in the mammalian host after recovery.
Mouse Bio-Plex Cytokine Assay A custom 10-plex cytokine assay (Bio-Plex: Bio-Rad, Hercules, USA) was used to quantify the plasma levels of 10 cytokines in mouse (Mo) plasma [interferon-gamma (IFN-g), tumor necrosis factor-alpha (TNF-a), interleukin-1 alpha (IL-1a), IL-1b, IL-4, IL-6, IL-10, IL-12 (p40), IL-12 (p70) and IL-13] as described by the manufacturer. Briefly, a 1 in 4 dilution of plasma collected from the different groups of mice during inoculation (days 0 to 28) was incubated with beads coupled to monoclonal antibodies specific for each component of the cytokine panel. Samples were washed before adding detection antibodies and developed for reading using the Bio-Plex ® 200 suspension array system. Absolute interleukin concentrations were calculated using Bio-Plex Manager ™ software and samples were analysed using a one-way ANOVA with Dunn's Multiple Comparison Test in GraphPad Prism 5 for each cytokine.
The final array designed contained the fully sequenced T. brucei genome as a backbone with selected T. evansi and T. congolense sequences that do not overlap with the T. brucei genome. Approximately 94. 9% of T. evansi genome is identical to the reference T. brucei genome  (62). A total of on average 9,300 whole gene transcripts from all three species were targeted with most having 8 to 25 probe pairs per gene. The array was produced using the 16 sample arrays per plate format. Microarray data produced from this experiment is available in the ArrayExpress database (http://www. ebi. ac. uk/arrayexpress) under accession number E-MTAB-11705.
whole transcript array before processing using the Gene Titan ® Multi-Channel (MC) instrument from Affymetrix. Intensity CEL files generated after hybridisation and scanning were analysed using Affymetrix ® Expression console software and files that passed all quality control parameters were converted into CHIP files that were subsequently interpreted using Affymetrix ® Transcriptome Analysis Console (TAC3) Software. Lists of genes were prepared using T. brucei annotation with figures on fold change (FC), ANOVA p-value and false discovery rate (FDR) adjusted p-value assigned to each gene described.
Gene Ontology Analysis Gene transcript abundance affected by irradiation at different doses after 6 hours and 20 hours were used to query TriTrypDB for gene ontology (GO) enrichment data  (63). GO enrichment terms identified were used together with fold changes calculated from the TrypMS array to calculate a Z score using tools from GOplot on R  (64). The Z score is a value that indicates whether the GO term associated with a set of genes [biological process (BP)/ molecular function (MF)/cellular components (CC)] is more likely to be decreased or increased and is calculated as Z = (updown)/ SQRT (Total) where up and down are the number of assigned transcripts up-regulated (logFC > 0) or down-regulated (logFC < 0) in the data set, divided by the square root of the total number of genes associated with the identified GO term.
Calculated Z scores versus -log adj p values were used to make combined bubble plots (with BP, MF and CC) for 100Gy, 140Gy, and 200Gy at 6 hours and 20 hours post-irradiation. A GO circle plot for the top 10 GO enrichment terms with the most significant p values was also plotted for 100Gy at 6 hours and 20 hours. GO chord plots that display the relationship between genes and GO terms was plotted for the ten most significant terms for 100Gy at 20 hours. Gene ontology analysis and visualization was performed using customized python codes and the GO plot manual (http:// wencke. github. io/).
Introduction Pretrained Language Models (PLMs), such as BERT  (Devlin et al. , 2019) , ELECTRA  (Clark et al. , 2020)  and T5  (Raffel et al. , 2020) , have become the core components for building highly effective ranking models. The success of PLMs is largely owed to the heavy pre-training on language modeling objectives on the one hand, and learning deeply-contextualized representations of input sequences using the transformer architecture  (Vaswani et al. , 2017)  on the other. Thanks to the fine-tuning strategy and the availability of large publicly-released training datasets, applying a PLM to document ranking is straightforward. Nogueira and Cho (2019)  was the first to propose a simple application of BERT to text ranking using fine-tuning on the large public MS MARCO  (Nguyen et al.
, 2016)  dataset. In this work, BERT was deployed as a relevance classifier trained to estimate the probability each document is "relevant" w. r. t a given query. Compared to the first wave of neural ranking models including DRMM  (Guo et al. , 2016) , DUET  (Mitra et al. , 2017) , and KNRM  (Xiong et al. , 2017) , referred to as pre-BERT models, BERT and its variants do not appear to require any specialized neural architectural components to capture different aspects of relevance between a query and a document  (Yates et al. , 2021). The same architecture based on homogeneous transformer layers is employed regardless of the downstream task. Qiao et al. (2019)  study the behaviour of BERT for ranking and revealed that it focuses more on document terms that directly match the query. Compared to pre-BERT models such as Conv-KNRM  (Dai et al.
-We conduct extensive experiments on standard TREC ad-hoc benchmark collections. -We study the contribution of exact match scores from a bag-of-words model to the end-to-end effectiveness of our models. -We compare our best configurations with state-of-the-art published results where available. -We publish our source code as well as our ready-to-use checkpoints at: https://github. com/BOUALILILila/ExactMatchMarking 2 Background and Related Work In this paper, we focus on ad hoc document retrieval (also referred to as document ranking) over corpora comprising either news articles or web pages. Following the standard formulation: Given a corpus of documents C, potentially large, the task of a ranking system is to produce a ranked list of k documents from the corpus in response to a user's information need expressed as query q.
Despite the reported successes of these neural models, there has recently been some skepticism about whether these successes, in the absence of large amounts of data, are not just inflated by comparison to weak baselines. The study conducted over a 100 papers by  (Yang et al. , 2019a)  on the Robust04 dataset showed that most models failed against strong non-neural baselines (RM3  (Lavrenko and Croft, 2001) ). Recently, the inception of the transformer architecture  (Vaswani et al. , 2017)  instigated a new wave of approaches  (Nogueira and Cho, 2019; MacAvaney et al. , 2019; Akkalyoncu Yilmaz et al. , 2019 ) that, at last, were able to significantly outperform well-tuned traditional IR baselines such as RM3  (Lavrenko and Croft, 2001). Nogueira and Cho (Nogueira and Cho, 2019)  describe the first successful application of BERT  (Devlin et al.
Following the same strategy, Birch (Akkalyoncu  Yilmaz et al. , 2019)  reports state-of-the-art effectiveness on the classic TREC newswire test collections Robust04, Core17 and Core18 using fine-tuned BERT on exclusively out of domain passage-level datasets (TREC Microblog, MS MARCO and TREC CAR). Their experiments demonstrate that relevance models can be transferred across different domains, which solves the problem of the lack of passage-level relevance annotations in the target domain. Similarly,  Dai and Callan (2019)  use passage-level evidence to fine-tune BERT by considering all passages from a relevant document as relevant. For inference, the document is split into overlapping passages and each passage is scored individually. Document scores based either on the score of the first passage, the best passages or the sum of all passage scores have been investigated and simple best passage score was found to be the best approach (BERT-MaxP).
Qiao et al. (2019)  show that BERT is an interactionbased model  (Guo et al. , 2016) , its advantage lies in the cross query-document attentions. Discarding these cross sequence interactions lead to a performance close to random. They also find that BERT assigns extreme matching scores to query-document pairs and most pairs get either one or zero ranking scores, showing it is well tuned by pre-training on large corpora. Câmara and Hauff (2020)  analyze BERT using diagnostic datasets built from retrieval heuristics  (Rennings et al. , 2019). Their experiments show that BERT does not fulfil most retrieval heuristics created by IR experts and argue that these axioms are not suitable to understand BERT performance. MacAvaney et al. (2020a)  introduce ABNIRML a new framework for analysing the behavior of neural IR models.
The authors found that neural ranking models have fundamentally different characteristics from prior ranking models such as high sensitivity to word order and increasing relevance scores when non-relevant content is added to the document. The work we present in this paper represents, to the best of our knowledge, the first paper detailing with a general approach of highlighting exact matching signals to enhance contextualized pretrained language models such as BERT and reporting an exhaustive set of experiments using long-document ranking benchmarks. Although, using a marking technique to emphasize exact term matches in the query-document pair was first proposed in our own previous work entitled MarkedBERT  (Boualili et al. , 2020 ) that represents our initial study. This last was limited to one marking technique on a passage ranking task.
(2017). It is pretrained on a large corpus of unlabeled data using two unsupervised tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). BERT relies on a fine-tuning approach where minimal task specific parameters are fine-tuned together with the pretrained parameters for the downstream task. The special classification token [CLS] added to the beginning of the input is used for making judgements about sentence pairs. BERT is pretrained on the NSP task to predict whether Segment A immediately precedes Segment B in the original text. It can be further fine-tuned inexpensively on a downstream task. We adopt the model configuration described by Nogueira and Cho (2019) referred to as monoBERT or vanilla BERT. In this configuration, the authors propose to apply BERT as a binary relevance classifier for text ranking.
, #q n #,. , #q m #,. , q |Q| } D = {d 1 ,. , #d i #,. , #d j #,. , #d l #,. , d |D| } when using the simple markers, or as follows when using the precise markers: Q = {q 1 ,. , [e n ]q n [/e n ],. , [e m ]q m [/e m ],. , q |Q| } D = {d 1 ,. , [e n ]d i [/e n ],. , [e n ]d j [/e n ],. , [e m ]d l [/e m ],. , d |D| } 4 Experimental setup This section presents the experimental setup used for studying the effectiveness of our models including datasets, baselines, fine-tuning and inference settings.
Datasets We conduct experiments on two standard ad-hoc benchmarks: Robust04 and GOV2. Robust04 1  is a news wire collection comprising 500K documents (TREC Disks 4 and 5) and 249 judged topics. Each topic is composed of three fields: The "title" is a short keyword query, the "description" is a longer wellformed natural language sentence that describes the information need and the "narrative" is a paragraph that provides guidance for relevance assessment. Table  4  provides an example of a TREC Robust04 topic. GOV2 2  is a Web collection crawled from government Websites in early 2004 comprising 25M documents and only 149 topics in the same format as Robust04 topics with   5  resumes some statistics on the two collections. As we focus only on reranking, the reported average document length is obtained from the top 1, 000 documents returned by BM25.
Traditional non-neural baselines BM25, we use the Anserini  (Yang et al. , 2017)  implementation with default parameters. For description queries, we set k 1 = 0. 9 for Robust04 and k 1 = 2. 0 for GOV2 and b = 0. 6 for both datasets. This unsupervised model serves both as a baseline and as the first stage retriever in all our experiments. BM25+RM3, a query expansion model based on RM3  (Lavrenko and Croft, 2001)  considered as a strong non-neural baseline. We use the Anserini  (Yang et al. , 2017)  implementation with the default parameters. For description queries, we use 20 expansion terms following  (Li et al. , 2020).
Training We use the base version (12 layers, 768 hidden size, 12 heads, 110M parameters) of BERT due to hardware limitations. We fine-tune both our vanilla baseline and our models augmented with the different marking strategies on the large publicly released MS MARCO passage dataset. We use a batch-size of 128 and the maximum sequence length (128 sequences × 512 tokens = 65,536 tokens/batch) for 100k on free Google Colab TPUs 3. We use Adam optimizer  (Kingma and Ba, 2015)  with the initial learning rate set to 3e -6 and linear decay of the learning rate. The drop out rate is set to 0. 1 for all our experiments. We use the open source implementation of BERT by Hugging Face  (Wolf et al. , 2020). It is important to note that fine-tuning an augmented model with a marking strategy does not add a computational cost compared to the vanilla model.
As a trade-off between latency and effectiveness, we only consider a maximum of 30 passages per document. The first and last passages are always picked while the remaining 28 are randomly chosen. The fine-tuned BERT models on exclusively out-of-domain data are used afterwards to predict the relevance of each passage w. r. t a query q independently. The best scoring passage is then taken as a proxy for the Document-level relevance: R(d, q) = max(R(p 1 , q),. , R(p n , q)) (3) For the queries we consider both the topic titles that are preferred by most pre-BERT models including BM25, and the descriptions that are more similar to MS MARCO's natural language questions.
We use the fine-tuned models on exclusively out-of-domain data (MS MARCO passage collection) to rerank the documents retrieved by BM25 in the first-stage. We do not train the models on the target collections (Robust04, GOV2), we use all their queries and relevance judgements as a held-out test set. Thus, this evaluation is an instance of a zero-shot transfer setting. Table  6  shows the reranking effectiveness of our different models and baselines on the top 1, 000 candidate documents retrieved by BM25 from Robust04 and GOV2 collections using both the title and description fields of the TREC topics. We recall that titles are short key word queries preferred by traditional bag-of-words models like BM25 and descriptions are well-written natural language queries similar to MS MARCO's questions on which the BERT models are fine-tuned.
Multi-Phase Fine-Tuning In previous experiments, we leveraged out-of-domain relevance assessments to fine-tune our BERT models. This fine-tuning aims at providing the model with general notions of relevance matching. However, transferring these relevance patterns to the target corpus may, in some cases, be ineffective. To overcome this domain-transfer limitation, we use additional fine-tuning on labeled data drawn from the same distribution as the final task, in other words, in-domain labeled data fine-tuning. This approach is known as "stage-wise" or "multiphase" fine-tuning  (Yates et al. , 2021). Once the models are fine-tuned on the MS MARCO passage dataset following the training setting described in section 4. 3, we further fine-tune them on the target task using 5-fold cross validation for both Robust04 and GOV2 collections.
Zero-shot transfer setting We use the fine-tuned models on exclusively out-of-domain data, i. e MS MARCO passage dataset, and apply inference on the window-passages obtained by splitting each document using the same passage length of 150 words and a 75 words stride used in the BERT experiments. Table  12  shows the results obtained at cutoff 1, 000 on both Robust04 and GOV2 collections. We recall the results of the Vanilla and Sim-Pair models with the BERT core for comparison. Exact Match Marking on ELECTRA. Results indicate clearly that adding exact match marking is still beneficial for the ELECTRA variant. As for the BERT version, Sim-Pair ELECTRA is more effective on Robust04 with an average improvement rate of +5% compared to only half, +2. 5%, on GOV2. However, exact match marking has more notable impact on titles rather than descriptions, when the vanilla ELECTRA baseline prefers clearly description queries.
Multi-phase fine-tuning Table  13  shows the results obtained using the multi-phase fine-tuning on both MS MARCO passage dataset and in-domain labeled data, described in section 5. 3 for BERT. The ELECTRA-based models outperform the BERT-based models on both collections regardless of the topic field used indicating that ELECTRA is a more effective core PLM than BERT in a mlti-phase finetuning setting. However, adding exact match marking has no significant impact in this setting. Sim-Pair ELECTRA performs slightly better than the vanilla ELECTRA baseline on the Robust04 collection across title, description and hybrid runs. On the other hand, exact match marking leads to better ranking accuracy on GOV2 titles, but provokes a slight degradation in performance when the description field is used for reranking (description and hybrid runs).
Comparison with state-of-the-art models In this section we try to situate our approach with regard to what has already been proposed for document ranking. We consider the following models that represent the state of the art: Birch (MS) and Birch (MS-MB). Aggregates document-level scores provided by BM25 with the sentence-level evidence from BERT to rank the documents (Akkalyoncu  Yilmaz et al. , 2019). The notation in parentheses indicate the fine-tuning dataset(s): Ms for MS MARCO and MS-MB refers to the model fine-tuned first on MS MARCO and then further fine-tuned on Microblog (MB) data. We use the results reported by  Li et al. (2020)  that uses BM25 instead of BM25+RM3 as the first-stage retriever. BERT-MaxP (MS). Uses the best-scoring passage as a proxy for the document relevance score  (Dai and Callan, 2019).