SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CSMED: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Wojciech Kusa $^{1*}$ , Oscar E. Mendoza $^{2}$ , Matthias Samwald $^{3}$ , Petr Knoth $^{4}$ , Allan Hanbury $^{1}$ $^{1}$ TU Wien  $^{2}$ University Milano-Bicocca  $^{3}$ Medical University of Vienna  $^{4}$ The Open University
Systematic literature reviews (SLRs) play an essential role in summarising, synthesising and validating scientific evidence. In recent years, there has been a growing interest in using machine learning techniques to automate the identification of relevant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability to systems treating automated literature screening as a classification task, as opposed to, for example, a retrieval or question-answering task. To address these challenges, we introduce CSMED, a meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMED serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMED-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMED, we conduct experiments and establish baselines on new datasets.
Systematic literature reviews (SLRs, or meta-reviews) are a critical tool in scientific research, used for synthesising and summarising evidence from multiple studies. The SLR process involves several stages, including citation screening (CS, or selection of primary studies) which is, in itself, a time-consuming step [58, 10]. CS involves identifying studies relevant to the SLR based on a set of, often complex, inclusion and exclusion criteria (e.g., the study must be examining the efficacy of Drug X on Condition Y).
In recent years, there has been an increasing interest in automating the SLR process [80, 60, 56, 22, 2], with works often focusing on improving the CS step by (a) using machine learning (ML) [44, 45], (b) natural language processing (NLP) [27, 36, 79], and (c) information retrieval (IR) [70, 90] techniques. Automated CS systems have the potential to significantly reduce the time and resources required for this critical step, thereby speeding up the SLRs production [75].
The development of standards provides invaluable resources for evaluating and comparing different models. Benchmarks, such as BEIR [74], GLUE [85] or BLURB [20] have shown improvements in reproducibility and progress tracking of machine learning models in various domains. Unfortunately, in the context of SLR automation, the absence of standard benchmarks and evaluation methodologies still hampers progress and inhibits the development of reliable and effective solutions.
With the fast-evolving landscape of machine learning, identifying state-of-the-art performance has become especially challenging and inefficient in the context of CS. The notorious proliferation of small custom CS datasets and single-usage evaluation approaches further exacerbates this issue.
*Corresponding author: wojciech.kusa@tuwien.ac.at
We show that current CS datasets exhibit several shortcomings that hinder their applicability for comprehensive and standardised evaluations. These datasets are poorly documented, with most lacking datasheets, clear licenses and terms of use. In addition, the limited applicability of older datasets arises from their small size and lack of crucial metadata, restricting their use to classification tasks. Finally, data leakage and dataset overlap is another issue, with some SLRs present in multiple collections.
To address these limitations, we present CSMED (Citation Screening Meta-Dataset), a comprehensive collection of CS datasets that can be used to benchmark and evaluate automated screening systems. Our collection builds upon nine existing datasets, and a new dataset for evaluating the full text classification task, counting 325 SLRs from the fields of medicine and computer science. Thanks to the data harmonisation, our new collection can mitigate the issues of lack of canonical splits, limited applicability, and dataset overlap. Our contributions are as follows:
1. We create CSMED, a meta-collection of nine datasets comprising 325 SLRs. CSMED is built upon BIGBIO [16] and can be used to evaluate and benchmark automated CS systems. We also provide a comprehensive summary of existing citation screening datasets.
2. We extend CSMED with additional metadata after analysing issues on the existing collections and previous evaluation frameworks. Our extended dataset can be used to evaluate CS as question answering or textual pairs classification tasks.
3. Using new metadata, we introduce CSMED-FT, a new dataset for the task of full text screening. To the best of our knowledge, this is the first dataset designed explicitly on screening long documents in SLR. This dataset can be used for the evaluation of the inference capabilities based on a very long context  $(4,000+$  words).
The remainder of the paper is structured as follows. In Section 2, we define the task of citation screening for systematic literature reviews. Section 3 provides an overview of related work in SLR automation and available benchmarks. Section 4 describes the CSMED meta-dataset in detail, including its creation, analysis and extension. In Section 5, we introduce the full text screening dataset together with baseline results on this dataset, and in Section 6, we discuss the implications of our work and potential extensions.
2 Task formulation
We start by introducing the task of citation screening for SLRs and presenting the notation used for its formulation. An SLR is characterised by various attributes, including the title, abstract, research question  $\mathcal{R}\mathcal{Q}$ , and eligibility criteria  $\mathcal{C}$ . We refer to all these attributes as the SLR protocol. Eligibility criteria comprise a set of rules and conditions that a document must meet for inclusion in the SLR. Given a large pool of documents denoted as  $\mathcal{D}$ , the main goal of automated citation screening is to assist researchers in identifying relevant publications for inclusion in an SLR. Each document  $d\in \mathcal{D}$  has attributes such as its title, abstract, main content, authors, and publication year. The task of CS for SLRs can be formally defined as follows:
Definition 2.1 (CS). Given a set of documents  $\mathcal{D}$  and a set of eligibility criteria  $\mathcal{C}$ , the task of CS for SLR is to determine for each document  $d\in \mathcal{D}$  whether it satisfies the criteria  $\mathcal{C}$ . This decision can be represented as a binary label  $y_{d}\in \{0,1\}$ , where  $y_{d} = 1$  if document  $d$  satisfies the criteria  $\mathcal{C}$ , and  $y_{d} = 0$  otherwise.
It is important to note that the manual CS is conducted in two steps, as shown in Figure 1: title and abstract screening and full text screening. In the first step, the relevance of each document is evaluated based on its title and abstract, while in the second step, a more thorough assessment is performed by examining the full text of the document.
Document retrieval The initial step involves document retrieval, which aims to generate a set of potentially relevant documents  $\mathcal{D}' \subseteq \mathcal{D}$  given  $\mathcal{RQ}$ . This step commonly involves querying bibliographic databases with specific keywords and Boolean expressions. We can formulate this step as a retrieval function  $r$ , such that  $r(\mathcal{RQ}, \mathcal{C}) = \mathcal{D}'$ . However, the retrieved set  $\mathcal{D}'$  may contain a large number of false positives (irrelevant documents).
Figure 1: Illustration of the citation screening process, separated into two tasks (1) title and abstract screening and (2) full text screening. Tasks are represented as a specific example of question-answering when a single question asks for a fulfilfilment of all eligibility criteria  $\mathcal{C}$  at once.
Binary classification for relevance prediction Following document retrieval, the primary task is to assess the relevance of each document in the set  $\mathcal{D}'$  concerning the eligibility criteria  $\mathcal{C}$ . This is conducted in two stages, differing in which attributes of documents are considered (titles and abstracts vs. full texts). We treat this as a binary classification problem, where each document  $d \in \mathcal{D}'$  is assigned a binary label  $y_d \in \{0,1\}$  to indicate its relevance ( $y_d = 1$ ) or irrelevance ( $y_d = 0$ ) to the SLR per the criteria  $\mathcal{C}$ .
Question answering for relevance An alternative formulation of the citation screening task is to frame it as a question-answering problem. In this approach, we transform the eligibility criteria  $\mathcal{C}$  into a set of questions  $\mathcal{Q} = \{q_1,\dots ,q_{|\mathcal{C}|}\}$ , where each question  $q_{k}$  corresponds to a specific criterion in  $\mathcal{C}$ . For each document  $d\in \mathcal{D}'$ , we obtain a set of predicted answers  $\hat{A}^d = \{\hat{a}_k^d |\mathrm{meets}(q_k,\hat{a}_k^d)\}$ , where  $\mathrm{meets}(q_k,\hat{a}_k^d)$  denotes that the document  $d$  should meet the criterion expressed by the question  $q_{k}$ . The final relevance label  $\hat{y}_d$  of a document  $d$  can be determined by aggregating the predicted answers  $\hat{A}^d$  using a logical combination function, such as the logical AND operation.
This question-answering formulation offers a more fine-grained assessment of a document's relevance concerning various aspects of the eligibility criteria  $\mathcal{C}$ . Other similar formulations of the CS task include document ranking or natural language inference (NLI).
We first motivate the work by providing context on the importance of SLRs and then focus on reviewing citation screening automation methods. Finally, we outline limitations of existing CS datasets.
3.1 Systematic literature reviews
SLRs are particularly important in the medical domain [29]. The Cochrane Collaboration, $^{2}$  the largest organisation responsible for creating SLRs in medicine, has created the foundations of Evidence-Based Medicine [24]. There are more than 220,000 records published between 2000 and 2022 tagged as SLRs in PubMed $^{3}$  meaning that, on average, there were 10,000 SLRs published per year.
As SLRs focus on reproducibility and finding all relevant evidence about a given topic, the traditional framework involves tasks mainly done manually. It includes steps like defining the search strategy (designing complex Boolean queries) or the screening of every document by at least two reviewers, resulting in an average production time of more than one year [77].
Previous research focused on evaluating automation capabilities for several steps of the traditional framework, such as citation screening (CS) [11, 27, 75], search query (re-)formulation [70, 72], data extraction [59], SLR summarisation [86] or generation of reviews based on the title [94].
$^{2}$ https://www.cochrane.org
<sup>3</sup>https://pubmed.ncbi.nlm.nih.gov/
3.2 Citation screening automation
As described in Section 2, CS can be seen as a binary classification problem. However, due to a large number of retrieved studies, the significant class imbalance, and the need to identify nearly all relevant documents, this task is inherently complex. Screening automation is a general term for various approaches aimed at reducing workload during the CS stage [60]. These approaches can be classified as either screening reduction, which involves using classification or ranking algorithms to automatically exclude non-relevant publications or screening prioritisation, which focuses on ranking relevant records earlier in the screening process [56]. Automated screening systems leverage techniques from NLP, ML, IR, and statistics, all with the common objective of reducing manual screening time. The disparity in strategies from different fields hinders direct comparison and benchmarking. Next, we discuss some points of disagreement.
NLP approaches typically focus on the level of individual SLRs, treating each review as an independent dataset; whereas IR approaches would consider a set of reviews as a collection, the topics of the reviews analogous to queries, and report aggregated evaluation. Moreover, different publications across various venues adopt diverse evaluation measures, making even more complex the assessment of similar, if not identical, tasks. Evaluation of automatic approaches traditionally relies on binary relevance ratings, very often obtained from the title and abstract screening [60, 32]. When the screening problem is treated as a ranking task, such as screening prioritisation or stopping prediction; the performance is measured in terms of rank-based metrics and metrics at a fixed cut-off, such as  $nDCG@n$ , Precision@ $n$ , and last relevant found [71, 28]. On the other hand, when the screening problem is treated as a classification task, the performance in this case is measured based on the confusion matrix and the notions of Precision and Recall are commonly used [41, 60]. One challenge arising from these two distinct approaches is the difficulty in going beyond simple effectiveness measures and comparing the real-world savings for users. Further details on datasets and evaluation approaches can be found in a comprehensive review in the Appendix A.
3.3 Limitations of existing datasets
Through our review (see Appendix A), we identified twelve CS datasets reported in former research papers, of which ten have been publicly released. During this analysis, we identified several shortcomings; some are also prevalent in other machine learning problems. Below, we summarise our findings, highlighting the key issues.
Poor documentation One major concern with previous datasets is the lack of sufficient documentation. None of the datasets we examined implement a datasheet [18], which is an essential tool for ensuring transparency and reproducibility. Additionally, seven datasets do not provide clear licenses or terms of use. An inconsistency was also found for one of the datasets [71] in terms of the number of the available content: the paper states 93 SLRs, but we found a list of 176 reviews on the corresponding GitHub repository.
Limited applicability Previous datasets are often small and lack crucial metadata like SLR research question or eligibility criteria, limiting their use to only evaluation of classification tasks. Older datasets typically provide only the title of the review, which limits their applicability for the comprehensive evaluation of neural language understanding models. The most widely used dataset to date [11] was released in 2006. As ML and NLP techniques continue to advance rapidly, it is crucial to have up-to-date datasets that reflect the complexities and nuances of the current research landscape.
Lack of canonical splits Another significant challenge of previous datasets is the absence of canonical train-test splits. Depending on the field of research, practices may vary. As discussed before, in the ML and NLP domains, the prevailing practice is to use inter-review splits, where each review is treated as an individual dataset, and a set of citations is selected for training and testing. Conversely, IR publications often report intra-review splits, treating each review as a "topic" or query, and averaging the results across multiple queries.
In this sense, only the three  $\mathrm{TAR^4}$  datasets contain pre-defined canonical splits, yet only at the intra-review level. For three other datasets [11, 82, 27], previous works have demonstrated significant
$^{4}$ TAR stands for Technology-Assisted Reviews and was a shared task organised at CLEF between 2017 and 2019 by Kanoulas et al. [30, 31, 32].
variability in model evaluation based on the selection of cross-validation splits, particularly for the smallest datasets that contain a limited number of relevant documents [79, 38]. The lack of standardised splits, especially in collections with fewer SLRs, makes it challenging to compare different approaches and hinders the fair evaluation of models' performance.
Dataset overlap We also evaluated overlapping throughout the previous datasets and discovered that at least 11 SLRs were present in multiple collections [71, 30, 31, 32]. Additionally, the TAR 2019 dataset contains three SLRs that are present both in its training and test splits, accounting for approximately  $6\%$  of the test partition [32]. While this overlap is not a significant concern when evaluating unsupervised methods like BM25 [69], it poses a potential threat to conducting fair comparisons with large language models (LLMs). Machine learning models, and especially LLMs, have the capability to memorise their training data, making it critical to address dataset overlap to ensure unbiased evaluations [25] (see Appendix E for a detailed analysis of the overlapping in previous datasets).
Lack of common evaluation Another notable deficiency among the previous datasets is the absence of a common set of evaluation measures. Only the three TAR datasets provide scripts for evaluating submissions. For example, the most widely used dataset by Cohen et al. [11] was evaluated using several disparate evaluation measures such as WSS [11], AUC or Precision@r%. However, recent research has exposed limitations and problems with both WSS and AUC as metrics for this task [40].
Availability in biomedical benchmarks Recent efforts have focused on creating larger collections of more diverse datasets to evaluate the performance of biomedical NLP models. These efforts include benchmarks like BLUE [64], HunFlair [92], BLURB [20], and BigBio [16], which provide datasets and tasks for evaluating biomedical language understanding and reasoning. Additionally, there are biomedical datasets geared towards prompt-based learning and evaluation of few and zero-shot classification, such as Super-NaturalInstructions [91] and BoX [62]. Out of all benchmarks mentioned above, only BoX contains one CS dataset covering five SLRs, however, this dataset is private. Coverage for other SLR tasks is also limited.
To summarise, previous datasets exhibit certain drawbacks that limit their suitability for comprehensive and standardised evaluation. While the TAR 2017-19 collections stand out as the only ones containing canonical splits and a set of evaluation measures, some of their topics overlap with another previous dataset [71], and we also identified data leakage in the newest TAR 2019 dataset. Consequently, we believe that developing a new collection is necessary to address these issues and establish a robust foundation for evaluation of CS and SLR automation.
4 The CSMED meta-dataset
The recent advancements and paradigm shifts in NLP and ML; with the extensive use of pre-trained models and transfer learning [46, 15], and the more recent prompt-based learning [49, 9]; have significantly transformed the field and enhanced the predictive capabilities of models across various tasks. Inspired by the success of benchmark collections in the field of biomedical NLP, we conducted a thorough review of available datasets and benchmarks to identify the most representative datasets for the task of citation screening, finding that this task is heavily underrepresented. The available datasets still primarily cater to training supervised algorithms, lacking the scale and granularity necessary to evaluate state-of-the-art models. To address these limitations and provide a more comprehensive resource for training and evaluating data-centric methods in SLR automation, we create CSMED, consolidating nine previously released collections of SLRs. We further extend a subset of SLRs in CSMED with additional metadata coming from the review protocol.
Our data analysis methodology involved creating visualisations and summary tables based on curated datasets. We analyse dataset statistics like available data splits, licensing information, dataset and reviews size as well as dataset overlap. This allows us to provide both a detailed view of individual reviews and an overview of datasets containing multiple reviews (see Appendix B for further details on visualisations).
4.1 Dataset creation
Currently, nine out of ten public CS datasets we identified have been included in CSMED, with the remaining one to be included. We provide a summary of the datasets in Table 1, and further details can be found in Appendix A. In total, CSMED consists of 325 SLRs, making it the largest publicly available collection in this domain and the only one providing access to the datasets via a harmonised API.
Table 1: A list of source citation screening datasets included in the CSMED. The first four datasets contain non-Cochrane SLRs, whereas the other five are based on Cochrane reviews. 'Avg. ratio of included' column presents ratio of included publication from the title and abstract screening stage, 'Avg. size' refers to averaged across SLRs document count in the dataset. The 'Additional data' column describes if the review contains metadata other than coming from the citation list: (A): Search queries, (B): Review protocol containing review title, abstract and search strategy, (C): Review updates consisting of changes to included papers. 'DTA' stands for diagnostic test accuracy reviews.  $^{\ddagger}$  - indicates a discrepancy in the number of reviews in the paper versus the GitHub repository.  $^{\dagger}$  - indicates the total count of reviews from all nine datasets before duplicates were removed.
Source	# reviews	Domain	Avg. size	Avg. ratio of included	Additional data	Cochrane reviews
[11]	15	Drug	1,249	7.7%	—	—
[82]	3	Clinical	3,456	7.9%	—	—
[27]	5	Mixed	19,271	4.6%	—	—
[22]	7	Comp. Science	340	11.7%	B	—
[71]	93/176‡	Clinical	1,159	1.2%	A	✓
[30]	50	DTA	5,339	4.4%	B	✓
[31]	30	DTA	7,283	4.7%	B	✓
[32]	49	Mixed	2,659	8.9%	B	✓
[2]	25	Clinical	4,402	0.4%	C	✓
Total	360†	3,471	4.4%
To ensure interoperability and facilitate the ease of use, we designed data loaders for the datasets in accordance with the BigBio text classification schema [16]. This choice offers several advantages. BigBio has the largest coverage of biomedical datasets and supports access to the datasets via API. Moreover, it is compatible with popular libraries such as Hugging Face's datasets [47] and the EleutherAI Language Model Evaluation Harness [17], thereby reducing the experimental costs.
Taking advantage of the lists of publications that most of the sources of datasets share as PubMed IDs, we extend the BigBio data loaders to enable the download of PubMed articles. Our harmonised data loaders selectively load the PubMed articles that are a part of each dataset. The single exception is the dataset by Hannousse and Yahiouche [22], which is the only publicly available collection of non-medical SLRs. For this dataset, we extract the content using the SemanticScholar API. As a result, CSMED serves also as the first resource that gathers SLRs from diverse domains.
4.2 Extending metadata
Our goal at this stage is not to create yet another gold standard dataset for SLRs, but rather improve the quality of current data and provide insights into promising avenues for future research. We begin by presenting the possibilities of extending the subset of Cochrane SLRs to experiment with screening beyond supervised classification.
We then categorise CSMED datasets into two groups: (1) datasets containing Cochrane medical SLRs and (2) datasets comprising other SLRs. This distinction is made because from following the Cochrane protocol, more extensive information on the review is provided. We use the additional data available on reviews websites to extend CSMED. Among the new information, the one we find most valuable is the eligibility criteria, which no longer limits the data to the evaluation of supervised binary classification but opens its application to question-answering or language inference tasks.
<sup>5</sup>https://wwwsemantic scholar.org/product/api