CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Authors: Wojciech Kusa
Oscar E. Mendoza
Matthias Samwald
Petr Knoth
Allan Hanbury
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: evaluation
NLP
meta-dataset
citation screening
systematic reviews
TL;DR: CSMeD is a consolidated meta-dataset addressing the limitations of existing datasets in evaluating automated citation screening for systematic literature reviews, offering unified access to 325 SLRs and introducing a new full text screening dataset.

Abstract: Systematic literature reviews (SLRs) play an essential role in summarising, synthesising and validating scientific evidence. In recent years, there has been a growing interest in using machine learning techniques to automate the identification of relevant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability to systems treating automated literature screening as a classification task, as opposed to, for example, a retrieval or question-answering task. To address these challenges, we introduce CSMED, a meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMED serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMED-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMED, we conduct experiments and establish baselines on new datasets.

CSMED: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Wojeiech Kusa Oscar E. Mendoza',
Matthias Samwald Petr Knoth Allan Hanbury'
TU Wien UUniversity Milano-Bicoeca
Medical University of Vienna "The Open University
Abstract
Systematic literature reviews (SIRs) play an essential role in summarising, synthe- sising and validating scientific evidence. In recent years, there has been growing interest in using machine learning techniques to automate the identification of rele- vant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability systems treating automated literature screening as a classifi- cation task, as opposed to, for example, retrieval or question- -anoon--nnwwnnng task. To address these challenges, we introduce CSMED, meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMRD serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMED-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMED, we conduct experiments and establish baselines on new datasets.
Introduction
Systematic literature reviews (SLRs, meta-reviews) are critical tool in scientific research, used for synthesising and summarising evidence from multiple studies. The SLR process involves several stages, including citation screening (CS, or selection of primary studies) which is, itself, time- consuming step [58 I0]. CS involves identifying studies relevant to the SLR based on set of, often complex, inclusion and exclusion criteria (e.g., the study must be examining the efficacy of Drug on Condition Y).
In recent years, there has been increasing interest in automating the SLR process 80
60
56
22
with works often focusing on improving the CS step by (a) using machine learning (ML)
45 (b)
natural language processing (NLP) (27 36 79). and (c) information retrieval (IR) 770 90 techniques.
Automated CS systems have the potential to significantly reduce the time and resources required for this critical step, thereby speeding up the SLRs production [75].
The development of standards provides invaluable resources for evaluating and comparing different models. Benchmarks, such as BEIR (74], GLUE [85] or BLURB (0) have shown improvements in reproducibility and progress tracking of machine learning models in various domains. Unfortunately, in the context of SLR automation, the absence of standard benchmarks and evaluation methodologies still hampers progress and inhibits the development of reliable and effective solutions.
With the fast-evolving landscape of machine learning, identifying sate--of-the-ar performance has become especially challenging and inefficient the context of CS. The notorious prolifcration of small custom CS datasets and single-usage evaluation approaches further exacerbates this issue.
"Corresponding author: woj rojci.ch. iusa@tuvien.ac.cr
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

we discuss the implications of We show that current CS datasets exhibit several shortcomings that hinder their applicability for comprehensive and standardised evaluations. These datasets are poorly documented, with most lacking datasheets, clear licenses and terms of use. In addition, the limited applicability of older datasets arises from their small size and lack of crucial metadata, restricting their use to classification tasks. Finally, data leakage and dataset overlap is another issue, with some SLRs present in multiple collections.
To address these limitations, we present CSMED (Citation Screening Meta- Dataset),, a compre- hensive collection of CS datasets that can be used to benchmark and evaluate automated screening systems. Our collection builds upon nine existing datasets, and a new dataset for evaluating the full text classification task, counting 325 SLRs from the fields of medicine and computer science. Thanks to the data harmonisation, our new collection can mitigate the issues of lack of canonical splits, limited applicability, and dataset overlap. Our contributions are as follows:
1. We create CSMED, a meta-collection of nine datasets comprising 325 SLRs. CSMED is built upon BIGBIO [6 and can be used evaluate and benchmark automated CS systems. We also provide a comprehensive summary of existing citation screening datasets.
2. We extend CSMED with additional metadata after analysing issues on the existing collec- tions and previous evaluation frameworks. Our extended dataset can be used to evaluate CS as question answering or textual pairs classification tasks.
3. Using new metadata, we introduce CSMED-FT, a new dataset for the task of full text screening. To the best of our knowledge, this is the first dataset designed explicitly on screening long documents in SLR. This dataset can be used for the evaluation of the inference capabilities based on a very long context (4,000+ words).
The remainder of the paper is structured as follows. In Section 2 we define the task of citation screening for systematic literature reviews. Section provides overview of related work in SLR automation and available benchmarks. Section 4 describes the CSMED meta-dataset in detail, including its creation, analysis and extension. In Section [ we introduce the full text screening dataset together with baseline results on this dataset, and in Section 6 our work and potential extensions.
2 Task formulation
We start by introducing the task of citation screening for SLRs and presenting the notation used for its formulation. An SLR is characterised by various attributes, including the title, abstract, research question RQ, and eligibility criteria C. We refer to all these attributes as the SLR protocol. Eligibility criteria comprise a set of rules and conditions that a document must meet for inclusion in the SLR. Given large pool of documents denoted as D, the main goal of automated citation screening is to assist researchers in identifying relevant publications for inclusion in an SLR. Each document d E D has attributes such its title, abstract, main content, authors, and publication year. The task of CS for SLRs can be formally defined as follows:
Definition 2.1 (CS), Given a set of documents 'D and a set of eligibility criteria C, the task of CS for SLR is to determine for each document d D whether it satisfies the criteria C. This decision can be represented as a binary label ya E {0, 1}. where ya = 1 if document d satisfies the criteria C, and yd otherwise.
It is important to note that the manual CS is conducted in two steps, shown in Figure [ title and abstract screening and full text screening. In the first step, the relevance of each document is evaluated based on its title and abstract, while in the second step, more thorough assessment is performed by examining the full text of the document.
Document retrieval The initial step involves document retrieval, which aims to generate a set of potentially relevant documents D' C D given RQ. This step commonly involves querying bibliographic databases with specific keywords and Boolean expressions. We can formulate this step as retrieval function r. such that r(R2, C) = D'. However, the retrieved set D' may contain large number of false positives (irrelevant documents).

---

Title
Title: Daplomycin
screening
Alorhnnd
Abetrace
de
acidee
Yes
Publications
Document retrieval
aegarry cheen Hencee ordern
and Efficaay Daptomydin
conforaaiiooe oeplonation
Citation Screening for "Systemic antibiatics for treating diabetic foot infections"
Figure I: Illustration of the citation screening process, separated into two tasks (1) title and abstract screening and (2) full text screening. Tasks are represented as a specific example of question- answering when a single question asks for fullfilment of all eligibility criteria C at once.
Binary classification for relevance prediction Following document retrieval, the primary task is to assess the relevance of each document in the set DD' concerning the eligibility criteria C. This conducted two stages, differing in which attributes of documents are considered (titles and abstracts vs. full texts). We treat this as binary classification problem, where each document d E D' assigned a binary label ya E {0, } to indicate relevance (μa = 1) or irrelevance (ya = 0) to the SLR per the criteria C.
Question answering for relevance An alternative formulation of the citation screening task is to frame as a question-ansswriigg problem. In this approach, we transform the eligibility criteria C into {911- ... .9cI }. where each question qk corresponds to a specific criterion
set of questions in C. For each document d E D', we obtain a set of predicted answers Á = {ağ (eets((k a where meets gb: ) denotes that the document d should meet the criterion expressed by the question Ts. The final relevance label ÿJ document d can be determined by aggregating the predicted answers Ad using a logical combination function, such as the logical AND operation.
This question-ansee answering formulation offers a more fine- ane-grained assessment of a document's 's relevance concerning various aspects of the eligibility criteria C. Other similar formulations of the CS task include document ranking or natural language inference (NLI).
Related work
We first motivate the work by providing context on the importance of SLRs and then focus on reviewing citation screening automation methods. Finally, we outline limitations of existing CS datasets.
3.1 Systematic literature reviews
SLRs are particularly important in the medical domain 29 The Cochrane Collaboration 2 the largest organisation responsible for creating SLRs in medicine, has created the foundations of Evidence- Based Medicine 24 There more than 220,000 records published between 2000 and 2022 tagged as SLRs in PubMed meaning that, on average, there were 10,000 SLRs published per year.
As SLRs focus reproducibility and finding all relevant evidence about given topic, the traditional framework involves tasks mainly done manually. It includes steps like defining the search strategy (designing complex Boolean queries) or the screening of every document by at least two reviewers, resulting in an average production time of more than one year 77].
Previous research focused on evaluating automation capabilities for several steps of the traditional framework, such as citation screening (CS) 27 75), search query (re- formulation 770 722. data extraction [59]. SLR summarisation
86 or generation of reviews based on the title [4]
https://pubmed. ncbi. www...lm.........mmmm.mmm......m nih, nih.gov/ gov/
3

---

3.2 Citation screening automation
As described in Section 2 CS can be seen as a binary classification problem. However, due to a large number of retrieved studies, the significant class imbalance, and the need to identify nearly all relevant documents, this task is inherently complex. Screening automation is a general term for various approaches aimed at reducing workload during the CS stage [60]. These approaches can be classified as either screening reduction, which involves using classification or ranking algorithms to automatically exclude non-relevant publications or screening prioritisation, which focuses on ranking relevant records earlier in the screening process [56]. Automated screening systems leverage techniques from NLP, ML, IR, and statistics, all with the common objective of reducing manual screening time. The disparity in strategies from different fields hinders direct comparison and benchmarking. Next, we discuss some points of disagreement.
NLP approaches typically focus on the level of individual SLRs, treating each review as an indepen- dent dataset; whereas IR approaches would consider a set of reviews as a collection, the topics of the reviews analogous to queries, and report aggregated evaluation. Moreover, different publications across various venues adopt diverse evaluation measures, making even more complex the assessment of similar, if not identical, tasks. Evaluation of automatic approaches traditionally relies on binary
relevance ratings, very often obtained from the title and abstract screening [60] [32. When the
screening problem is treated as a ranking task, such as screening prioritisation or stopping prediction; the performance is measured in terms of rank-based metrics and metrics at fixed cut-off, such as nDGG@n, Precision@n, and last relevant found [7L 28 On the other hand, when the screening problem is treated as classification task, the performance in this case is measured based on the confusion matrix and the notions of Precision and Recall are commonly used [41] 60). One challenge arising from these two distinct approaches is the difficulty in going beyond simple effectiveness measures and comparing the real-world savings for users. Further details on datasets and evaluation approaches can be found in comprehensive review in the Appendix A
3.3 Limitations of existing datasets
Through our review (see Appendix A). we identified twelve CS datasets reported in former research papers, of which ten have been publicly released. During this analysis, we identified several short- comings; some are also prevalent in other machine learning problems. Below, we summarise our findings, highlighting the key issues.
Poor documentation One major concern with previous datasets is the lack of sufficient docu- mentation. None of the datasets we examined implement a datasheet [88] which is an essential tool for ensuring transparency and reproducibility. Additionally, seven datasets do not provide clear licenses or terms of use. An inconsistency was also found for one of the datasets [71] in terms of the number of the available content: the paper states 93 SLRs, but we found list of 176 reviews on the corresponding GitHub repository.
Limited applicability Previous datasets are often small and lack crucial metadata like SLR re- search question or eligibility criteria, limiting their use to only evaluation of classification tasks. Older datasets typically provide only the title of the review, which limits their applicability for the comprehensive evaluation of neural language understanding models. The most widely used dataset to date I was released in 2006. As ML and NLP techniques continue to advance rapidly, it is crucial to have up-to-date datasets that reflect the complexities and nuances of the current research landscape.
Lack of canonical splits Another significant challenge of previous datasets is the absence of canonical train-test splits. Depending on the field of research, practices may vary. As discussed before, in the ML and NLP domains, the prevailing practice is to use inter-review splits, where each review is treated as an individual dataset, and a set of citations is selected for training and testing. Conversely, IR publications often report intra-review splits, treating each review as a "topic" or query, and averaging the results across multiple queries.
In this sense, only the three TAR datasets contain pre-defined canonical splits, yet only at the intra-review level. For three other datasets [1] 82 27 previous works have demonstrated significant
'TAR stands for Technology- 2019 by Kanoulas et al. 30 31 32]
sechnology-Assisted Assisted Reviews and was a shared task organised at CLEF between 2017 and

---

variability in model evaluation based on the selection of cross-validation splits, particularly for the smallest datasets that contain a limited number of relevant documents 38). The lack of standardised splits, especially in collections with fewer SLRs, makes it challenging to compare different approaches and hinders the fair evaluation of models' performance.
Dataset overlap We also evaluated overlapping throughout the previous datasets and discovered that at least 11 SLRs were present in multiple collections [7] 30 31 32). Additionaly, the TAR 2019 dataset contains three SLRs that are present both in its training and test splits, accounting for approximately 6% of the test partition 322. While this overlap is not significant concern when evaluating unsupervised methods like BM25 [69] it poses a potential threat to conducting fair comparisons with large language models (LLMs). Machine learning models, and especially LLMs, have the capability to memorise their training data, making it critical to address dataset overlap (see Appendix F for detailed analysis of the overlapping in
to ensure unbiased evaluations 25
previous datasets).
Lack of common evaluation Another notable deficiency among the previous datasets is the absence of common set of evaluation measures. Only the three TAR datasets provide scripts for evaluating submissions. For example, the most widely used dataset by Cohen et al. M was evaluated using several disparate evaluation measures such as W SS IT, AUC or Precision@ However, recent research has exposed limitations and problems with both W SS and AUC as metrics for this task 44)
Availability in biomedical benchmarks Recent efforts have focused on creating larger collections of more diverse datasets to evaluate the performance of biomedical NLP models. These efforts include benchmarks like BLUE [64]. HunFlair [92]. BLURB (20). and BigBio [6), which provide datasets and tasks for evaluating biomedical language understanding and reasoning. Additionally, there are biomedical datasets geared towards prompt-based learning and evaluation of few and zero-shot classification, such as Huper-Naturallnstrutte mentioned above, only BoX contains one CS dataset covering five SLRs, however, this dataset is private. Coverage for other SLR tasks is also limited.
[91] and BoX [62]. Out of all benchmarks
To summarise, previous datasets exhibit certain drawbacks that limit their suitability for compre- hensive and standardised evaluation. While the TAR 2017-19 collections stand out as the only ones containing canonical splits and set of evaluation measures, some of their topics overlap with another previous dataset [7]), and we also identified data leakage in the newest TAR 2019 dataset. Consequently, we believe that developing a new collection is necessary to address these issues and establish a robust foundation for evaluation of CS and SLR automation.
4
The CSMED meta-dataset
The recent advancements and paradigm shifts in NLP and ML; with the extensive use of pre-trained models and transfer learning [46] [5], and the more recent comptt-based learning [49 | have significantly transformed the field and enhanced the predictive capabilities of models across various tasks. Inspired by the success of benchmark collections in the field of biomedical NLP, we conducted a thorough review of available datasets and benchmarks to identify the most representative datasets for the task of citation screening, finding that this task is heavily underrepresented. The available datasets still primarily cater to training supervised algorithms, lacking the scale and granularity necessary to evaluate state-of-thee art models. To address these limitations and provide a more comprehensive resource for training and evaluating data-centric methods in SLR automation, we create CSMED, consolidating nine previously released collections of SLRs. We further extend a subset of SLRs in CSMED with additional metadata coming from the review protocol.
Our data analysis methodology involved creating visualisations and summary tables based on curated datasets. We analyse dataset statistics like available data splits, licensing information, dataset and reviews size as well as dataset overlap. This allows us to provide both detailed view of individual reviews and an overview of datasets containing multiple reviews (see Appendix B for further details on visualisations).
5

---

4.1 Dataset creation
Currently, nine out of ten public datasets we identified have been included CSMED, with the remaining to be included. We provide summary of the datasets in Table L and further details can be found in Appendix In total, CSMED consists of 325 SLRs, making it the largest publicly available collection in this domain and the only one providing access the datasets via a harmonised API.
Table 1: A list of source citation screening datasets included in the CSMED. The first four datasets contain non- Ca-CCCnrre SLRs, whereas the other five are based on Cochrane reviews. 'Avg. ratio of included' column presents ratio of included publication from the title and abstract screening stage, 'Avg. size' refers to averaged across SLRs document count the dataset. The 'Additional data' column deseribes if the review contains metadata other than coming from the citation list: (A): Search queries, (B): Review protocol containing review title, abstract and search strategy, (C): Review updates consisting of changes included papers. 'DTA' stands for diagnostic test accuracy reviews. indicates a discrepancy in the number of reviews in the paper versus the GitHub repository. indicates the total count of reviews from all nine datasets before duplicates were removed.
Avg. size
Avg. ratio of included
Additional data
Cochrane reviews
Source
reviews
Domain
15
Drug
1,249
7.7%
82
Clinical
3,456
7.9%
27
Mixed
19,271
4.6%
22
Comp. Science
340
11.7%
B
71
93/176
Clinical
1,159
1.2%
30
50
DTA
5,339
4.4%
B
30
DTA
7,283
4.7%
B
32
Mixed
2,659
8.9%
25
Clinical
4,402
0.4%
C
Total
360
3,471
4.4%
To ensure interoperability and facilitate the ease use, we designed data loaders for the datasets in accordance with the BigBio text classification schema [16]. This choice offers several advantages. BigBio has the largest coverage of biomedical datasets and supports access to the datasets via APL. Moreover, it is compatible with popular librarics such Hugging Face's datasets [47] and the EleutherAl Language Model Evaluation Harness 17 thereby reducing the experimental costs.
Taking advantage of the lists of publications that most of the sources of datasets share as PubMed IDs, we extend the BigBio data loaders to enable the download of PubMed articles. Our harmonised data loaders selectively load the PubMed articles that are a part of each dataset. The single exception dataset by Hannousse and Yahiouche 222, which
only publicly available collection of non-medical SLRs. For this dataset, we extract the content using the SemanticScholar API 5 As result, CSMED serves also as the first resource that gathers SLRs from diverse domains.
4.2 Extending metadata
Our goal at this stage is not to create yet another gold standard dataset for SLRs, but rather improve the quality of current data and provide insights into promising avenues for future research. We begin by presenting the possibilities of extending the subset Cochrane SLRs to experiment with screening beyond supervised classification.
We then categorise CSMED datasets into two groups: (1) datasets containing Cochrane medical SLRs and (2) datasets comprising other SLRs. This distinction is made because from following the Cochrane protocol, more extensive information on the review is provided. We use the additional data available reviews websites to extend CSMED. Among the new information, the one we find most valuable is the eligibility criteria, which no longer limits the data to the evaluation of supervised binary classification but opens its application question question-answering or language inference tasks.
https: http://ww..mmanicscccllaaa.t http://hoo......errrrocu//ta//pp

---

We carefully examine the subset of SLRs produced by Cochrane, aiming to identify potential enhancements and extensions that would help mitigating the existing limitations previous datasets. Every Cochrane SLR first registers and publishes the protocol containing the review title, abstract, search strategy and the eligibility criteria. This information all that human experts need to produce the final review, i.e., they first find the relevant studies and then conduct the meta-analysis of their results. As described in Section 2 the screening process can be also modelled as a wersstionaanswering where every publication is compared against the eligibility criteria in order to make the decision about the inclusion similar to the clinical decision support task of matching clinical trials to patients 66 68
To expand CSMED, we searched the Cochrane Library for all SLRs from the meta-dataset based on the Cochrane review ID and take their latest open- -ccesss version. We extract available information about the review: review title and abstract, eligibility criteria, search strategy and references. Cochrane reports a list of included and excluded publications at the full text screening stage (this can be treated as approximately all included publications during the title and abstract screening stage). Moreover, all excluded publications have reason for exclusion selected by reviewer. As the original relevance judgements were limited to publications from the PubMed database, we assign PubMed IDs to these publications. We also define appropriate BigBio data loaders so the task can be seen as or textual pairs classification task.
Table 2: Details of the CSMED expanded meta-dataset. Column '#docs' refers to the total number the dataset, '#included' mentions number of included
of documents included in all SLRS within documents the title and abstract screening stage and 'Avg. %included" the percentage of included publications averaged from all reviews.
Avg. #docs
Avg. % included
Avg. #words in document
Dataset name
#reviews
#docs
#included
CSMED-TRAIIN-BAS -SMED-TRAIN-COCHEER CSMED-DEY-COCHRANNE
30
128,438
7,958
4,281
9.6%
229
195
372,422
7,589
1,910
21.9%
180
100
229,376
4,365
2,294
20.8%
201
CSMED-ALI
325
730,236
19,912
2,247
20.5%
195
Details of the new expanded CSMED provided in Table 2 We were not able to find suitable data for all SLRs, hence the expanded CSMED smaller than the original meta-dataset. In total, the new expanded dataset consists of 295 unique Cochrane SLRs and 30 Con-Cohra SLRs. The entire set of basic SLRs is designated for training. From the Cochrane reviews, we randomly selected 195 to the training split and the remaining 100 to the development split. We abstain from designating test split because CSMED aggregates existing datasets. Given the constraints of these datasets, creating a new, unbiased test collection is recommended.
4.3 Baseline experiment
We evaluate five models in zero-shot setting: two statistical models BM25 and TF-IDF, and three Transformer-based models. Predictions run the
split, and we
test four different input representations using the following sections from the SLR protocol: (1) title, (2) abstract, (3) search strategy and (4) eligibility criteria section. We evaluate models using
R@95% 40 [1], PP995 [41, Last Rel [30 B1 [2], n nDGG@10, AP,, Recall at rank with k in (10, 50, 100) (R@k). Detailed experimental setup and expanded results be found in Appendix F
Table presents presents the summary results of best persfrrerforinn statistical and neural models, while Table 9 in Appendix FF contains complete results. The MiniLM neural model consistently outperforms the BM25 variant across nearly all query representations and evaluation measures. For the BM25 model, using the systematic review abstract text as a query representation results in the highest performance in all metrics compared to using the SLR title and criteria sections. Conversely, for
'In the current approach, we consider only binary relevance (included versus excluded). However, in practice. more categories can be defined by reviewers (e.g. study can be assigned as background publication or meta-analysis).
htttps://wwe httpp://ww..cccaaaeellloorr www.wawwwwbbbrrrrr.c.o.. http://wwwraeeeeaaaa.a./o//aaaaa hwwp://rwwrrrooomo///////oooooo

---

Table 3: Summary of results DEVED--DE-OOCANNE dataset. Bold values indicate best score.
Model
Represenlation
TNR@95%
nP@95%
Last Rel
NDCG@10
MAP
R@10
R@:50
R@ 100
Title
0.469
0.142
72.2
0.438
0.388
0.349
0.623
0.704
Abstract
0.474
0.170
63.6
0.503
0.453
0.379
0.657
0.757
BM25
Search strategy
0,379
0.093
72.1
0.336
0.311
0.268
0.507
0.625
Criteria
0.145
67.0
0.452
0.417
0.345
0.629
0.725
Title
0.230
0.476
0.429
0,376
0.774
MPNet
Abstract
0.516
0.265
63.8
0.556
0.482
0.420
0.692
0.777
Search strategy
0.429
0,181
68.6
0,400
0.372
0.328
0.614
0.699
Criteria
0.545
0.216
$8.5
0,514
0.488
0.393
0.691
0.784
the MiniLM model, using eligibility criteria leads to superior scores in INR@95%% and three other evaluation measures. This indicates that larger models are beginning to effectively leverage the criteria information. Notably, the top-performing model in the zero-shot setting, MiniLM using SLR eligibility criteria, achieves TN R095% equal to almost 0.55. This means that, on average, this model can remove more than half of the true negatives when achieving recall of 95%. Exploring more advanced language models might reveal the potential for further harnessing the criteria information.
CSMED-FT: full text classification dataset
In this section, we introduce CSMED-FT full text screening dataset and present baseline experiments.
5.1 Dataset creation
LLM advancements have enabled processing long text snippets 7 95 55 21 Commercial tools now support inputs of up to 32k [6]] or even up to 100k tokens 4. propose CSMED-FT, the full text screening dataset to enable research associated with the comprehensive understanding of very long documents, and evaluate such capabilities. We first gather full text versions of publications from CSMED SLRs, and then create the appropriate setting with canonical splits.
We SemanticScholar and CORE 35 34 This process successfully finds URLs to, on average, 27% of all included and excluded publications from SLRs. After downloading full text PDFs, we use GROBID documents into X.L format.
APIS find URLs to open-access full text documents.
parse the content of these
We establish canonical splits considering the timestamps, such that the newest reviews belong the test set. Specifically, we select 31 Cochrane reviews published in the last year (between 01/06/2022 and 31/05/2023) create for the development set, and 176 reviews (listed by Scells et al. 77 as the training set. Filtering out reviews with no associated available full text publications results in 148/36/29 reviews in train/dev/test splits.
set, another 60 reviews (mentioned in Nussbaumer-Streit al. (57)
Details of CSMED-FT FT presented in Table 4 We also release subset of 50 randomly selected documents from the test set as SEMMED-FT-TEST-SMA.... creating prompt for LLMs with an input of few thousands tokens is feasible albeit costly See Appendix D for further details on the creation of CSMED-FT.
At the moment of writing this publication,
Table 4: Details of the CSMED-FF FT dataset. Column ##oocs' refers to the total number documents included in the dataset and 'Hincluded' mentions number of included documents on the full text step. CSMEED-TT-EES--SAA. is subset of CSMED-FT-TEE
Avg. #words in document
Avg. #words in review
Dataset name
#reviews
#docs.
#included
% included
CSMED-PT-TRAIN
148
2,053
904
44.0%
4,535
1,493
CSMED-FT-DEV CSMED-FT-TEST CSMED-FT-TESTTSSALL
36
644
202
31.4%
4,419
1,402
29
636
278
43.7%
4,957
2,318
16
22
44.0%
5,042
2.354
AAccording to the OpenAl model pricing on 22/09/2023, screening 500 full text documents with the GPT-4- 32k model would cost more than 400 USD.

---

CSMED-FT FT could be proxy for a very long document natural language inference (NLI) task. Popular NLI datasets (SciTail [33,, McTest [66] DocNLI (93) contain both hypotheses and premises of an average length considerably shorter than 1,000 words; whereas in CSMED-FT, the premise (review protocol) has an average length of more than 1,000 words, and the hypothesis (publication) contains more than 4,000 words.
5.2 Experiment
We present how CSMED-FT can be used to evaluate LLMs capabilities making eligibility decisions on very long documents. We run experiments both on fine-tum fine-tuning of Transformer models and zero-shot prompting of GPT models.
Model selection As the combined input size of systematic review and publication can be big (9,246 mean number of tokens on a training split measured with a GPT-4 tokeniser), we only select models that allow inputs of at least 4k tokens context. We fine-tune the open-domain Longformer and Big BigBird, and domain-specific models pre-trained on clinical data: Sinical-BigBiin and ClinicalLongforme For zero-shot evaluation, we select via OpenAI API. GPT-4-8k and GPT-3.5-turbo- 16k are the only models capable of handling more than 4k-input tokens, with context window size of 8k and 16k tokens respectively.
GPT-4-8k and EPT-3.5-turbo-- 16k accessed
Preprocessing and evaluation For all models, we concatenate the screening protocol with each publication; we truncate the review description text to half of the available context window (2,000 tokens for 4k models, 4,000 tokens for 8k model and 8,000 tokens for 16k model) and complete the input with a publication.
For GPT models, if a whole publication text does not fit the context window, we run multiple predictions with sliding window and aggregate the results. the case of GPT-3.5-urbo-16. model, only for 4 out of 50 documents the model was unable to process the full text of combined review and publication inside one prompt.
We fine-tune the Transformer models on CSMED-FT-TRAIN for four epochs and run evaluation on CSMED-FF -D--FF--EV Due the budget limitation, for the GPT-4-8k model, we run the evaluation only on CSMED-FT-TEE -ESST--F-ESSSSSAAA (see Appendix G for further details on experimental settings). Finally, we evaluate the models using macro-averaged Precision, Recall and Fi-score measures.
Results of the full text experiment are summarised in Table 5 On CSMED-FF FEDD-FF-TESSS TEST- SMALL, GPT-4-8k strongly outperforms other models. However, this difference is not statistically significant. The GPT-3.5-turbo-16. achieves the highest Precision; this improvement can be attributed the model's expanded context window and the limitations other GPT-based models have with our simple aggregation rules. However, this might also be caused by overfitting towards the positive class, as this model includes almost twice many publications as other models. On CSMED-FT-TESS set, Clinical Birgd,,, significantly outperforms zero-shot GPT-3.5 model and pre-trained models based on the LongFormer architecture.
Results
Interestingly, both BigBird- based models outperform their counterparts using the Longformer archi- tecture. The typical overall tendency comain-pre-ttained models achieving higher scores over their open-domain counterparts is also preserved. We believe that fine-tuning the Transformer models first on larger NLI/QA corpora could help improve the results.
Discussion
In this paper, we have addressed the challenge of standardised evaluation in CS automation. By revisiting existing screening datasets, we evaluated their suitability as benchmarks in the context of modern ML methods. Our analysis revealed limitations such small size and data leakage issues.
To overcome these challenges, we introduced CSMED, a meta-dataset consolidating nine publicly released collections, providing programmatic access to 325 SLRs. CSMED serves as comprehensive resource for training and evaluating automated citation screening models and can be used for tasks that involve textual pairs classification, question answering and NLI. Additionally, we included

---

Table 5: Results of the full text screening experiment averaged over documents. The statistical significance was assessed with a McNemar's t-test (p 0.05) with Bonferroni correction for multiple testing. Clinical-Big Biriicall-igBBiee BigBird on the CSMED-FT- FE-SSTT split showed statistically significant improvements and GP7-3.5-turbee
compared to the stratified random baseline, Longformer, Sfinical-Longfrre 16k, indicated by T. Stratified baseline is averaged from 100 different random seeds. %% incl.' describes the percentage of documents predicted as relevant by models.
CSMED-FT-TEST-SMAL
CSMED-FT-TESS
incl. Precision
Recall
Fl-score
% incl.
Precision
Recall
Fl-score
ORACLE
44%
43.7%
stratified random
50%
0.497
0.498
0.495
0.499
0.499
0.498
'include all'
100%
0.220
0,500
0.306
100%
0.219
0.500
0.304
Longformer 7
40%
0.467
0.468
0.466
40.4%
0.398
0.400
0.398
BigBird-roherta-baas
95
42%
0.572
0.571
0.572
45.1
0.575
0.575
0.575
Limical-Longfommer
48
36%
0.547
0.544
0,542
35.1%%
0.436
0.441
0.435
Clinical-BigBing [4
36%
0,590
0.584
0,583
32.8%
0.623
0.611
0.609
54%
0,585
0.586
0.580
GPT-4-8k-0314
58%
0.674
0.672
0.660
GPT-3.5-turbo-166
80%
0.712
0.638
0.576
75.9%
0.538
0.528
0.475
new dataset within CSMED for evaluating full text publication classification and conducted initial experiments showing that there a room for improvement in understanding long contexts.
The focus of CSMED providing unified access over number of diverse citation screening datasets has many benefits. First, the evaluation code can be re-used, making sure that the evaluation is handled properly. Secondly, integration with the BigBio framework enables quick prototyping of prompts. We also improve the documentation for existing datasets and provide a comprehensive data card for CSMED-FT. Our extended version of CSMED is also deduplicated. Finally, it is step towards providing a multi-domain SLR dataset and bridging the gap between IR and NLP research in the domain of screening automation, enabling direct comparisons of the methods.
Limitations and future work While we attempted to extract the data protocols as accurately as possible, extraction of data was not possible for all previous reviews. This was primarily due to the changing standards in Cochrane reviews throughout the years. In future work, ideally, direct access to Cochrane metadata would be needed to make sure that all information is covered. Even though the PubMed publications most probably will not change, what can change is the API and scripts necessary to download the data. There exists also the possibility that one of the sources will introduce a restriction on using their data for training and evaluation of machine learning models. We tried to further mitigate this potential issue by selecting open-access SLRs produced by Cochrane. Finally, we acknowledge that using machine assistance for citation selection can raise concerns about research quality, emphasising the vital role of human oversight throughout the process.
Future work will focus on further improving data quality, connecting the output reviews from screening tools like CRUISE-Screening [39], adding datasets covering other domains and different SLR tasks and designing a dataset for prospective evaluation of review automation which could ensure no data leakage [12]. For the prospective dataset, predictions could be made soon as the protocol published, and the gold standard data becomes available when the review is eventually published, albeit with the drawback of potentially long waiting time for review publication.
7 Conclusion
Our paper introduces CSMED, a meta-dataset that addresses the lack of standardisation in SLR automation. By consolidating datasets and providing a unified access point, CSMED facilitates the development and evaluation of automated citation screening and full text classification models. We believe it has the potential to advance the field and lead to more robust automated SLR systems. We envision CSMED as a living, evolving collection, and we invite researchers to contribute to expanding with SLR datasets from other domains.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval DoSSIER (H2020-EU.1.3.1) ID: 860721).
References
[1] http:////.hhhuo.o..oo//matttttaa
2008-2022.
[2] Amal Alharbi and Mark Stevenson. A dataset of systematic review updates. SIGIR 2019 - Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1257-1260) 2019. doi: 1..11//31118443331133
[3] Amal Alharbi and Mark Stevenson. Refining Boolean queries to identify relevant studies for systematic review updates. Journal of the American Medical Informatics Association, 27(11):16588 1666, 10 2020. ISSN 1527-974X. doi: 10. 10.1093famii//cca144. URL httph:ttt//// //tp:s:///diiii org/10. (at99//jamaa//aaaaaa
[4] Anthropic. claude-v 13--10kk Blog post "Introducing 100K Context Windows'.
https:/ htte:://wwn /ww
anthropic. cecbbropc.coo/iinee wiliiddee/0/oooooo// -ww.oe--oottttt--iicioiie
2023.
[5] Maisie Badami, Boualem Benatallah, and Marcos Baez. Adaptive search query gemeration and refinement
in systematic literature review. Information Systems, page 102231, 2023. ISSN 0306-4379. doi: https://
doi.org/ I0. 1016/j.i 102231. /tt..ttpppp.w.....aaaaec.c..o.ca http://ccaaaaaaaeeooooo////oca/a
80306437923000574
[6] Elias Bassani. retriv: A Python Scarch Engine for the Common Man, May 2023. URL https://githubb
com/ Att///ennna/reiiii/a/
[7] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 25X:::004.0555500 2020.
[8] Kathrin Blagec, Jakob Kraiger, Wolfgang Frühwirt, and Matthias Samwald. Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals. 1 2022. URL
httt:://aarz..rrr/baa/2221111 07040v1
[9] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[10] Justin Clark, Paul Glasziou, Chris Del Mar, Alexandra Bannach-Browe Paulina Stehlik, and Anna Mae Scott. full systematic review completed in 2 weeks using automation tools: study. Journal of clinical epidemiology. 121:81-90. 2020.
[11] A. M. Cohen, W. R. Hersh, K. Peterson, and Po Yin Yen. Reducing workload in sys- tematic review preparation using automated citation classification. Journal of the Ameri- can Medical Informatics Association, 13(2):206-219, 3 2006. ISSN 10675027. doi: 10. 1197/amia.M1999. URL /ttp///ptiitttt/ httee//artcceeee//1447553333 http//44555///////////////// pmc/ hta.cc/es//ccc14474//eeeeoo abstracthttps:// wwww/ ncbi nlm. nih,
[12] Aaron M Cohen, Kyle Ambert, and Marian McDonagh. A prospective evaluation of an automated classifi- cation system to support ciddnnce-based medicine and systematic review. In AMIA annual symposium proceedings, volume 2010, page 121. American Medical Informatics Association, 2010.
[13] Siddhartha R Dalal, Paul G Shekelle, Susanne Hempel, Sydne J Newberry, Aneesa Motala, and Kanaka Shetty. A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Medical Decision Making, 33(3):343 3155)33333353---0 2013.
[14] Jay Young, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. Ms^2: Multi-document summarization of medical studies. In EMNLP. 2021.
[15] Li Dong, Nan Yang. Wenhui Wang. Furu Wei. Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wir Wuen Hon. Unitied language model pre- prettraining for natural language understanding and generation. Advances in neural information processing systems, 32, 2019.

---

[16] Jason Fries. Leon Weber. Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Sunny Kang, Rosaline Su, Wojciech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Oit, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang. Alison Callahan, Daniel León Perifián, Théo Gigant, Patrick Haller, Jenny Chim, Jose Posada, John Giorgi, Karthik Rangasai Sivara- man, Mare Pamics, Mariannaa Nezhurina, Robert Martin, Michael Cullan, Moritz Freidank, Nathan Dahlberg, Shubhanshu Mishra, Shamik Bosc, Nicholas Broad, Yanis Labrak. Shlok Deshmukh, Sid Kiblawi, Ayush Singh, Minh Chien Vu, Trishala Neeraj. Jonas Golde, Albert Villanova del Moral, and Benjamin Beilharz. Bighio: A framework for data-centric biomedical natural language pro- cessing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Ad- vances in Neural Information Processing Systems, volume 35, pages 25792 25792-25806. Curran Asso- ciates, Inc.. 2022. URL https: p/tpopee///prooceediigg.... www.eie....ppaapepppppeeeeeeepee cc/pappe__aaee 1588222977aff4444444411666666663
http//ppppr////0/ Paper- Dateerr--attttts__aaa Bet.hmaad_enhaaksssssss__a pdf
[17] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https://doi. org/ 10.
5678
[18] Timnit Gebru, Jamic Morgenstem, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. Datashects for datasets. Commun. ACM, 64(12):86 92, 2021. doi: 10. 144534458733. URL https::////i..rrg 10. 1145/ 3458723
[19] Maura R Grossman, Gordon V Cormack, and Adam Roegiest. TREC 2016 Total Recall Track Overview. In TREC, 2016.
[20] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Heal., 311):2:1-2:23, 2022. doi: 10. 145/3458777 URL
attps://doi. org/10. 0145/345877
[21] Mandy Guo, Joshua Ainslie, David Uthus. Santiago Ontanon, Jianmo Ni. Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:211..07916. 2021.
[22] Abdelhakim Hannousse and Salima Yahiouche. A semi-automatic document screening system for computer science systematic reviews. In Mediterranean Conference on Pattern Recognition and Artificial Intelligence, pages 201 215.2 Springer, 2022.
[23] Kazuma Hashimoto, Georgios Kontonatsios, Makoto Miwa, and Sophia Ananiadou. Topic detection using paragraph vectors to support active learning in systematic reviews. Journal of Biomedical Informatics, 62: 59 65, 8 2016.
[24] Julian PT Higgins. James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J Page, and Vivian A Welch. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons, 2019.
[25]
hitz-zentroa. LM Contamination Index GitHub repository, 2023. URL https://github. com/
Bitz-zentroa/ill amm-onntaminationn
[26] Jingwen Hou, Xiaochen Wang. Jean- Jean-Jacques Dubois, R. Byron Rice, Amanda Haddock, and Yue Wang. Extreme systematic reviews: A large literature screening dataset to support environmental policymak- ing. In Proceedings of the st ACM International Conference on Information & Knowledge Manage- ment, CIKM '22, page 4029-400 0029-4033, New York, NY, USA, 2022. Association for Computing Machinery. URL httph:tttttt.oooooetoooo org/ 1145/3511808
ISBN 9781450392365. doi: 10. 3557600
[27] Brian Howard, Jason Phillips. Kyle Miller, Arpit Tandon, Deepak Mav. Mihir R. Shah, Stephanie Holmgren, Katherine E. Pelch, Vickie Walker, Andrew A. Rooney, Malcolm Macleod, Ruchir R. Shah, and Kristina Thayer. SWIFT-Review: A text-mining work- bench for systematic review. Systematic Reviews, 5(1):1-16, 5 2016. ISSN 20464405. doi: 10.1186/113343301---222
URL https://link. springer com/ wttp//artcleeee.... 1186/ s13643- 016 0263 -wt.0022663---//// ahttps://iiti www.nnk...ringe....... com/ wwwp://aii///e..oot........ 11tp:///1.1111111111113133333111
[28] Brian E. Howard, Jason Phillips, Arpit Tandon, Adyasha Maharana. Rebecca Elmore, Deepak Mav, Alex Sedykh, Kristina Thayer, B. Alex Merrick, Vickie Walker, Andrew Rooney, and Ruchir R. Shah. SWIFT- Active Screener: Accelerated document screening through active learning and integrated recall estimation. Environment International, 138:105623 2020. ISSN 0160-4120. doi: 10.1016/.ENVINIT
12

---

[29] Akers Jo, Aguiar-Ibáäte Raquel, Burch Jane, Chambers Duncan, Eastwood Alison, Fayter Debra, Hempel Susanne, Light Kate, Rice Stephen, Rithalia Amber, Stewart Lesley. Stock Christian, Wilson Paul, and Woolacott Nerys. Systematic Reviews; CRD's guidance for undertaking reviews health care. CRD, University of York, York, 2009. ISBN
URL|wwR. WeL. york. ac. uk/ /nddss///rdd
[30] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. cally assisted reviews in empirical medicine overview. CEUR Workshop Proceedings, 1866:1- 29, 9 2017. ISSN 1613-0073. URI. https::/ppreporrtttatt strath. ac. ac.akc..eeeeee.. ww/en/ wwt.u//e/puliiaaaionn. publications/ clef-2017 wtt.eehhnnnnliiallllaaiiiieeeeee -tppeerrreeee------eeieiriaaaaaa
CLEF 2017 technologi-
[31] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. CLEF 2018 technologi-
cally assisted reviews empirical medicine overview. CEUR Workshop Proceedings, 2125.
2018. ISSN 1613-0073. URL / straath...strat..e.. ac wwwwwk//eeeuuueee http:caalliininnso.o.c clef- 2018-techno www.noooooicalllyyaasseeeee..e.e wwwplly-sssssseddrreeeeeee reviews- in empirical-medii medicine-
[32] Evangelos Kanoulas. Dan Li, Leif Azzopardi, and Rene Spijker. CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview. In CLEF, 2019.
[33] Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[34] Petr Knoth and Zdenek Zdrahal. Core: three access levels underpin open access. D-Lib Magazine, 18 (11/12), 2012. URL http:/ttt///oooooooeeoooo
ac uk/35755/
[35] Petr Knoth, Drahomira Herrmannova, Matteo Cancellieri, Lucas Anastasiou, Nancy Pontika, Samuel Pearce, Bikash Gyawali, and David Pride. Core: A global aggregation service for open access papers. Nature Scientific Data, 10(1):366, June 2023.
[36] Georgios Kontonatsios, Austin J. Brockmeier, Piotr Przybyla. John McNaught, Tingting Mu, John Y. Goulermas, and Sophia Ananiadou. A semi-supervised approach using label propagation support citation scrcening. Journal of Biomedical Informatics, 72:67-76, 2017.
[37] Georgios Kontonatsios, Sally Spencer, Peter Matthew, and loannis Korkontzelos. Using neural network- based feature extraction method to facilitate citation screening for systematic reviews. Expert Systems with Applications: X, 6: 6:000030,7 2020. ISSN 25901885. doi: 10. ww..0066..ssax.22222...... 202000000000
[38] Wojeiech Kusa, Allan Hanbury, and Petr Knoth. Automation of citation screening for systematic literature reviews using neural networks: A replicability study. In Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog. Kjetil Nørvåg, and Vinay Setty, editors, Advances in Information Retrieval, pages 584-598, Cham, 2022. Springer International Publishing. ISBN 978-3-0330-99736666 URL https://arxiv org/abs////// a20/ 07534v1
[39]
Wojciech Kusa, Petr Knoth, and Allan Hanbury. CRUISE- Screenin: Living Literature Reviews Toolbox. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Manage- ment, CIKM '23, page 5071 15575,, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701245. doi: 10. 1414448838800361473366 3614736
URL httpss/tt:///oo wrgg 10. 1145/3588780
[40] Wojeiech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. An Analysis of Work Saved over Sampling in the Evaluation of Automated Citation Screening in Systematic Literature Reviews. Intelligent Systems with Applications, 18:200193, 2023. ISSN 2667- 3053. doi: https://doii org/ 10. 10t.11.....1...... 1016//.ssa..2U22 URL /ttps:/tss://ww. www.:sscieneo wcwenceddeect...............e..e com/ http://cienceeeae/ httpa////w...a.eaeae////e///////
[41] Wojeicch Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. VoMBaT: A Tool for Visualising Evaluation Mcasure Behaviour in High-Recall Search Tasks. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23), page 5, Taipei, Taiwan, July 23-27 2023. ACM. URL htt.::ttt/://att http://ddo.ooggg//// 10. 1141 /1144/5555611.../ 3591802
[42] Wojeiech Kusa, Oscar F. Mendoza, Petr Knoth, Gabriella Pasi, and Allan Hanbury. Effective Matching of Patients to Clinical Trials using Entity Extraction and Neural Re-ranking, Journal of Biomedical Informatics, JBI, 2023.
[43] Wojeiech Kusa, Guido Zuccon, Petr Knoth, and Allan Hanbury. Outcome-based evaluation of systematic review automation. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '23, page 125-133, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9798400700736. doi: 10. 00.1145/3578337.36000 135. URL
1145/
3578337.36056
13

---

[44] Eric W Lee and Joyce C Ho. Pgb: A pubmed graph benchmark for heterogeneous network representation learning. arXiv preprint arXiv:2305.022. 022699 2023.
[45] Eric W Lee and Joyce C Ho. Sr-comber: Heterogeneous network embedding using community multi-view enhanced graph convolutional network for automating systematic reviews. In European Conference on Information Retrieval, pages 553-568. Springer, 2023.
[46] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence- to sequence pre-training for natural language generation. translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Association for hattpp://aclantaeeaa: .rggggg/
Computational Linguistics. doi:
2020 acl acl-main. 703
[47] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. arXiv preprint arXiv:2109.02844 2021.
[48] Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin Wang, and Yuan Luo. Clinical-longformer and blinical-bigbirl: Transformers for long clinical sequences. arXiv preprint arXiv:2201.J 1838, 2022.
[49] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. 7 2021. URL http:tt:: //cttp://arxiv org/ abs/ 2107 13586v1
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2018
[51] lain Marshall, Joël Kuiper, and Byron C Wallace. Robotreviewer: evaluation of system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association, 23(1):193-001, 2016.
[52] Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O' Blennis. A new algorithm for reducing the workload of experts in performing systematic reviews. Journal of the American Medical Informatics Association, 17(4))446-453, 7 2010. doi: 10.1 /16/////...000.000.......
[53] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approxi- mation and projection. The Journal of Open Source Software, 3(29):861, 2018
[54] Makoto Miwa, James Thomas, Alison Mara-Eves, and Sophia Ananiadou. Reducing systematic review workload through certainty-hased screening. Journal of Biomedical Informatics, 51:242-253, 10 2014. ISSN 1532-0464. doi: 10.1016/..BI..0144, JBL...1...6. 005.
[55] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random- sacdoss-ccces infinite context length for transformers. arXiv preprint arXiv:2305. 16300, 2023.
[56] Christopher Norman. Syxtematic review automation methods. PhD thesis, Université Paris-Saclay Universiteit van Amsterdam, 2 2020. URL https://tel. arcchie/ ouvervss. frr/ee-000000620
[57] Barbara Seashaumer-Steeint Irma Klerings, Gernot Wagner, Thomas L. Heise, Andreca I. Dobrescu, Susan Armijo- Olivo, Jan M. Stratil, Emma Persad, Stefan K. Lhachimi, Megan G. Van Noord, Tarquin Mittermayr, Hajo Zeeb, Lars Hemkens, and Gerald Gartlchner. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epiidemiologgicll study. Journal of Clinical Epidemiology. 102:1-11, 2018 ISSN 0895-4356. doi: https://do..rr//000001116 1016/ www.:iipii.i.. 201.inep.2018.0..0222 //wpk.sciencee... www.rssciinceiiretcccm wttpeeetttcc/ccceeeeeeeeeeeeeeee http:///aaaaceeataaeee/////////.
URL https
[58] Barbara Nusshaumer-Strect Moriah Ellen, Irma Klerings, Raluca Sfeteu, Nicoletta Riva, Mersiha Mahmić- Kaknjo, Georgios Poulentzas, P Martinez, Eduard Baladia, Liliya Eugenevna Ziganshina, et al. Resouree use during systematic review production varies widely; scoping review. Journal of Clinical Epidemiology. 139:287-296, 2021.
[59] Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, lain J Marshall, Ani Nenkova, and Byron Wallace. corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2018, page 197. NIH Public Access, 2018
[60] Alison O' Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews, 4(1):5, 1 2015, ISSN 20464053. doi: 011188604464003--44
14

---

[61] OpenAL. Gpt-4 technical report. 2023.
[62] Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. In- BoXBART: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACI. 2022, pages 112-128, Seattle, United States, July 2022. Association for Computational Linguistics. doi:
org/2022. wiwt:/inndiinnnnnnee naac1.10
[63] Mihir Prafullsinh Parmar. Automation of title and abstract screening for clinical systematic reviews. Master's thesis, Arizona State University, 2021. URL htttps://kepp.11: asu, leweeeelleeeeeeeeee _tt/sy/ylyslllm/m
fedora/c7PParaa aea_asuu_0000 T1L_11011111010111111211 21179.pdf
[64] Yifan Peng, Shankai Yan, Zhiyong Lu. Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58-65, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 20.18633/1//1// 19-50066. URL httts:://aclancholooat
org/Nn 5006
[65] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNE 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference Natural Language Processing, Proceedings of the Conference, pages 3982 3992, 8 2019. URL https:///a/ wrgg 1908. 10084v1
[66] Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. October 2013. Association for Computational Linguistics. URL htcdps://aclantbooogg..ooo http:/llgg..gg/111111100 1000000
[67] Kirk Roberts, Dina Demner-Fushman. Ellen M Voorhees, William Hersh, Steven Bedrick. Alexander J Lazar, and Shubham Pant. Overview the tree 2017 precision medicine track. In TREC, 2017.
[68] Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, Steven Bedrick, and Willian R Hersh. Overview of the tree 2021 clinical trials track. In Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021), 2021.
[69] Stephen E Robertson. Steve Walker, Susan Jones, Micheline M Slancock-Beaulies Mike Gatford, et al. Okapi at tree-3. Nist Special Publication Sp, 109:109, 1995.
[70] Harrisen Scells and Guido Zuccon. Generating hetter queries for systematic reviews. In The 41st international ACM SIGIR conference on research development information retrieval, pages 475-4844 2018.
[71] Harrisen Scells, Guido Zuecon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva. A test collection for evaluating retrieval of studies for inclusion in systematic reviews. SIGIR 2017 Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1237-1240, 8 2017. doi: 10. 041144/777116.00000
[72] Harrisen Scells, Guido Zuccon, and Bevan Koopman. A comparison of automatic boolean query formula- tion for systematic reviews. Information Retrieval Journal, 24(1):3-28, 2021.
[73] Gaurav Singh, James Thomas, and John Shawe-Taylor. Improving active learning in systematic reviews. arXiv preprint ar0iv:1801. 09496, 2018
[74] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Tryna Gurevych. BEIR: Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. 4 2021. URL. https://arxil org/ abs/ wwtp////...... 08666344
[75] Guy Tsafnat, Adam Dunn, Paul Glasziou, and Enrico Coiera. The automation of systematic reviews, 2013.
[76] Guy Tsafnat, Paul Glasziou, Miew K. Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera. Systematic review automation technologics, 4 2014. ISSN 20464053.
[77] Guy Tsafnat, Paul Glasziou, George Karystianis, and Enrico Coiera. Automated screen- ing of research studies for systematic reviews using study characteristics. Systematic Re- views 2018 7:1, 7(1):119, 4 2018. ISSN 2046-4053. doi: 10. l 186/81 36413333----11---7777 URL
https: //in:////innk.. springer wtt.cooo//at/e httmm//rtiieeseeeeemme /10. 1186/ 018 0724 7https 7/link. sainnk..pringgr.ccmm www.o//atiieeeiicc 10. 01111111111100018-00-772222
[78] Raymon van Dinter, Cagatay Catal, and Bedir Tekinerdogan. A decision support system for automating document retrieval and citation screening. Expert Systems with Applications, 182, 11 2021.
15

---

[79] Raymon van Dinter. Cagatay Catal, and Bedir Tekinerdogan. A Multi-Channe Convolutional Neural Network approach automate the citation screening process. Applied Soft Computing, 112:107765, 11 2021. ISSN 1568-4946. doi: 10. A.001//..ASO..222.....
[80] Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. Automation of systematic litcrature reviews: A systematic literature review. Information and Software Technology, 136:106589. 8 2021. ISSN 09505849, doi: /011///.nfsoff.22111 106589. MRRLLhttt:://inkttii www.wib..laeeeeeeee...oooo..eeoe com/retrieee/ /eteeeeeeeeeeeeeeeerrreteeeeeeee pii pi1/90950558922000690
[81] Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos. Active learning for biomedical citation screening. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 173-181, 2010. doi: 10. 01.114/1135844.1838888
[82] Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. Semi- automated screening of biomedical citations for systematic reviews. BMC bioinformatics, 11(1):1-11, 2010.
[83] Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. Who should label what? instance allocation in multiple expert active learning. In Proceedings of the 2011 SIAM international conference data mining, pages 176- 187. SIAM, 2011.
[84] Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and lain James Marshall. Generating (factual?) narrative summaries of rets: Experiments with neural multi-document summarization. AMIA Annual Symposium, ahs/2008. 1293, 2020.
[85] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, GLUE: A multi-task beachmark and analysis platform for natural language understanding. arXiv preprint arXiv: arXiv:1804.072 18w..0044776) 2018.
[86] Lucy Lu Wang, Jay De Young, and Byron Wallace. Overview of MSLR2022: A shared task on multi- document summarization for literature reviews. In Proceedings of the Third Workshop on Scholarly Document Processing, pages 175-180, Gyeongju, Republic of Korca, October 2022. Association for Computational Linguistics. URL https: htttpp::/aaclannhologg.. wwtp:/t2../222.e.. sdp-1.20
[87] Shuai Wang, Harrisen Scells, Ahmed Mourad, and Guido Zuccon. Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study. 12 2021. URL https://arxiv.. wwwaarrwiv....iaaeee.oooe www.:////////////eee aa/aab///1111 2112 04090v1
[88] Shuai Wang, Harrisen Scells, Justin Clark, Bevan Koopman, and Guido Zuccon. From little things big things grow: A collection with seed studies for medical systematic review literature search, arXiv preprint arXiv:2204.03096, 2022.
[89] Shuai Wang. Hang Li, and Guido Zuccon. Mesh suggester: A library and system for mesh term suggestion for systematic review boolean query construction. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. WSDM '23, page 1176-1179, New York, NY, USA, 2023. Association for Computing Machinery, ISBN 978145039409 https://dol orgg/.orr111 10. 1145/3539553 3573025
doi: 011445/333997.3573000 URL
[90] Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. Can CharGPT write a good boolean query for systematic review literature search? arXiv preprint arXiv:2302.0344:4 2023.
[91] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolahashi, Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok. Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson. Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit. Neeraj Varshney. Phani Rohitha Kaza, Pulkit Verma, Ravschaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- NaturalInstructioon:: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL httpa://aclanthology org/ 2022. emnlp-main.34 340
[92] Leon Weber, Mario Sänger, Jannes Münchmeyer, Maryam Habibi, Ulf Leser, and Alan Akbik. Hunflair: an casy-to-use tool for state-of the biomedical named entity recognition. Bioinformatics, 37(17): 2792-2794, 2021.
16

---

[93] Wenpeng Yin. Dragomir Radev. and Caiming Xiong. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguisticx: ACL- IJCNLP 2021. pages 4913-4922, Online, August 2021. Association for Computational Linguisties. doi:
10. 12.188631/22221.6mm findingg-- ac1.
htt.ttpp://alllatoooogg.//////// wrg//021...22.2... 2021 fidinggs- acc.nniinnnn..i.....e.... 435
[94] Hye Sun Yun, Jain J Marshall, Thomas Trikalinos, Byron C Wallace. Appraising potential uses and harms of LLMs for medical systematic reviews. arXiv preprint arXiv:2305.11828, 2023.
[95] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang. et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283 17297, 2020.
[96] Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang. Buzhou Tang, and Qingeai Chen. CBLUE: A Chinese biomedical language understanding evaluation benchmark. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 7888-7915, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 120.2a//o/.2....oooogg..... URL attps://aclanttoloo. org/2022 ael-long..05 544
17


Reviews:
[{"review_id": "KmGBTbj6X2", "reviewer": "Reviewer_6uGL", "summary": "", "questions": "", "limitations": "Undiscussed but reasonable.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- SLR is an important and impactful problem, creating resources for methodological communities to benchmark on this task is a meaningful contribution.\n- The papers analysis of numerous past SLR datasets and the inconsistencies of their experimental setup is a useful analysis.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "L1vDJmlI6l", "reviewer": "Reviewer_TzT9", "summary": "", "questions": "", "limitations": "The authors of this paper do address some of the limitations of how data must be downloaded due to copyright restrictions of the data providers, which may lead the dataset to break in the future without maintenance.  \n\nOne possible negative impact of this paper is the reliance on machines to perform the work of citation screening for systematic reviews, which may have negative impacts on the quality of the research presented in such reviews. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* This paper provides a helpful & needed resource for systematic reviews of literature by enabling the comprehensive benchmarking of models used to screen citations for inclusion.  Not only does it unify the formatting of multiple datasets, but it also augments them with meaningful metadata from Cochrane to more closely match the data available for citation screening with protocols used in real life.  Doing so more closely matches the described task with clinical practice, improving the meaningfulness of benchmark results.\n* The baseline examined for this benchmark provide a sufficiently diverse test of different methods that would reasonably be used for citation screening for systematic reviews.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "YiIk4HZCJ1", "reviewer": "Reviewer_h67z", "summary": "", "questions": "", "limitations": "NA", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "Provided a meta-collection of SLR dataset and curated SLRs for full text screening.\n\nBenchmarked methods for full text screening.\n\nSuch resources would be useful to the research community though the current performance is not very satisfying.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Q1nuxJO6de", "reviewer": "Reviewer_zrvA", "summary": "", "questions": "", "limitations": "1. In general, the contribution is not good enough for publication at the NeurIPS benchmark. The unified format of previous filtered datasets might be useful for people working in this community, but the contribution is incremental.\n\n2. the experimental part lacks deeper analysis, such as a failing case or detailed analysis of the problem.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The contribution of a unified format is the biggest contribution, saving efforts for future people.\n\n2. They identify some issues or errors with previous benchmarks, which is valuable for future research.\n\n3. The preprocess of all the datasets to accommodate popular Python libraries makes them easily usable.", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper conducts a review and analysis of citation screen datasets for systematic review automation. The authors consolidate and release a meta-dataset resource that includes collections from 9 previous resources, called LivSB, and conduct baseline experiments on screening classification using this dataset. 

Pros:
- Introduces a benchmark for SLR paper screening by collecting data from 9 resources, unifying their formats, and augmenting the dataset with additional data/metadata
- Provides baseline performance of several reasonable classification models
- Provides an analysis of past SLR screening datasets

Cons:
- Misalignment between title and content: One high level recommendation from the AC is to alter the title to more closely describe the scope of the work in the paper. This manuscript does not investigate or propose standarized evaluation for SLR automation; rather, it introduces a benchmark dataset for SLR publication screening. The title misconstrues the paper content and sets unrealistic expectations. I strongly recommend the title be changed.
- There are several other recommendations made by reviewers that must be incorporated in a revision, most notably addressing any data statistics inconsistencies, reporting results on the entire benchmark, reporting statistical significance, and fixing issues with the Streamlit app (dataset statistics are still broken). 

**I have some reservations about this manuscript, and would recommend conditional acceptance based on addressing the above issues if that were an option. However, since that's not an option, I am recommending acceptance and I am optimistic that the authors will address issues as necessary.**
Decision: Accept (Poster)