SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We carefully examine the subset of SLRs produced by Cochrane, aiming to identify potential enhancements and extensions that would help mitigating the existing limitations of previous datasets. Every Cochrane SLR first registers and publishes the protocol containing the review title, abstract, search strategy and the eligibility criteria. This information is all that human experts need to produce the final review, i.e., they first find the relevant studies and then conduct the meta-analysis of their results. As described in Section 2, the screening process can be also modelled as a question-answering, where every publication is compared against the eligibility criteria in order to make the decision about the inclusion,[6] similar to the clinical decision support task of matching clinical trials to patients [67, 68].
To expand CSMED, we searched the Cochrane Library<sup>7</sup> for all SLRs from the meta-dataset based on the Cochrane review ID and take their latest open-access version. We extract available information about the review: review title and abstract, eligibility criteria, search strategy and references. Cochrane reports a list of included and excluded publications at the full text screening stage (this can be treated as approximately all included publications during the title and abstract screening stage). Moreover, all excluded publications have a reason for exclusion selected by a reviewer. As the original relevance judgements were limited to publications from the PubMed database, we assign PubMed IDs to these publications. We also define appropriate BigBio data loaders so the task can be seen as question-answering or textual pairs classification task.
Table 2: Details of the CSMED expanded meta-dataset. Column '#docs' refers to the total number of documents included in all SLRS within the dataset, '#included' mentions number of included documents on the title and abstract screening stage and 'Avg. %included' the percentage of included publications averaged from all reviews.
Dataset name	#reviews	#docs	#included	Avg. #docs	Avg. % included	Avg. #words in document
CSMED-TRAIN-BASIC	30	128,438	7,958	4,281	9.6%	229
CSMED-TRAIN-COCHRANE	195	372,422	7,589	1,910	21.9%	180
CSMED-DEV-COCHRANE	100	229,376	4,365	2,294	20.8%	201
CSMED-ALL	325	730,236	19,912	2,247	20.5%	195
Details of the new expanded CSMED are provided in Table 2. We were not able to find suitable data for all SLRs, hence the expanded CSMED is smaller than the original meta-dataset. In total, the new expanded dataset consists of 295 unique Cochrane SLRs and 30 non-Cochrane SLRs. The entire set of basic SLRs is designated for training. From the Cochrane reviews, we randomly selected 195 to the training split and the remaining 100 to the development split. We abstain from designating a test split because CSMED aggregates existing datasets. Given the constraints of these datasets, creating a new, unbiased test collection is recommended.
4.3 Baseline experiment
We evaluate five models in a zero-shot setting: two statistical models BM25 and TF-IDF, and three Transformer-based models. Predictions are run on the CSMED-DEV-COCHRANE split, and we test four different input representations using the following sections from the SLR protocol: (1) title, (2) abstract, (3) search strategy and (4) eligibility criteria section. We evaluate models using  $TNR@95\%$  [40, 41],  $nP@95\%$  [41], Last Rel [30, 31, 32],  $nDCG@10$ ,  $MAP$ , Recall at rank  $k$  with  $k$  in {10, 50, 100} ( $R@k$ ). Detailed experimental setup and expanded results can be found in Appendix F.
Table 3 presents the summary results of best-performing statistical and neural models, while Table 9 in Appendix F contains complete results. The MiniLM neural model consistently outperforms the BM25 variant across nearly all query representations and evaluation measures. For the BM25 model, using the systematic review abstract text as a query representation results in the highest performance in all metrics compared to using the SLR title and criteria sections. Conversely, for
In the current approach, we consider only binary relevance (included versus excluded). However, in practice, more categories can be defined by reviewers (e.g. a study can be assigned as a background publication or meta-analysis).
<sup>7</sup>https://www.cochranelibrary.com/cdsr/reviews
Table 3: Summary of results on CSMED-DEV-COCHRANE dataset. Bold values indicate best score.
Model	Representation	TNR@95%	nP@95%	Last Rel	NDCG@10	MAP	R@10	R@50	R@100
BM25	Title	0.469	0.142	72.2	0.438	0.388	0.349	0.623	0.704
Abstract	0.474	0.170	63.6	0.503	0.453	0.379	0.657	0.757
Search strategy	0.379	0.093	72.1	0.336	0.311	0.268	0.507	0.625
Criteria	0.430	0.145	67.0	0.452	0.417	0.345	0.629	0.725
MPNet	Title	0.467	0.230	66.6	0.476	0.429	0.376	0.684	0.774
Abstract	0.516	0.265	63.8	0.556	0.482	0.420	0.692	0.777
Search strategy	0.429	0.181	68.6	0.400	0.372	0.328	0.614	0.699
Criteria	0.545	0.216	58.5	0.514	0.488	0.393	0.691	0.784
the MiniLM model, using eligibility criteria leads to superior scores in TNR@95% and three other evaluation measures. This indicates that larger models are beginning to effectively leverage the criteria information. Notably, the top-performing model in the zero-shot setting, MiniLM using SLR eligibility criteria, achieves  $TNR@95\%$  equal to almost 0.55. This means that, on average, this model can remove more than half of the true negatives when achieving a recall of 95%. Exploring more advanced language models might reveal the potential for further harnessing the criteria information.
5 CSMED-FT: full text classification dataset
In this section, we introduce CSMED-FT full text screening dataset and present baseline experiments.
5.1 Dataset creation
LLM advancements have enabled processing long text snippets [7, 95, 55, 21]. Commercial tools now support inputs of up to  $32\mathrm{k}$  [61] or even up to  $100\mathrm{k}$  tokens [4]. We propose CSMED-FT, the full text screening dataset to enable research associated with the comprehensive understanding of very long documents, and evaluate such capabilities. We first gather full text versions of publications from CSMED SLRs, and then create the appropriate setting with canonical splits.
We use SemanticScholar and CORE [35, 34] APIs to find URLs to open-access full text documents. This process successfully finds URLs to, on average,  $27\%$  of all included and excluded publications from SLRs. After downloading full text PDFs, we use GROBID [1] to parse the content of these documents into an XML format.
We establish canonical splits considering the timestamps, such that the newest reviews belong to the test set. Specifically, we select 31 Cochrane reviews published in the last year (between 01/06/2022 and 31/05/2023) to create a test set, another 60 reviews (mentioned in Nussbaumer-Streit et al. [57]) for the development set, and 176 reviews (listed by Scells et al. [71]) as the training set. Filtering out reviews with no associated available full text publications results in 148/36/29 reviews in train/dev/test splits.
Details of CSMED-FT are presented in Table 4. We also release a subset of 50 randomly selected documents from the test set as CSMED-FT-TEST-SMALL. At the moment of writing this publication, creating a prompt for LLMs with an input of few thousands tokens is feasible albeit costly, $^{8}$  See Appendix D for further details on the creation of CSMED-FT.
Table 4: Details of the CSMED-FT dataset. Column '#docs' refers to the total number of documents included in the dataset and '#included' mentions number of included documents on the full text step. CSMED-FT-TEST-SMALL is a subset of CSMED-FT-TEST.
Dataset name	#reviews	#docs.	#included	% included	Avg. #words in document	Avg. #words in review
CSMED-FT-TRAIN	148	2,053	904	44.0%	4,535	1,493
CSMED-FT-DEV	36	644	202	31.4%	4,419	1,402
CSMED-FT-TEST	29	636	278	43.7%	4,957	2,318
CSMED-FT-TEST-SMALL	16	50	22	44.0%	5,042	2,354
<sup>8</sup>According to the OpenAI model pricing on 22/09/2023, screening 500 full text documents with the GPT-4-32k model would cost more than 400 USD.
CSMED-FT could be a proxy for a very long document natural language inference (NLI) task. Popular NLI datasets (SciTail [33], McTest [66] or DocNLI [93]) contain both hypotheses and premises of an average length considerably shorter than 1,000 words; whereas in CSMED-FT, the premise (review protocol) has an average length of more than 1,000 words, and the hypothesis (publication) contains more than 4,000 words.
5.2 Experiment
We present how CSMED-FT can be used to evaluate LLMs capabilities in making eligibility decisions on very long documents. We run experiments both on fine-tuning of Transformer models and zero-shot prompting of GPT models.
Model selection As the combined input size of systematic review and publication can be big (9,246 mean number of tokens on a training split measured with a GPT-4 tokeniser), we only select models that allow inputs of at least 4k tokens context. We fine-tune the open-domain Longformer and BigBird, and domain-specific models pre-trained on clinical data: Clinical-BigBird and ClinicalLongformer. For zero-shot evaluation, we select GPT-3.5-turbo-0301, GPT-4-8k and GPT-3.5-turbo-16k accessed via OpenAI API. GPT-4-8k and GPT-3.5-turbo-16k are the only models capable of handling more than 4k-input tokens, with context window size of 8k and 16k tokens respectively.
Preprocessing and evaluation For all models, we concatenate the screening protocol with each publication; we truncate the review description text to half of the available context window (2,000 tokens for 4k models, 4,000 tokens for 8k model and 8,000 tokens for 16k model) and complete the input with a publication.
For GPT models, if a whole publication text does not fit the context window, we run multiple predictions with a sliding window and aggregate the results. In the case of GPT-3.5-turbo-16k model, only for 4 out of 50 documents the model was unable to process the full text of combined review and publication inside one prompt.
We fine-tune the Transformer models on CSMED-FT-TRAIN for four epochs and run evaluation on CSMED-FT-DEV. Due to the budget limitation, for the GPT-4-8k model, we run the evaluation only on CSMED-FT-TEST-SMALL (see Appendix G for further details on experimental settings). Finally, we evaluate the models using macro-averaged Precision, Recall and F1-score measures.
Results Results of the full text experiment are summarised in Table 5. On CSMED-FT-TEST-SMALL, GPT-4-8k strongly outperforms other models. However, this difference is not statistically significant. The GPT-3.5-turbo-16k achieves the highest Precision; this improvement can be attributed to the model's expanded context window and the limitations other GPT-based models have with our simple aggregation rules. However, this might also be caused by overfitting towards the positive class, as this model includes almost twice as many publications as other models. On CSMED-FT-TEST set, Clinical-BigBird, significantly outperforms zero-shot GPT-3.5 model and pre-trained models based on the LongFormer architecture.
Interestingly, both BigBird-based models outperform their counterparts using the Longformer architecture. The typical overall tendency to domain-pre-trained models achieving higher scores over their open-domain counterparts is also preserved. We believe that fine-tuning the Transformer models first on larger NLI/QA corpora could help improve the results.
In this paper, we have addressed the challenge of standardised evaluation in CS automation. By revisiting existing screening datasets, we evaluated their suitability as benchmarks in the context of modern ML methods. Our analysis revealed limitations such as small size and data leakage issues.
To overcome these challenges, we introduced CSMED, a meta-dataset consolidating nine publicly released collections, providing programmatic access to 325 SLRs. CSMED serves as a comprehensive resource for training and evaluating automated citation screening models and can be used for tasks that involve textual pairs classification, question answering and NLI. Additionally, we included a
Table 5: Results of the full text screening experiment averaged over documents. The statistical significance was assessed with a McNemar's t-test  $(p < 0.05)$  with Bonferroni correction for multiple testing. Clinical-BigBird on the CSMED-FT-TEST split showed statistically significant improvements compared to the stratified random baseline, Longformer, Clinical-Longformer, and GPT-3.5-turbo-16k, indicated by  $\dagger$ . Stratified baseline is averaged from 100 different random seeds. ' % incl.' describes the percentage of documents predicted as relevant by models.
CSMED-FT-TEST-SMALL	CSMED-FT-TEST
% incl.	Precision	Recall	F1-score	% incl.	Precision	Recall	F1-score
ORACLE	44%	—	—	—	43.7%	—	—	—
stratified random	50%	0.497	0.498	0.495	—	0.499	0.499	0.498
‘include all’	100%	0.220	0.500	0.306	100%	0.219	0.500	0.304
Longformer [7]	40%	0.467	0.468	0.466	40.4%	0.398	0.400	0.398
BigBird-roberta-base [95]	42%	0.572	0.571	0.572	45.1%	0.575	0.575	0.575
Clinical-Longformer [48]	36%	0.547	0.544	0.542	35.1%	0.436	0.441	0.435
Clinical-BigBird [48]	36%	0.590	0.584	0.583	32.8%	0.623†	0.611†	0.609†
GPT-3.5-turbo-0301	54%	0.585	0.586	0.580	—	—	—	—
GPT-4-8k-0314	58%	0.674	0.672	0.660	—	—	—	—
GPT-3.5-turbo-16k	80%	0.712	0.638	0.576	75.9%	0.538	0.528	0.475
new dataset within CSMED for evaluating full text publication classification and conducted initial experiments showing that there is a room for improvement in understanding long contexts.
The focus of CSMED on providing unified access over a number of diverse citation screening datasets has many benefits. First, the evaluation code can be re-used, making sure that the evaluation is handled properly. Secondly, integration with the BigBio framework enables quick prototyping of prompts. We also improve the documentation for existing datasets and provide a comprehensive data card for CSMED-FT. Our extended version of CSMED is also deduplicated. Finally, it is a step towards providing a multi-domain SLR dataset and bridging the gap between IR and NLP research in the domain of screening automation, enabling direct comparisons of the methods.
Limitations and future work While we attempted to extract the data protocols as accurately as possible, extraction of data was not possible for all previous reviews. This was primarily due to the changing standards in Cochrane reviews throughout the years. In future work, ideally, direct access to Cochrane metadata would be needed to make sure that all information is covered. Even though the PubMed publications most probably will not change, what can change is the API and scripts necessary to download the data. There exists also the possibility that one of the sources will introduce a restriction on using their data for training and evaluation of machine learning models. We tried to further mitigate this potential issue by selecting open-access SLRs produced by Cochrane. Finally, we acknowledge that using machine assistance for citation selection can raise concerns about research quality, emphasising the vital role of human oversight throughout the process.
Future work will focus on further improving data quality, connecting the output reviews from screening tools like CRUISE-Screening [39], adding datasets covering other domains and different SLR tasks and designing a dataset for a prospective evaluation of review automation which could ensure no data leakage [12]. For the prospective dataset, predictions could be made as soon as the protocol is published, and the gold standard data becomes available when the review is eventually published, albeit with the drawback of a potentially long waiting time for review publication.
Our paper introduces CSMED, a meta-dataset that addresses the lack of standardisation in SLR automation. By consolidating datasets and providing a unified access point, CSMED facilitates the development and evaluation of automated citation screening and full text classification models. We believe it has the potential to advance the field and lead to more robust automated SLR systems. We envision CSMED as a living, evolving collection, and we invite researchers to contribute to expanding it with SLR datasets from other domains.
This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval - DoSSIER (H2020-EU.1.3.1., ID: 860721).
[93] Wenpeng Yin, Dragomir Radev, and Caiming Xiong. DocNLI: A large-scale dataset for document-level natural language inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4913-4922, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021-findings-acl.435. URL https://aclanthology.org/2021-findings-acl.435.
[94] Hye Sun Yun, Iain J Marshall, Thomas Trikalinos, and Byron C Wallace. Appraising the potential uses and harms of LLMs for medical systematic reviews. arXiv preprint arXiv:2305.11828, 2023.
[95] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283-17297, 2020.
[96] Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen. CBLUE: A Chinese biomedical language understanding evaluation benchmark. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7888-7915, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.544. URL https://aclanthology.org/2022.acl-long.544.