LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews
Authors: Christian Jaumann, Andreas Wiedholz, Annemarie Friedrich
Accepted at: acl25

LGAR: Zero-Shot LLM-Guided Neural Ranking for Abstract Screening in Systematic Literature Reviews

Christian Jaumann1,2    Andreas Wiedholz1    Annemarie Friedrich2
1 XITASO GmbH, Germany    2 University of Augsburg, Germany
christian.jaumann|annemarie.friedrich@uni-a.de andreas.wiedholz@xitaso.com

Abstract
The scientific literature is growing rapidly, making it hard to keep track of the state-of-the-art. Systematic literature reviews (SLRs) aim to identify and evaluate all relevant papers on a topic. After retrieving a set of candidate papers, the abstract screening phase determines initial relevance. To date, abstract screening methods using large language models (LLMs) focus on binary classification settings; existing question-answering (QA) based ranking approaches suffer from error propagation. LLMs offer a unique opportunity to evaluate the SLR’s inclusion and exclusion criteria, yet, existing benchmarks do not provide them exhaustively. We manually extract these criteria as well as research questions for 57 SLRs, mostly in the medical domain, enabling principled comparisons between approaches. Moreover, we propose LGAR, a zero-shot LLM-Guided Abstract Ranker composed of an LLM-based graded relevance scorer and a dense re-ranker. Our extensive experiments show that LGAR outperforms existing QA-based methods by 5-10 pp. in mean average precision. Our code and data is publicly available.1

1 Introduction
The scientific literature grows rapidly, by an estimated average of roughly 7% per year over the past 22 years (Oliveira et al., 2022), with 3.3 million articles having appeared worldwide in 2022 (Schneider, 2023). Keeping track of the state-of-the-art of a particular research topic is hence inherently difficult. Systematic Literature Reviews (SLRs) are an academic standard of investigating the current state-of-the-art in a specific field. They follow a strict protocol (e.g., Kitchenham and Brereton (2013)). Since conducting SLRs is typically very expensive and time-consuming (Michelson and Reuter, 2019), efforts have been made to automate various parts of the process. Most existing automation approaches focus on facilitating the selection of relevant articles, one of the most time-consuming steps (van Dinter et al., 2021). Large language models (LLMs) offer unprecedented means and zero-shot capabilities for interpreting text in complex retrieval and ranking scenarios such as abstract screening, where a set of inclusion and exclusion criteria and research questions have to be compared to an abstract. Recent studies suggest that methods based on LLMs can reduce up to 60-70% of the workload necessary for paper selection in the abstract screening phase of SLRs (thi Tran et al., 2023; Sandner et al., 2024).

Due to the difficulty of leveraging LLMs for ranking in a scalable manner (Zhu et al., 2023), almost all prior work on abstract screening with LLMs works in a binary classification setting, refraining from inducing a ranking of candidate papers. In practice, a ranked list is much more useful, because the decision of the cut-off point can be left

![LGAR method for abstract screening consisting of a 2-stage ranking approach: (1) the LLM ranker makes use of a realistic specification of the SLR to assign high-level graded relevance scores, (2) dense re-rankers produce a fully ordered list in scalable time.](page_1012_384_388_388.png)

Figure 1: LGAR method for abstract screening consisting of a 2-stage ranking approach: (1) the LLM ranker makes use of a realistic specification of the SLR to assign high-level graded relevance scores, (2) dense re-rankers produce a fully ordered list in scalable time.

1https://github.com/XITASO/lgar/
to the human user flexibly. The only existing LLM-based ranker (Akinseloyin et al., 2024)2 generates questions for SLR criteria and generates answers for each abstract. The answers are converted to numeric scores using a sentiment classifier, extended with embedding-based similarities, and averaged. While promising, this pipeline-based approach suffers from error propagation in the question generation and sentiment analysis steps.

Existing abstract screening datasets (De Bruin et al., 2023; Kanoulas et al., 2019) provide only part of the SLR criteria. None of them provides the research questions underlying the SLRs. Prior ranking studies approximate the information need by using the SLR’s title and Boolean document retrieval queries (e.g., Wang et al., 2023a,c), or use the incomplete set of criteria specified in the SLR abstracts (Akinseloyin et al., 2024). To evaluate LLM-based rankers using the same SLR descriptions as humans do, we manually collect criteria and research questions exhaustively from the full texts of 57 SLRs and validate our collection process. This important dataset extension will be made publicly available to foster future work.

We propose LGAR, a novel zero-shot LLM-Guided Abstract Ranker, which directly leverages LLMs to provide graded relevance judgments. A dense ranker (Nogueira et al., 2020) then re-ranks the sets of abstracts with the same score. Our experiments using our new complete set of criteria and recent open-weights models for all baselines show that LGAR outperforms the existing state-of-the-art system (Akinseloyin et al., 2024) by a large margin (5-10pp. in mean average precision).

Our contributions are as follows: (1) We provide the exhaustive and validated set of inclusion and exclusion criteria, as well as research questions for two SLR datasets, enabling realistic experiments. (2) We propose LGAR, a zero-shot system for LLM-based abstract screening that performs robustly across datasets and LLM families. (3) Our extensive experimental evaluation highlights the superior performance of LGAR compared to existing zero-shot and fine-tuned methods. We perform a detailed ablation study for the task of abstract ranking with LLMs. We find that using large scales with up to 20 options is most effective. This differs from general ranking tasks, where scales with up to five choices have been reported to be optimal.

2The approach was also recently tested on four SLRs on the impact of climate change on diseases Sujau et al. (2025).

2 Background and Related Work

This section provides a brief overview of SLR automation with a focus on abstract screening.

SLR automation. In the first stage of an SLR a set of candidate papers is retrieved from scientific databases, typically using Boolean queries. The automatic formulation of these queries based on review protocol text has been approached using semantic parsing (Alharbi et al., 2018; Scells et al., 2020b,c), rank fusion (Scells et al., 2020a), and LLM-based generation (Wang et al., 2023b,a). SLRs have also been supported by applying summarization methods (Bui et al., 2016) and by extracting fine-grained entity types and relations (Panayi et al., 2023). Most active learning (AL) tools for SLR automation use traditional machine learning (ML), e.g., AS Review (van de Schoot et al., 2021), Abstrackr (Wallace et al., 2012), Colandr (Cheng et al., 2018), FASTREAD (Yu et al., 2018), Rayyan (Ouzzani et al., 2016), or RobotAnalyst (Przybyla et al., 2018). Carvalho and Parra (2019) compare neural classifiers in an AL setup. AL assumes substantial training data. In contrast, we focus on zero-shot settings.

Abstract screening. Early approaches for binary relevance classification for abstract screening apply traditional ML techniques (Kim and Choi, 2014; Khabsa et al., 2016; Matwin et al., 2010; Matwin and Sazonova, 2012). Qin et al. (2021) fine-tune several BERT models (Devlin et al., 2019) as binary classifiers and combine their predictions. First learning-to-rank setups for abstract screening use non-neural techniques (Anagnostou et al., 2017; Lagopoulos et al., 2018). Lee (2017) combines sentence-level relevance scores computed by a convolutional neural network. Wang et al. (2023c) use a cross-encoder approach following monoBERT (Nogueira et al., 2019). The first abstract screening methods using LLMs all operate in classification settings, applying ChatGPT in a zero-shot setting (Syriani et al., 2023; Guo et al., 2023). Robinson et al. (2023) also fine-tune Llama (Touvron et al., 2023) and Guanaco (Dettmers et al., 2024) to generate binary inclusion labels. Wang et al. (2024b) compare a larger set of open-weights LLMs on a calibrated binary classification task.

Ranking with LLMs. Integrating LLMs with ranking approaches is a highly active research area that has just recently emerged (Zhu et al., 2023). LLMs can be integrated as query rewriters, as dense retrievers, as “readers” to generate natural language
<table>
  <tr>
    <th>Dataset</th>
    <th># SLRs</th>
    <th># Papers</th>
    <th>Domain</th>
    <th>Avg. inclusion rate (%)</th>
  </tr>
  <tr>
    <td>SYNERGY</td>
    <td>26</td>
    <td>169288</td>
    <td>medicine (21), psychology (6), computer science (2), chemistry (2), biology (3), mathematics (1)</td>
    <td>4.31</td>
  </tr>
  <tr>
    <td>CLEF TAR2019<br>– DTA</td>
    <td>8</td>
    <td>30521</td>
    <td>medicine (diagnostic technology assessment)</td>
    <td>7.05</td>
  </tr>
  <tr>
    <td>CLEF TAR2019<br>– Intervention</td>
    <td>20</td>
    <td>41996</td>
    <td>medicine (clinical intervention trials)</td>
    <td>5.49</td>
  </tr>
  <tr>
    <td>CLEF TAR2019<br>– Qualitative</td>
    <td>2</td>
    <td>6536</td>
    <td>medicine (qualitative studies)</td>
    <td>1.32</td>
  </tr>
  <tr>
    <td>CLEF TAR2019<br>– Prognosis</td>
    <td>1</td>
    <td>3367</td>
    <td>medicine (prognosis)</td>
    <td>5.70</td>
  </tr>
</table>

Table 1: Overview of labeled benchmark datasets for automating systematic literature reviews. Counts for the domains in SYNERGY do not add up to 26 because several SLRs fall under multiple scientific disciplines.

answers, or for data augmentation. We here focus on their application in the ranking stage, where they can be applied using a pointwise (Sun et al., 2023), pairwise (Qin et al., 2024), or listwise approach (Tang et al., 2024). The latter two are inefficient for long candidate lists, which is usually the case in abstract screening. In pointwise ranking, the generation likelihood of the query given the document can be used as relevance score (Sachan et al., 2022), or the LLM can be prompted to generate a relevance score (Liang et al., 2023). Zhuang et al. (2024) show that adding intermediate relevance options can improve ranking performance over binary setups. As abstract screening queries are long and complex, we turn to relevance generation.

To the best of our knowledge, the only existing ranking approach for abstract screening using LLMs (Akinseloyin et al., 2024) uses a QA setup. For each inclusion/exclusion criterion, a question is generated using ChatGPT, and the LLM is prompted to answer them for each paper. Answers are converted to numerical scores based on whether a BART (Lewis et al., 2020) model trained on sentiment analysis assigns Positive (1), Neutral (0.5), or Negative (0). The final score for each paper is computed by averaging these scores with embedding similarities of the questions, the original criteria, and the abstract. Akinseloyin et al. (2024) report that while promising, their pipeline-based approach suffers from error propagation. Our own system, LGAR, uses a more effective combination of zero-shot LLM-based pointwise relevance generation and monoT5-based re-ranking.

3 Dataset Extensions

In this section, we describe our extension of two abstract screening benchmarks with inclusion/exclusion criteria and research questions to facilitate more realistic evaluation setups. We extend the SYNERGY dataset (De Bruin et al., 2023) and the test set of the CLEF Technological Assisted Review 2019 (TAR2019) dataset (Task 2) (Kanoulas et al., 2019).3 Both datasets are in English and provide the title of each SLR and a set of papers labeled with binary relevance judgments.4 Table 1 gives an overview of the corpus statistics. The inclusion rate specifies the percentage of abstracts included in the SLR out of the original candidate set retrieved using Boolean queries. The majority of SLRs address topics in medicine.

3.1 Extension with Inclusion/Exclusion Criteria and Research Questions

For SLRs, inclusion and exclusion criteria as well as research questions are specified explicitly as a basis for decision making. Akinseloyin et al. (2024) use the criteria section from the structured abstracts (but no research questions) for TAR2019. However, the full set of criteria and research questions is not always spelled out comprehensively in the SLR’s abstract (e.g., it is not the case for SYNERGY).

To provide the models with a realistic evaluation setup, which can be reproduced and re-used by future work, we manually retrieve the inclusion and exclusion criteria and research questions from the full texts of all 57 SLRs of both datasets. If the review itself does not explicitly mention research questions, we infer them from its objectives or extract them from the review protocols which are provided for the TAR2019 dataset. We focus on primary research questions and objectives. Some SLRs also provide sub-questions and secondary objectives. We do not extract these because they are not typically defined at the start of the abstract screening phase, but more often during the refinement phase of the SLR when several articles have already been checked in greater detail.

3License SYNERGY: CC0 1.0 Universal; License TAR2019: MIT
4The SYNERGY dataset has very recently been extended with inclusion and exclusion criteria also collected from full texts, this is concurrent to our contribution. They do not provide research questions.
In the full text, the criteria are often combined with explanations, definitions, and reasons for specifying a particular criterion. We only extract the criteria, separating them from surrounding descriptive text, and reformulate them as key points. In the medical domain, there are sometimes multiple criteria describing the type of studies or type of participants. We summarize these into one criterion describing all possible types of the respective category. We ensure that the extracted text does not refer to other material in the review, such as figures or tables. In very rare cases, SLRs also formulate inclusion and exclusion criteria based on metadata such as publication dates which are taken care of during document retrieval. Our set of extracted criteria focuses on semantic information needs.

3.2 Validation

The extraction of criteria and research questions for all SLRs was performed by the first author of this paper. We validate the extracted information on 15 randomly selected SLRs (5 SLRs from each the SYNERGY, DTA and Intervention dataset). The second author of this paper extracted the criteria and research questions with the same protocol without prior inspection of the data extracted by the first author. We align the extracted criteria and research questions using the cosine similarities of their BERT embeddings and then verify manually whether they indeed correspond to each other.

For the 15 SLRs in our validation study, the first and second author each extracted 23 research questions, which matched exactly. This high inter-rater agreement stems from the structured nature of medical and computer science SLRs, which often present a section titled “Objectives” or explicitly state research questions, which allows straightforward extraction. Incidentally, each author extracted 99 inclusion/exclusion criteria, of which 97 match. Disagreements can be narrowed down to one author including more details to an criterion than the other one or including/neglecting a criterion that was not mentioned explicitly as criterion but described as important factor in the review process. Overall, we include that inclusion/exclusion criteria and research questions have been extracted reliably for our extension of the SYNERGY and CLEF TAR2019 datasets.

4 LLM-Guided Abstract Ranking

We propose LGAR, a ranking model comprised of a two-stage process to determine the relevance of a screened paper, as illustrated in Figure 1. In the first stage, the LLM assigns each paper a relevance score on a scale from 0 to \( k \), similar to the LLM-based ranking approach \( DIRECT(0,k) \) of Guo et al. (2024).

Ranking for abstract screening requires producing an ordered list of potentially thousands of papers. The differentiation that LLMs are able to produce is limited to the range of a dozen relevance scores (Zhuang et al., 2024), i.e., the first ranking step will produce many ties. We hence produce the global ranking by additionally ranking the papers within each set of papers that have obtained the same relevance score by the LLM using a zero-shot dense ranking model. Our proposed two-stage ranker is modular, providing for modular combinations of LLMs and dense rankers.

4.1 Graded Relevance Judgments by LLM

In the first ranking stage, the LLM is prompted with the SLR’s criteria, as well as research questions (shown in Figure 2). The general instruction encoded in the system message suggests that a paper should only be considered relevant if all inclusion criteria but none of the exclusion criteria are met. The method is configurable to use different prompting techniques, but the key elements remain the same. The system message provides the title and research questions of the SLR and the associated ranking task, in addition to a definition of relevance in the context of the SLR. The user message contains a more specific task description, instructing the LLM to determine a degree of relevance in the form of assigning a score from a Likert scale using the provided selection criteria, and to answer in a predefined format. It also provides the title and the abstract of the paper.

Prompting Techniques. LLM responses are highly dependent on the type of prompt used (Brown et al., 2020; Wu et al., 2023; Wang et al., 2023d). Therefore, we test several different prompting techniques, i.e., zero-shot, Chain of Thought (CoT), and CoT with self-consistency (n=3). We also tried 2-shot variants without observing any improvements (see Appendix A.4). In self-consistency setups, the relevance scores of each paper are averaged over three runs. For CoT prompting, we instruct the model to “think step by step” (Kojima et al., 2022).
System Message: You are a researcher conducting a systematic literature review (SLR) with the title [title]. The review aims to answer the following research questions: [research_questions]. Your task is to decide how relevant the provided paper is to the review, given a list of inclusion criteria. A paper is relevant if all inclusion criteria but none of the exclusion criteria are met.

User Message: Task: You will be presented with a paper’s title and abstract. Your task is to decide how relevant the given paper is to the review. Return a number for your decision ranging from [relevance_lower_value] to [relevance_upper_value], where [relevance_lower_value] means that you are absolutely sure that the paper should not be included and [relevance_upper_value] means that you are absolutely sure that the paper should be included, and where an intermediate value means that you are unsure. Please read the title and the abstract carefully and then make your decision based on the provided inclusion and exclusion criteria.
Title: [title_paper]
Abstract: [abstract]
Inclusion criteria: [inclusion_criteria]
Exclusion Criteria: [exclusion_criteria]
Give your answer in the following format:
"Decision: [relevance_lower_value] - [relevance_upper_value]"

Figure 2: Zero-shot prompts used in LGAR.

Output Parsing. To make the responses as deterministic and reproducible as possible, we use a temperature of 0. The scores are extracted using regular expression and we verify that they are in the desired relevance scale. If score extraction fails, and when using self-consistency, the temperature is set to 0.5 to encourage more diverse results. In the rare case that the LLM does not respond in the expected format, the request is repeated up to three times, without providing the model’s previous faulty response. If the response is still not in the expected format, the paper is assigned the average of the sum of all LLM generated relevance scores for the respective SLR.

4.2 Neural Dense Re-Ranking

In the second stage, we re-rank the papers within the groups generated by the LLM with neural dense retrievers backed by pretrained language models such as monoBERT (Nogueira and Cho, 2019) or ColBERT (Khattab and Zaharia, 2020). These cross- and bi-encoders based on BERT still rank among the state-of-the-art for re-ranking tasks. Each query-document pair is the concatenation of the query followed by the title and abstract of the paper in question, truncated to 512 wordpiece tokens. As query, we use (1) only the SLR’s title following Wang et al. (2023c) or (2) the title and the research questions of the SLR. We use neural re-rankers that were fine-tuned on the general retrieval dataset MS MARCO (Nguyen et al., 2016), but not specifically tuned for abstract screening.

5 Experiments

To compare LGAR to existing approaches, we conduct an extensive experimental evaluation using our newly constructed benchmark. In this section, we describe our experimental design, including our selection of models, our choice of evaluation metrics, and the baselines. We then present our experimental results, followed by several ablation studies.

5.1 Model Selection and Experimental Settings

To enhance the reproducibility of our results, our model selection of instruction-following LLMs is constrained to open-weights alternatives. We focus on Llama3.3-70B-Instruct (Dubey et al., 2024), which is one of the best performing open-weights models for its size at the time of writing. To assess the extent of performance disparities among diverse model families and varying model sizes, we conduct comparative analyses with the following models: Llama3.1-8B-Instruct, Qwen2.5-32B-Instruct,5 Qwen2.5-72B-Instruct,6 and Mistral-Large-Instruct-2411 (123B).7 We run all models using 16-bit quantization.

We report test results on the entire SYNERGY dataset (which does not provide any training data), and on the test portion of the TAR2019 dataset.8 To develop LGAR, we performed prompt-engineering using Llama3.3-70B-Instruct in interactive experiments using SLRs taken from an additional dataset in the medical domain (Guo et al., 2023) (see Appendix A.1 in particular for the exploration of different scale sizes) and one unpublished SLR. We also run the zero-shot dense rankers on our tuning data, identifying monoT5 (Nogueira et al., 2020) in the 3B variant as most effective (also see Appendix A.1). In our ablation tests, we additionally evaluate monoBERT (large) (Nogueira and Cho, 2019) and ColBERTv2 (Santhanam et al., 2022).

5.2 Evaluation Metrics

We adopt the evaluation metrics of the CLEF TAR 2019 challenge (Kanoulas et al., 2019): mean average precision (MAP), recall at k% of top-ranked abstracts (\( R@k\%, k = 1, 5, 10, 20, 50 \)), and work

5https://huggingface.co/Qwen/Qwen2.5-32B-Instruct
6https://huggingface.co/Qwen/Qwen2.5-72B-Instruct
7https://huggingface.co/mistralai/Mistral-Large-Instruct-2411
8In alignment with the general evaluation protocol used in the CLEF TAR 2019 Shared Task, we only use the test split of the CLEF TAR dataset comprising 31 SLRs for evaluating our zero-shot approach. We use the train split of 100 additional SLRs to replicate the fine-tuning of BERT-based models for comparing different model types.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th rowspan="2">WSS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>@95% @100%</th>
  </tr>
  <tr>
    <td rowspan="6">SYNERGY</td>
    <td>random reranking</td>
    <td>4.9</td>
    <td>6.2</td>
    <td>1.2</td>
    <td>5.1</td>
    <td>10.4</td>
    <td>20.0</td>
    <td>50.3</td>
    <td>3.0</td>
    <td>3.0</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>12.5</td>
    <td>36.3</td>
    <td>9.0</td>
    <td>26.4</td>
    <td>40.0</td>
    <td>54.4</td>
    <td>80.3</td>
    <td>33.6</td>
    <td>21.8</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>18.8</td>
    <td>52.3</td>
    <td>14.1</td>
    <td>37.5</td>
    <td>54.5</td>
    <td>69.1</td>
    <td>91.1</td>
    <td>50.0</td>
    <td>35.4</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours) †</b></td>
    <td>34.0</td>
    <td>63.6</td>
    <td>25.8</td>
    <td>55.0</td>
    <td>67.8</td>
    <td>80.5</td>
    <td>93.1</td>
    <td>59.5</td>
    <td><b>50.6</b></td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5) †</td>
    <td>36.8</td>
    <td>63.8</td>
    <td>32.7</td>
    <td>57.7</td>
    <td>71.0</td>
    <td>81.2</td>
    <td>94.3</td>
    <td>63.6</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td><b>40.7</b></td>
    <td><b>67.0</b></td>
    <td>34.1</td>
    <td><b>59.3</b></td>
    <td><b>72.0</b></td>
    <td>81.9</td>
    <td><b>94.4</b></td>
    <td><b>65.2</b></td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>38.8</td>
    <td>62.9</td>
    <td>33.4</td>
    <td><b>59.3</b></td>
    <td>70.3</td>
    <td>81.4</td>
    <td>92.8</td>
    <td>62.5</td>
    <td>45.8</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td><b>40.7</b></td>
    <td>66.4</td>
    <td><b>34.7</b></td>
    <td>58.6</td>
    <td>70.7</td>
    <td><b>82.6</b></td>
    <td>93.5</td>
    <td>64.5</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td>38.9</td>
    <td>59.8</td>
    <td>33.3</td>
    <td>58.4</td>
    <td>70.1</td>
    <td>79.7</td>
    <td>91.4</td>
    <td>58.6</td>
    <td>40.9</td>
  </tr>
  <tr>
    <td rowspan="6">TAR2019</td>
    <td>random reranking</td>
    <td>6.7</td>
    <td>9.9</td>
    <td>1.1</td>
    <td>5.5</td>
    <td>10.6</td>
    <td>20.2</td>
    <td>52.0</td>
    <td>6.6</td>
    <td>7.0</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>21.1</td>
    <td>44.8</td>
    <td>12.6</td>
    <td>34.2</td>
    <td>46.2</td>
    <td>60.5</td>
    <td>82.5</td>
    <td>40.9</td>
    <td>30.4</td>
  </tr>
  <tr>
    <td>monoT5_3B (T)</td>
    <td>30.1</td>
    <td>63.2</td>
    <td>17.7</td>
    <td>44.6</td>
    <td>59.3</td>
    <td>73.4</td>
    <td>94.0</td>
    <td>58.2</td>
    <td>50.7</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (theirs)</b></td>
    <td>42.8</td>
    <td>68.6</td>
    <td>23.9</td>
    <td>53.8</td>
    <td>70.8</td>
    <td>83.8</td>
    <td>96.0</td>
    <td>63.7</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours) †</b></td>
    <td>45.1</td>
    <td>70.8</td>
    <td>26.8</td>
    <td>57.5</td>
    <td>72.9</td>
    <td>83.4</td>
    <td><b>96.6</b></td>
    <td>66.1</td>
    <td>53.0</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5) †</td>
    <td>48.4</td>
    <td>76.4</td>
    <td>26.0</td>
    <td>60.9</td>
    <td>75.7</td>
    <td>86.8</td>
    <td>96.2</td>
    <td>69.9</td>
    <td><b>61.1</b></td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td><b>50.6</b></td>
    <td><b>76.5</b></td>
    <td><b>29.3</b></td>
    <td><b>63.6</b></td>
    <td><b>76.7</b></td>
    <td><b>88.3</b></td>
    <td>95.8</td>
    <td><b>71.2</b></td>
    <td>60.9</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>43.5</td>
    <td>72.0</td>
    <td>25.7</td>
    <td>57.5</td>
    <td>69.8</td>
    <td>86.0</td>
    <td>96.5</td>
    <td>67.0</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td>44.8</td>
    <td>74.9</td>
    <td>26.1</td>
    <td>57.9</td>
    <td>74.2</td>
    <td>85.7</td>
    <td>95.8</td>
    <td>68.8</td>
    <td>59.4</td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td><b>50.7</b></td>
    <td>64.8</td>
    <td><b>29.7</b></td>
    <td>62.1</td>
    <td>75.1</td>
    <td>86.0</td>
    <td>93.7</td>
    <td>61.4</td>
    <td>42.9</td>
  </tr>
</table>

Table 2: Zero-shot models; recall values are macro-averages over SLRs. All LLM-based models use Llama3.3-70B. T = title of the SLR, R = research questions of the SLR. Akinseloyin et al. (2024): ours = uses our annotated criteria, theirs = uses their criteria. In all combinations, monoT5 uses the title as query and the LLM the scale 0-19. † indicates that the approaches get the same information on SLR and paper.

saved over sampling (WSS) at different recall levels (\( WSS@r\%, r = 95, 100 \)), defined as follows by Cohen et al. (2006):

\[
\text{WSS}@r\% = \frac{\text{TN} + \text{FN}}{\text{N}} - (1 - r)
\] (1)

WSS aims to estimate the reduction in human screening workload for a given recall level. However, it has some drawbacks, such as that its value range depends on the inclusion rate of the respective SLR, making comparisons across multiple SLRs unfair (Kusa et al., 2023). Therefore, we additionally provide the min-max normalized version of WSS proposed by Kusa et al. (2023), which mitigates these issues and reports the True Negative Rate at recall level \( r \):

\[
\text{nWSS}@r\% = \text{TNR}@r\% = \frac{\text{TN}}{\text{TN} + \text{FP}}
\] (2)

To compute results, we use the evaluation scripts provided in CLEF TAR 2018 with one adaptation: we found that they compute the average recall of multiple SLRs using micro-averaging:

\[
R@k\% = \frac{\sum_{\{\text{SLRs}\}} \text{TP}}{\sum_{\{\text{SLRs}\}} (\text{TP} + \text{FN})}
\] (3)

This gives more weight to larger SLRs or SLRs with a higher inclusion rate. To obtain a more realistic estimate of performance for an unseen SLR, we instead follow Akinseloyin et al. (2024) by first computing recall scores per SLR, and then macro-averaging over the obtained scores.

5.3 Baselines

As non-neural information retrieval baseline, we provide the results of Okapi BM25 (Robertson and Zaragoza, 2009). We also report the results of monoT5, which is similar to the zero-shot model of Wang et al. (2023c) that uses BERT-base.

The closest existing work to ours is that of Akinseloyin et al. (2024), who use a QA setup and (except for a small case study) ChatGPT. For a fair comparison, we replicate the GPT_QA_Soft_Both_ReRank variant of their model (which also uses selection criteria for each SLR) using Llama3.3-70B-Instruct and the embeddings proposed by Wang et al. (2024a) (instead of the closed-source embeddings they use). To increase reproducibility, we use a temperature of 0 instead of 0.2 as in the original work. Our replication outperforms their scores provided for the TAR2019 dataset on all metrics except WSS (which is a somewhat unreliable metric and only reported for comparison with prior work). Detailed scores are provided in Appendix A.2.

5.4 Main Results and Findings

Table 2 compares the effectiveness of LGAR using Llama3.3-70B, relevance scale 0-19, and monoT5
(title-only) as re-ranker with our baselines. On both the SYNERGY and the TAR2019 datasets, zero-shot monoT5 using only the title as query outperforms BM25 that compares the titles and abstracts of each paper to the title and research questions of the SLR. For reasons explained in section 5.2, we focus on TNR instead of WSS when interpreting our results. Overall, LGAR achieves the strongest performance over all models, outperforming the QA-based model of Akinseloyin et al. (2024) by a large margin.

Impact of dense re-ranking. Our ablation removing monoT5, randomly ranking the papers within each set, shows that in terms of MAP, the difference is small for SYNERGY and negligible for TAR2019, which is somewhat expected as the LLM-based relevance judgment defines the overall order of the ranking. Nevertheless, the dense re-ranker contributes to workload savings by moving up articles that were incorrectly assigned a low relevance score by the LLM, which is particularly highlighted by the the strong increase of around 7-12% in TNR. Similarly, LGAR profits from having access to the SLR’s research questions (this difference is more pronounced on SYNERGY).

Comparison to QA-based system. LGAR (T, monoT5) and the QA model of Akinseloyin et al. (2024) (ours) differ with respect to model architecture and prompts, yet both receive exactly the same information about the SLR and the papers. Even in this setting (without using research questions), LGAR outperforms the QA model, showing that the performance increase is not merely due to additional information in the prompt but due to model architecture. However, given that the research questions should be available at the stage of abstract screening when conducting an SLR and that adding them appears to enhance the performance of our approach, it seems reasonable to make use of them.

CoT prompting. Instructing the model to think step by step does not improve the performance of LGAR. Using self-consistency to make up for erroneous runs improves the situation in particular for SYNERGY, but does not lead to any performance improvements over standard LGAR. Detailed information can be found in Appendix A.4.

5.5 Ablation Tests and Analysis

We now perform an extensive exploration into various settings of LGAR. Detailed scores are provided in Appendix A.3.

<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th>MAP</th>
    <th>TNR@95%</th>
    <th>R@20%</th>
  </tr>
  <tr>
    <td rowspan="5">SYNERGY</td>
    <td>Llama3.1-8B</td>
    <td>31.9</td>
    <td>61.3</td>
    <td>80.9</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>40.7</td>
    <td>67.0</td>
    <td>81.9</td>
  </tr>
  <tr>
    <td>Mistral-123B</td>
    <td>38.8</td>
    <td>65.8</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>38.7</td>
    <td>63.2</td>
    <td>78.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td><b>41.3</b></td>
    <td><b>68.5</b></td>
    <td><b>83.0</b></td>
  </tr>
  <tr>
    <td rowspan="5">TAR2019</td>
    <td>Llama3.1-8B</td>
    <td>42.3</td>
    <td>69.6</td>
    <td>86.1</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td><b>50.6</b></td>
    <td><b>76.5</b></td>
    <td>88.3</td>
  </tr>
  <tr>
    <td>Mistral-123B</td>
    <td>48.4</td>
    <td>70.8</td>
    <td>86.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>47.6</td>
    <td>73.4</td>
    <td>84.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>50.2</td>
    <td>73.0</td>
    <td><b>89.0</b></td>
  </tr>
</table>

Table 3: Comparison of LLMs. Scale: 0-19. Re-ranker: monoT5 with title-only as query. 0-shot prompting.

5.5.1 Effectiveness of different LLMs

![Comparison of scales in terms of MAP and TNR@95%. The optimum is around 0-14 and 0-19.](page_1012_1012_420_312.png)

Figure 3: Comparison of scales in terms of MAP and TNR@95%. The optimum is around 0-14 and 0-19.

Table 3 reports the effectiveness of using different LLMs with a relevance scale of 0-19 and monoT5 (title-only) as re-ranker. Performance degrades for smaller models, but using the larger Mistral-123B did not provide any benefits over Llama3.3-70B. Qwen2.5-72B excels on SYNERGY and also performs strongly on TAR2019. Overall, LGAR (which was developed for Llama3.3-70B) is relatively robust with regard to the underlying LLM (within the same model sizes).

5.5.2 Impact of scale size

We here conduct a principled analysis of various choices for scales. Figure 3 reports the performance of LGAR using several different scale sizes. Zhuang et al. (2024) found that for general ranking tasks, 0-4 worked best. By contrast, for our complex ranking task, it is beneficial to increase the scale size even further. In terms of workload savings, 0-14 performs best, while the best ranking is produced by 0-19. Increasing the scale size beyond 0-19 does not further enhance the performance. Interestingly, this replicates our initial findings on
Figure 4: Average number of different relevance scores actually used by LLM and average group size by scale, all SLRs of SYNERGY and TAR2019, Llama3.3-70B as LLM ranker and monoT5 (title-only) as re-ranker.

Figure 5: MAP of zero-shot dense rankers (without LGAR). T = title / R = research questions used as query.

our tuning data, validating our choice of 0-19. As can be seen in Figure 4, the number of different relevance scores actually used by the LLM drops for larger scales. The average number of papers ending up in one group is smallest for 0-19, which means in the 0-19 setting, the LLM takes over most ranking workload compared to other settings.

5.5.3 Effectiveness of dense rankers
In Table 4, we compare the performance of BM25 with that of three zero-shot dense rankers (without using the LLM step) using two different queries: title-only (T) and title and research questions (T+R). Details are reported in the Appendix A.1. On both the SYNERGY and the TAR2019 dataset, the best configuration by far uses monoT5 with only the title as query. While BM25 and monoBERT can deal with the longer input query (T+R), the performance of ColBERT and monoT5 degrades over only using the title.

<table>
  <tr>
    <th>Dataset Model</th>
    <th>MAP</th>
    <th>TNR@95%</th>
    <th>R@20%</th>
  </tr>
  <tr>
    <td rowspan="3">Interv.</td>
    <td>BioBERT_ft (T)</td>
    <td>45.9</td>
    <td>70.8</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>LGAR (T)</td>
    <td>55.0</td>
    <td><b>75.0</b></td>
    <td><b>88.9</b></td>
  </tr>
  <tr>
    <td>LGAR (T+R)</td>
    <td><b>56.8</b></td>
    <td><b>75.0</b></td>
    <td>88.4</td>
  </tr>
  <tr>
    <td rowspan="3">DTA</td>
    <td>BioBERT_ft (T)</td>
    <td>35.4</td>
    <td>76.0</td>
    <td>82.5</td>
  </tr>
  <tr>
    <td>LGAR (T)</td>
    <td>34.5</td>
    <td><b>78.1</b></td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>LGAR (T+R)</td>
    <td><b>38.1</b></td>
    <td>76.8</td>
    <td><b>85.0</b></td>
  </tr>
</table>

Table 4: Comparison to fine-tuned ranker. BioBERT_ft is our replication of Wang et al. (2023c). Re-ranker for LGAR: monoT5.

Figure 6: Boxplots of MAP scores of LGAR (T+R, monoT5) and Akinseloyin et al. (2024) (ours) on both datasets.

of Wang et al. (2023c). We replicate the model to report all metrics in the exact same setting. Our replicated model slightly outperforms the results reported by Wang et al. (2023c) (see Appendix A.2). We fine-tune BioBERT (large) (Lee et al., 2020) using the framework of Gao et al. (2021) and the training splits for the Intervention and DTA subset of TAR2019 (there is no training data for the rest of TAR2019 and for SYNERGY). Our zero-shot LGAR model outperforms the fine-tuned model by 7-10pp. in terms of MAP (see Table 4).

5.5.5 Performance distribution across SLRs
Figure 6 compares the distribution of MAP scores achieved by LGAR with that by our replication of the approach of Akinseloyin et al. (2024) on both datasets. MAP scores differ greatly between SLRs, which is due to their very different inclusion rates. LGAR appears to have an even greater range of scores due to its higher peak MAP scores. Overall, LGAR achieves at least the same performance as the QA-based approach, with a clear tendency to be better.

6 Discussion and Outlook
Our results suggest that LLMs are very effective in interpreting inclusion and exclusion criteria for
abstract screening. They provide a good initial categorization. Re-ranking is particularly effective for finding relevant papers accidentally categorized as irrelevant by the LLM, thus significantly contributing to workload savings.

Compared to the approach of Akinseloyin et al. (2024), it does not seem necessary to split the criteria into several parts, which significantly reduces the number of LLM requests. At the same time, LGAR is less “explainable” than models evaluating every criterion separately. We hypothesize that LGAR performs more robustly precisely because it is not forced to evaluate each criterion separately, which likely introduces many noisy scores in the QA-based model. For many criteria, the abstract may not contain sufficient information to allow a meaningful assessment. The QA-based baseline is supposed to generate Neutral in such cases, yet, detecting when information is insufficient to answer a question is yet a difficult task for LLMs (Ji et al., 2022; Gautam et al., 2023). In future work, it would be interesting to further support the human user by identifying the criteria that led to including or excluding a paper.

7 Conclusion

In this paper, we have proposed LGAR, a novel LLM-Guided Abstract Ranker for SLRs. LGAR is based entirely on zero-shot open-weights models, yet outperforms models specifically tuned for this task. We contribute a dataset extension of exhaustively extracted criteria and research questions and compare to baselines in a fair way, using the same backbones and inputs, finding that LGAR performs best on two abstract screening datasets.

Limitations

The SYNERGY and TAR2019 datasets have been created based on actual SLRs, i.e., the set of relevant papers has been extracted from the SLRs’ bibliographies. Hence, strictly speaking, both datasets that we use in this study provide relevance labels that correspond to the full text screening stage. Yet, as redistributing the full text of scientific publications is not generally possible due to license restrictions, both datasets only provide titles and abstracts of the papers. Hence, it is theoretically possible that some papers were still included after the abstract screening stage of an SLR, but excluded in the full text screening stage. In these cases, the title and abstract of the paper, which are the basis for abstract screening, may not contain the necessary information. Despite this minor approximation, the SYNERGY and the TAR2019 datasets nevertheless provide suitable benchmarks for automatic abstract screening methods, an assumption that we share with prior work (Akinseloyin et al., 2024; Wang et al., 2023c,a, 2024b).

Another limitation of the evaluation is that even though the evaluation was conducted with 57 different SLRs, the approach could only be tested mainly in the medical domain. This is due to the composition of the datasets that are widely used when evaluating automation methods for SLRs. Although other domains are represented in the SYNERGY dataset, we believe that the number of SLRs for these respective domains is too small to give a thorough answer to the question of whether the performance of the approach is equivalent to that in the medical domain.

In addition, since many of these SLRs were published several years ago, some of their data may have already been exposed to the LLMs in their training. Hence, we can not exclude the possibility of data contamination.

Furthermore, since LLMs can perpetuate and amplify biases present in the training data, LGAR may inadvertently reinforce these biases when ranking abstracts. This could lead to certain research being unfairly prioritized or overlooked, potentially skewing the outcomes of SLRs.

Risks. When using this approach in conducting an SLR, the main risk is to overlook a paper that is relevant to the research questions being addressed in the SLR. Missing a relevant paper is problematic when writing an SLR, which can happen when defining a cut-off in the ranked list when applying LGAR to the abstract screening stage of an SLR. However, we argue that this risk is also present when this is done manually, since manual screening of hundreds or thousands of papers is also error-prone.

Acknowledgments

The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v110ee. BayernKI funding is provided by Bavarian state authorities.

Furthermore, the authors gratefully acknowledge
the resources on the LiCCA HPC cluster of the University of Augsburg, co-funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 499211671.

References

Opeoluwa Akinseloyin, Xiaorui Jiang, and Vasile Palade. 2024. A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association, page ocae166.

Amal Alharbi, William Briggs, and Mark Stevenson. 2018. Retrieving and ranking studies for systematic reviews: University of sheffield’s approach to CLEF ehealth 2018 task 2. In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018, volume 2125 of CEUR Workshop Proceedings. CEUR-WS.org.

Antonios Anagnostou, Athanasios Lagopoulos, Grigorios Tsoumakas, and Ioannis P. Vlahavas. 2017. Combining inter-review learning-to-rank and intra-review incremental training for title and abstract screening in systematic reviews. In Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017, volume 1866 of CEUR Workshop Proceedings. CEUR-WS.org.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Duy Duc An Bui, Guilherme Del Fiol, John F. Hurdle, and Siddhartha Jonnalagadda. 2016. Extractive text summarization system to aid data extraction from full text in systematic review development. J. Biomed. Informatics, 64:265–272.

Andres Carvallo and Denis Parra. 2019. Comparing word embeddings for document screening based on active learning. In Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France, July 25, 2019, volume 2414 of CEUR Workshop Proceedings, pages 100–107. CEUR-WS.org.

SH Cheng, C Augustin, A Bethel, D Gill, S Anzaroot, J Brun, B DeWilde, RC Minnich, R Garside, YJ Masuda, et al. 2018. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conservation Biology, 32(4):762–764.

Aaron M. Cohen, William R. Hersh, K. Peterson, and Po-Yin Yen. 2006. Research paper: Reducing workload in systematic review preparation using automated citation classification. J. Am. Medical Informatics Assoc., 13(2):206–219.

Jonathan De Bruin, Yongchao Ma, Gerbrich Ferdinands, Jelle Teijema, and Rens Van de Schoot. 2023. SYN-ERGY Open machine learning dataset on study selection in systematic reviews.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783.

Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, volume 12651 of Lecture Notes in Computer Science, pages 280–286. Springer.

Vagrant Gautam, Miaoran Zhang, and Dietrich Klakow. 2023. A lightweight method to generate unanswerable questions in english. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Mike Paget, and Christopher Naugler. 2023. Automated paper screening for clinical reviews using large language models. CoRR, abs/2305.00844.

Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Le Yan, and Yue Zhang. 2024. Generating diverse criteria on-the-fly to improve point-wise LLM rankers. CoRR, abs/2404.11960.

Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, and Xiangang Li. 2022. To answer or not to answer? improving machine reading comprehension model with span-based contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1292–1300, Seattle, United States. Association for Computational Linguistics.
Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. Clef 2019 technology assisted reviews in empirical medicine overview. CEUR Workshop Proceedings, 2380. 20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 ; Conference date: 09-09-2019 Through 12-09-2019.

Madian Khabsa, Ahmed K. Elmagarmid, Ihab F. Ilyas, Hossam Hammady, and Mourad Ouzzani. 2016. Learning to identify relevant studies for systematic reviews using random forest and external information. Mach. Learn., 102(3):465–482.

Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39–48. ACM.

Seunghee Kim and Jinwook Choi. 2014. An svm-based high-quality article classifier for systematic reviews. J. Biomed. Informatics, 47:153–159.

Barbara Kitchenham and Pearl Brereton. 2013. A systematic review of systematic review process research in software engineering. Information and Software Technology, 55(12):2049–2075.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yūtaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. 2023. An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews. Intell. Syst. Appl., 18:200193.

Athanasios Lagopoulos, Antonios Anagnostou, Adamantios Minas, and Grigorios Tsoumakas. 2018. Learning-to-rank and relevance feedback for literature appraisal in empirical medicine. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, September 10-14, 2018, Proceedings, volume 11018 of Lecture Notes in Computer Science, pages 52–63. Springer.

Grace Eunkyung Lee. 2017. A study of convolutional neural networks for clinical document classification in systematic reviews: Sysreview at CLEF ehealth 2017. In Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017, volume 1866 of CEUR Workshop Proceedings. CEUR-WS.org.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4):1234–1240.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023. Holistic evaluation of language models. Trans. Mach. Learn. Res., 2023.

Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O’Blenis. 2010. A new algorithm for reducing the workload of experts in performing systematic reviews. J. Am. Medical Informatics Assoc., 17(4):446–453.

Stan Matwin and Vera Sazonova. 2012. Direct comparison between support vector machine and multinomial naive bayes algorithms for medical abstract classification. Journal of the American Medical Informatics Association, 19(5):917–917.

Matthew Michelson and Katja Reuter. 2019. The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemporary Clinical Trials Communications, 16:100443.

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268.

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. Association for Computational Linguistics.
Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR, abs/1901.04085.

Rodrigo Frassetto Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. CoRR, abs/1910.14424.

Eduardo A Oliveira, Maria Christina L Oliveira, Enrico A Colosimo, Daniella B Martelli, Ludmila R Silva, Ana Cristina Simões E Silva, and Hercílio Martelli-Júnior. 2022. Global scientific production in the pre-covid-19 era: An analysis of 53 countries for 22 years. Anais da Academia Brasileira de Ciências, 94(suppl 3):e20201428.

Mourad Ouzzani, Hossam Hammady, Zbys Fedorowicz, and Ahmed Elmagarmid. 2016. Rayyan—a web and mobile app for systematic reviews. Systematic reviews, 5:1–10.

Antonia Panayi, Katherine Ward, Amir Benhadj-Schaff, A Santiago Ibanez-Lopez, Andrew Xia, and Regina Barzilay. 2023. Evaluation of a prototype machine learning tool to semi-automate data extraction for systematic literature reviews. Systematic Reviews, 12(1):187.

Piotr Przybyla, Austin J Brockmeier, Georgios Kontostasios, Marie-Annick Le Pogam, John McNaught, Erik von Elm, Kay Nolan, and Sophia Ananiadou. 2018. Prioritising references for systematic reviews with robotanalyst: a user study. Research synthesis methods, 9(3):470–488.

Xuan Qin, Jiali Liu, Yuning Wang, Yanmei Liu, Ke Deng, Yu Ma, Kang Zou, Ling Li, and Xin Sun. 2021. Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. Journal of Clinical Epidemiology, 133:121–129.

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 1504–1518. Association for Computational Linguistics.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Ambrose Robinson, William Thorne, Ben P. Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, and Xingyi Song. 2023. Bio-sieve: Exploring instruction tuning large language models for systematic review automation. CoRR, abs/2308.06610.

Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3781–3797. Association for Computational Linguistics.

Elias Sandner, Bing Hu, Alice Simiceanu, Luca Fontana, Igor Jakovljevic, Andre Henriques, Andreas Wagner, and Christian Gütl. 2024. Screening automation for systematic reviews: A 5-tier prompting approach meeting cochrane’s sensitivity requirement. 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pages 150–159.

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States. Association for Computational Linguistics.

Harrison Scells, Guido Zuccon, and Bevan Koopman. 2020a. You can teach an old dog new tricks: Rank fusion applied to coordination level matching for ranking in systematic reviews. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part I, volume 12035 of Lecture Notes in Computer Science, pages 399–414. Springer.

Harrison Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020b. Automatic boolean query formulation for systematic review literature search. In WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1071–1081. ACM / IW3C2.

Harrison Scells, Guido Zuccon, Bevan Koopman, and Justin Clark. 2020c. A computational approach for objectively derived systematic review search strategies. In Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part I, volume 12035 of Lecture Notes in Computer Science, pages 385–398. Springer.

Benjamin Schneider. 2023. Publications output: Us trends and international comparisons. science & engineering indicators 2024. nsb-2023-33. National Science Foundation.

Masood Sujau, Masako Wada, Emilie Vallee, Natalie Hillis, and Teo Susnjak. 2025. Accelerating disease model parameter extraction: An llm-based ranking approach to select initial studies for literature review automation. Preprints.

Weiwei Sun, Zheng Chen, Xinyu Ma, Lingyong Yan, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Instruction distillation makes large language models efficient zero-shot rankers. CoRR, abs/2311.01555.
Eugene Syriani, Istvan David, and Gauransh Kumar. 2023. Assessing the ability of chatgpt to screen articles for systematic reviews. CoRR, abs/2307.06464.

Raphael Tang, Xinyu Crystina Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2024. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 2327–2340. Association for Computational Linguistics.

Viet thi Tran, Gerald Gartlehner, Sally Yacoub, Isabelle Bouttron, Lukas Schwingshackl, Julia Stadelmaier, Isolde Sommer, Farzaneh Aboulayeh, Sivem Afach, Joerg Meerpohl, Philippe Ravaud, Paris Est, and PhD Viet-Thi Tran MD. 2023. Sensitivity, specificity and avoidable workload of using a large language models for title and abstract screening in systematic reviews and meta-analyses. In medRxiv.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.

Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parissa Zahedi, Jan de Boer, Felix Weijdenma, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willenssen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, and Daniel L. Oberski. 2021. An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2):125–133. Publisher: Nature Publishing Group.

Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021. Automation of systematic literature reviews: A systematic literature review. Information and Software Technology, 136:106589.

Byron C. Wallace, Kevin Small, Carla E. Brodley, Joseph Lau, and Thomas A. Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: abstractr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, IHI ’12, page 819–824, New York, NY, USA. Association for Computing Machinery.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Association for Computational Linguistics.

Shuai Wang, Harrisen Scells, Bevan Koopman, Martin Potthast, and Guido Zuccon. 2023a. Generating natural language queries for more effective systematic review screening prioritisation. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP ’23, page 73–83, New York, NY, USA. Association for Computing Machinery.

Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023b. Can chatgpt write a good boolean query for systematic review literature search? In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 1426–1436. ACM.

Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023c. Neural rankers for effective screening prioritisation in medical systematic review literature search. In Proceedings of the 26th Australasian Document Computing Symposium, ADCS ’22, New York, NY, USA. Association for Computing Machinery.

Shuai Wang, Harrisen Scells, Shengyao Zhuang, Martin Potthast, Bevan Koopman, and Guido Zuccon. 2024b. Zero-shot generative large language models for systematic review screening automation. In Advances in Information Retrieval, pages 403–420, Cham. Springer Nature Switzerland.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023d. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Dingjun Wu, Jing Zhang, and Xinmei Huang. 2023. Chain of thought prompting elicits knowledge augmentation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6519–6534, Toronto, Canada. Association for Computational Linguistics.

Zhe Yu, Nicholas A. Kraft, and Tim Menzies. 2018. Finding better active learners for faster literature reviews. Empir. Softw. Eng., 23(6):3161–3186.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhao Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. CoRR, abs/2308.07107.

Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 358–370, Mexico City, Mexico. Association for Computational Linguistics.
A Appendix

A.1 Initial Interactive Tuning of LGAR

In Table 5, the results of preliminary experiments are shown. These experiments were performed on the dataset provided by Guo et al. (2023), which consists of 11 medical SLRs with a total of about 50000 papers. We excluded the SLR aiha from our evaluation because the provided dataset did not contain any relevant papers. By not using our test sets, i.e., the SYNERGY and TAR2019 dataset, for the initial optimization of our approach, we aimed to make our results more realistic since we did not optimize our approach on these test sets. We also refrained from using the train set of the CLEF TAR 2019 dataset for the initial tuning, since optimizing on this dataset, which is quite similar to the corresponding test set, could give our approach an unfair advantage. Therefore, we decided to use the dataset of Guo et al. (2023), which focuses on different medical topics. Another reason for using this dataset for the initial tuning was that it is rarely used by other publications, so we wouldn’t have any baselines to compare ourselves to on the dataset anyway.

The results of using different scales on this dataset (see Table 5) suggest that increasing the scale size significantly improves performance up to scale 0-14. The scale 0-19 performs only slightly worse, while further increasing the scale size does not improve performance. Since 0-19 still performs quite well, e.g., in terms of MAP, we decided to use this scale size for our main evaluation, thus giving the LLM as many degrees of freedom as possible without compromising performance.

Similarly, in preliminary experiments when using the dataset presented in Guo et al. (2023), we also investigate the performance of different dense rankers that are used in the second stage of LGAR (see Table 6). While monoT5 with title and research questions as query achieves the best ranking performance, using monoT5 with title-only as query significantly improves performance after screening about 20% of the papers, resulting in a better performance on the metrics regarding potential workload savings. Therefore, we decided to use this configuration as our dense ranker for our main evaluation. The results of these subsequent experiments confirmed our choice to use monoT5 with title-only as the query (see Table 10).

A.2 Detailed Evaluation of LGAR

In Table 7, the more detailed results of the main findings presented in this work are shown. Here, we report separately the results of all four splits with different medical topics of the TAR2019 dataset. In the main evaluation, the averaged results of all SLRs of TAR2019 are reported in order to keep the table clear. It is worth noting that the splits shown in this table are not equal in size, e.g., the Intervention split includes 20 SLRs, while the Prognosis split includes only one SLR.

A.3 Detailed Results of Ablations

Table 8 shows detailed results of using different LLMs as initial ranker of LGAR. Llama3.3-70B performs best on most metrics for the TAR2019 dataset, which is also the trend when evaluated on the SYNERGY dataset.

Table 9 shows the results of using different scales on the SYNERGY and TAR2019 datasets. The results confirm our choice to use the 0-19 scale, which was identified as one of the best performing scales in preliminary experiments (see Table 5). Figure 7 depicts the distribution of relevance scores on the relevance scale averaged over all SLRs. Most papers receive a score of 0, which is expected, since most of the papers retrieved as candidates from scientific databases are not relevant to the respective SLR.

![Distribution of relevance scores averaged for all SLRs of SYNERGY and TAR2019. LLM: Llama3.3-70B. Scale: 0-19. Re-ranker: monoT5 with title-only as query.](page_1012_1342_388_246.png)

Figure 7: Distribution of relevance scores averaged for all SLRs of SYNERGY and TAR2019. LLM: Llama3.3-70B. Scale: 0-19. Re-ranker: monoT5 with title-only as query.

Table 10 shows detailed results of the comparison of dense ranking models briefly shown in Figure 5. The results confirm our choice of using monoT5 as a dense ranker with title-only as the query identified in our preliminary experiments (see Table 6). On both the SYNERGY and
<table>
  <tr>
    <th rowspan="2">Dataset Scale</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>@95%</th>
    <th>@100%</th>
  </tr>
  <tr>
    <td>Guo 0-1</td>
    <td>44.7</td>
    <td>51.4</td>
    <td>26.7</td>
    <td>59.0</td>
    <td>69.6</td>
    <td>79.6</td>
    <td>93.0</td>
    <td>46.1</td>
    <td>24.9</td>
  </tr>
  <tr>
    <td>0-2</td>
    <td>47.0</td>
    <td>71.9</td>
    <td>26.2</td>
    <td>60.4</td>
    <td>74.3</td>
    <td>89.9</td>
    <td>97.2</td>
    <td>65.1</td>
    <td>33.4</td>
  </tr>
  <tr>
    <td>0-4</td>
    <td>52.2</td>
    <td>74.8</td>
    <td>27.7</td>
    <td>64.2</td>
    <td>78.5</td>
    <td>92.1</td>
    <td>97.9</td>
    <td>68.9</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>0-9</td>
    <td>51.6</td>
    <td>75.5</td>
    <td>28.7</td>
    <td>62.6</td>
    <td>81.4</td>
    <td>91.7</td>
    <td>98.0</td>
    <td>72.2</td>
    <td>35.4</td>
  </tr>
  <tr>
    <td>0-14</td>
    <td><b>54.8</b></td>
    <td><b>80.9</b></td>
    <td>29.5</td>
    <td><b>65.2</b></td>
    <td>78.1</td>
    <td>91.9</td>
    <td><b>99.0</b></td>
    <td><b>73.0</b></td>
    <td>39.1</td>
  </tr>
  <tr>
    <td>0-19</td>
    <td>54.7</td>
    <td>75.1</td>
    <td>30.4</td>
    <td>63.4</td>
    <td>80.0</td>
    <td>91.8</td>
    <td>98.4</td>
    <td>69.0</td>
    <td>37.6</td>
  </tr>
  <tr>
    <td>0-24</td>
    <td>53.8</td>
    <td>74.8</td>
    <td>29.0</td>
    <td>64.0</td>
    <td>80.4</td>
    <td>91.8</td>
    <td>97.9</td>
    <td>68.0</td>
    <td><b>39.8</b></td>
  </tr>
  <tr>
    <td>0-29</td>
    <td>51.8</td>
    <td>76.9</td>
    <td>28.3</td>
    <td>61.0</td>
    <td>79.5</td>
    <td>91.2</td>
    <td>98.3</td>
    <td>70.9</td>
    <td>39.5</td>
  </tr>
</table>

Table 5: Results of different scales on dataset of Guo et al. (2023). Re-ranker: monoT5 with title-only as query. LLM: Llama3.3-70B.

<table>
  <tr>
    <th rowspan="2">Dataset Model</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>@95%</th>
    <th>@100%</th>
  </tr>
  <tr>
    <td>Guo BM25 (T)</td>
    <td>9.5</td>
    <td>32.5</td>
    <td>2.1</td>
    <td>19.5</td>
    <td>30.0</td>
    <td>47.2</td>
    <td>76.4</td>
    <td>27.2</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>15.5</td>
    <td>32.2</td>
    <td>9.8</td>
    <td>23.8</td>
    <td>35.7</td>
    <td>51.5</td>
    <td>80.6</td>
    <td>27.0</td>
    <td>17.4</td>
  </tr>
  <tr>
    <td>ColBERT (T)</td>
    <td>9.9</td>
    <td>38.0</td>
    <td>3.6</td>
    <td>12.4</td>
    <td>25.8</td>
    <td>46.8</td>
    <td>86.5</td>
    <td>32.9</td>
    <td>17.0</td>
  </tr>
  <tr>
    <td>ColBERT (T+R)</td>
    <td>9.5</td>
    <td>32.8</td>
    <td>2.6</td>
    <td>12.9</td>
    <td>23.1</td>
    <td>42.1</td>
    <td>81.7</td>
    <td>27.3</td>
    <td>13.6</td>
  </tr>
  <tr>
    <td>monoBERT (T)</td>
    <td>14.5</td>
    <td>41.1</td>
    <td>7.0</td>
    <td>28.6</td>
    <td>42.3</td>
    <td>61.8</td>
    <td>86.3</td>
    <td>37.4</td>
    <td>20.6</td>
  </tr>
  <tr>
    <td>monoBERT (T+R)</td>
    <td>18.1</td>
    <td>41.7</td>
    <td>7.9</td>
    <td>34.1</td>
    <td>46.2</td>
    <td><b>63.5</b></td>
    <td>88.4</td>
    <td>39.6</td>
    <td>20.7</td>
  </tr>
  <tr>
    <td>monoT5_3B (T)</td>
    <td>15.7</td>
    <td><b>45.3</b></td>
    <td>8.8</td>
    <td>34.5</td>
    <td>47.3</td>
    <td>63.2</td>
    <td><b>89.8</b></td>
    <td><b>40.8</b></td>
    <td><b>21.3</b></td>
  </tr>
  <tr>
    <td>monoT5_3B (T+R)</td>
    <td><b>19.1</b></td>
    <td>37.5</td>
    <td><b>12.3</b></td>
    <td><b>37.2</b></td>
    <td><b>48.6</b></td>
    <td>61.8</td>
    <td>87.0</td>
    <td>32.8</td>
    <td>19.7</td>
  </tr>
</table>

Table 6: Results of dense rankers on dataset of Guo et al. (2023). T = title of the SLR, R = research questions of the SLR used as query.

TAR2019 datasets, this configuration performs best on most metrics.

A.4 2-shot Experiments

Since previous research on LLMs has shown that adding examples to the prompts can be beneficial to their performance (Brown et al., 2020), we test the few-shot setting for the sake of completeness: admittedly, each paper can be considered irrelevant/relevant due to various aspects of the selection criteria, i.e., it is an extremely difficult task to generate examples that are representative for all irrelevant/relevant papers.

All abstract screening datasets provide binary relevance labels, yet we work with assigning graded relevance judgments ranging from 0 to \( k \). We use only two examples (one positive and one negative) for few-shot prompting. In the prompt, the positive example is always followed by the negative example. In practice, a domain expert could provide these examples together with a formulation of the reasoning they applied. As an approximation, we select the few-shot examples from zero-shot runs of the respective prompting technique, i.e., our default prompt or CoT prompt, for each SLR as follows. We select the set of examples to which the zero-shot model assigns either the score 0 (for instances labeled as irrelevant in the original dataset) or \( k \) (for instances labeled as relevant in the original dataset), that exhibit the expected format, i.e., the decision of the model can be extracted using regular expressions. We then select one positive and one negative example from these sets randomly and remove them from the evaluation sets.

In Table 11, we compare the performance of 0-shot prompting with 2-shot, 2-shot CoT, and 2-shot CoT with self-consistency. The results suggest that adding examples does not improve the performance of LGAR, and in fact it is harmful. The reason for this could be that the randomly chosen examples are not representative of all irrelevant/relevant papers. Furthermore, since the examples are generated by the LLM, they could also contain reasoning errors. This could be improved by having domain experts generate the examples. However, the challenge of finding good representatives for an SLR remains. Therefore, we did not pursue the use of few-shot examples.

A.5 Hardware and Experiment Statistics

We ran our experiments on Nvidia A40 (40 GB) and Nvidia A100 (80 GB) GPUs, using up to 8 GPUs in parallel. The total amount of GPU hours was 7345h on A40 and about 3664h on A100.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="6">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th rowspan="2">WSS</th>
    <th colspan="2">@95% @100%</th>
  </tr>
  <tr>
    <th>MAP</th>
    <th>TNR@95%</th>
    <th>R@1%</th>
    <th>R@5%</th>
    <th>R@10%</th>
    <th>R@20%</th>
    <th>R@50%</th>
    <th>@95%</th>
    <th>@100%</th>
  </tr>
  <tr>
    <td rowspan="7">SYNERGY</td>
    <td>random reranking</td>
    <td>4.9</td>
    <td>6.2</td>
    <td>1.2</td>
    <td>5.1</td>
    <td>10.4</td>
    <td>20.0</td>
    <td>50.3</td>
    <td>3.0</td>
    <td>3.0</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>12.5</td>
    <td>36.3</td>
    <td>9.0</td>
    <td>26.4</td>
    <td>40.0</td>
    <td>54.4</td>
    <td>80.3</td>
    <td>33.6</td>
    <td>21.8</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>18.8</td>
    <td>52.3</td>
    <td>14.1</td>
    <td>37.5</td>
    <td>54.5</td>
    <td>69.1</td>
    <td>91.1</td>
    <td>50.0</td>
    <td>35.4</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours)<sup>†</sup></b></td>
    <td>34.0</td>
    <td>63.6</td>
    <td>25.8</td>
    <td>55.0</td>
    <td>67.8</td>
    <td>80.5</td>
    <td>93.1</td>
    <td>59.5</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td>38.9</td>
    <td>59.8</td>
    <td>33.3</td>
    <td>58.4</td>
    <td>70.1</td>
    <td>79.7</td>
    <td>91.4</td>
    <td>58.6</td>
    <td>40.9</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5)<sup>†</sup></td>
    <td>36.8</td>
    <td>63.8</td>
    <td>32.7</td>
    <td>57.7</td>
    <td>71.0</td>
    <td>81.2</td>
    <td>94.3</td>
    <td>63.6</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td><b>40.7</b></td>
    <td><b>67.0</b></td>
    <td><b>34.1</b></td>
    <td><b>59.3</b></td>
    <td><b>72.0</b></td>
    <td><b>81.9</b></td>
    <td><b>94.4</b></td>
    <td><b>65.2</b></td>
    <td><b>49.3</b></td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>38.8</td>
    <td>62.9</td>
    <td>33.4</td>
    <td>59.3</td>
    <td>70.3</td>
    <td>81.4</td>
    <td>92.8</td>
    <td>62.5</td>
    <td>45.8</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td><b>40.7</b></td>
    <td><b>66.4</b></td>
    <td><b>34.7</b></td>
    <td><b>58.6</b></td>
    <td><b>70.7</b></td>
    <td><b>82.6</b></td>
    <td><b>93.5</b></td>
    <td><b>64.5</b></td>
    <td><b>50.4</b></td>
  </tr>
  <tr>
    <td rowspan="7">Interv.</td>
    <td>random reranking</td>
    <td>6.8</td>
    <td>10.2</td>
    <td>1.0</td>
    <td>5.5</td>
    <td>10.5</td>
    <td>20.0</td>
    <td>52.6</td>
    <td>6.6</td>
    <td>7.6</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>24.9</td>
    <td>44.6</td>
    <td>13.6</td>
    <td>35.9</td>
    <td>48.1</td>
    <td>62.3</td>
    <td>85.2</td>
    <td>41.0</td>
    <td>31.7</td>
  </tr>
  <tr>
    <td>Wang et al. (2023c) (Os)</td>
    <td>16.0</td>
    <td>-</td>
    <td>5.4</td>
    <td>21.0</td>
    <td>32.8</td>
    <td>50.4</td>
    <td>-</td>
    <td>36.2</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>36.0</td>
    <td>63.7</td>
    <td>21.7</td>
    <td>47.5</td>
    <td>62.6</td>
    <td>76.8</td>
    <td>94.0</td>
    <td>59.0</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (2024)</b></td>
    <td>45.0</td>
    <td>-</td>
    <td>-</td>
    <td>52.6</td>
    <td>69.7</td>
    <td>80.8</td>
    <td>96.2</td>
    <td>59.2</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (theirs)</b></td>
    <td>47.1</td>
    <td>67.1</td>
    <td>25.3</td>
    <td>57.4</td>
    <td>71.1</td>
    <td>82.8</td>
    <td>96.3</td>
    <td>62.7</td>
    <td>54.2</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours)<sup>†</sup></b></td>
    <td>50.4</td>
    <td>68.4</td>
    <td><b>30.2</b></td>
    <td>60.6</td>
    <td>73.2</td>
    <td>83.8</td>
    <td>96.9</td>
    <td>65.2</td>
    <td>49.8</td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td><b>57.0</b></td>
    <td>61.9</td>
    <td>29.1</td>
    <td><b>65.8</b></td>
    <td>78.0</td>
    <td>87.6</td>
    <td>94.2</td>
    <td>60.4</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5)<sup>†</sup></td>
    <td>55.0</td>
    <td><b>75.0</b></td>
    <td>27.8</td>
    <td>62.2</td>
    <td>76.9</td>
    <td>88.9</td>
    <td>96.0</td>
    <td>68.6</td>
    <td><b>63.2</b></td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td>56.8</td>
    <td><b>75.0</b></td>
    <td>28.5</td>
    <td>65.2</td>
    <td><b>78.6</b></td>
    <td>88.4</td>
    <td>95.6</td>
    <td><b>70.7</b></td>
    <td>61.9</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>54.3</td>
    <td>74.2</td>
    <td>25.6</td>
    <td>63.0</td>
    <td>76.6</td>
    <td>89.0</td>
    <td>95.4</td>
    <td>69.1</td>
    <td>62.2</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td>54.5</td>
    <td>73.8</td>
    <td>27.0</td>
    <td>64.8</td>
    <td>77.6</td>
    <td><b>89.2</b></td>
    <td>94.9</td>
    <td>68.5</td>
    <td><b>63.2</b></td>
  </tr>
  <tr>
    <td>Wang et al. (2023c) (ft)</td>
    <td>45.6</td>
    <td>-</td>
    <td>21.6</td>
    <td>58.0</td>
    <td>73.7</td>
    <td>84.2</td>
    <td>-</td>
    <td>64.6</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>BioBERT-ft (T)</td>
    <td>45.9</td>
    <td>70.8</td>
    <td>24.5</td>
    <td>55.5</td>
    <td>73.3</td>
    <td>84.2</td>
    <td><b>97.0</b></td>
    <td>66.8</td>
    <td>53.6</td>
  </tr>
  <tr>
    <td rowspan="7">DTA</td>
    <td>random reranking</td>
    <td>7.8</td>
    <td>5.8</td>
    <td>1.4</td>
    <td>5.9</td>
    <td>10.6</td>
    <td>20.4</td>
    <td>49.4</td>
    <td>3.6</td>
    <td>2.8</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>16.7</td>
    <td>50.5</td>
    <td>13.9</td>
    <td>37.6</td>
    <td>50.1</td>
    <td>65.0</td>
    <td>86.0</td>
    <td>46.4</td>
    <td>33.1</td>
  </tr>
  <tr>
    <td>Wang et al. (2023c) (Os)</td>
    <td>9.2</td>
    <td>-</td>
    <td>2.4</td>
    <td>13.2</td>
    <td>23.8</td>
    <td>39.1</td>
    <td>-</td>
    <td>25.8</td>
    <td>21.0</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>20.4</td>
    <td>64.0</td>
    <td>11.1</td>
    <td>45.0</td>
    <td>60.5</td>
    <td>74.6</td>
    <td>93.0</td>
    <td>58.4</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (2024)</b></td>
    <td>31.5</td>
    <td>70.6</td>
    <td>25.0</td>
    <td>52.5</td>
    <td>67.0</td>
    <td>84.4</td>
    <td>94.1</td>
    <td>55.6</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (theirs)</b></td>
    <td>37.3</td>
    <td>70.6</td>
    <td>25.0</td>
    <td>52.5</td>
    <td>67.0</td>
    <td>84.4</td>
    <td>94.1</td>
    <td>64.5</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours)<sup>†</sup></b></td>
    <td>35.9</td>
    <td>77.5</td>
    <td>23.4</td>
    <td>56.4</td>
    <td>74.0</td>
    <td>85.3</td>
    <td>94.9</td>
    <td>70.0</td>
    <td><b>62.6</b></td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td>38.0</td>
    <td>64.5</td>
    <td>30.6</td>
    <td>52.8</td>
    <td>66.6</td>
    <td>78.0</td>
    <td>90.3</td>
    <td>57.7</td>
    <td>34.1</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5)<sup>†</sup></td>
    <td>34.5</td>
    <td>78.1</td>
    <td>25.8</td>
    <td>57.6</td>
    <td>73.1</td>
    <td>84.2</td>
    <td>95.2</td>
    <td>71.3</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td><b>38.1</b></td>
    <td>76.8</td>
    <td><b>30.8</b></td>
    <td><b>59.9</b></td>
    <td>72.9</td>
    <td>85.0</td>
    <td>94.6</td>
    <td>69.5</td>
    <td>57.3</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>36.5</td>
    <td><b>79.4</b></td>
    <td>20.4</td>
    <td>57.0</td>
    <td><b>75.4</b></td>
    <td><b>87.0</b></td>
    <td>95.9</td>
    <td><b>71.8</b></td>
    <td>59.4</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td>37.6</td>
    <td>77.5</td>
    <td>24.9</td>
    <td>58.9</td>
    <td>73.4</td>
    <td>85.5</td>
    <td>95.1</td>
    <td>69.9</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td>Wang et al. (2023c) (ft)</td>
    <td>31.8</td>
    <td>-</td>
    <td>26.0</td>
    <td>50.0</td>
    <td>67.1</td>
    <td>81.7</td>
    <td>-</td>
    <td>68.6</td>
    <td>58.4</td>
  </tr>
  <tr>
    <td>BioBERT-ft (T)</td>
    <td>35.4</td>
    <td>76.0</td>
    <td>25.3</td>
    <td>54.8</td>
    <td>69.7</td>
    <td>82.5</td>
    <td><b>96.7</b></td>
    <td>69.5</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td rowspan="7">Qualitative</td>
    <td>random reranking</td>
    <td>1.6</td>
    <td>24.6</td>
    <td>0.9</td>
    <td>3.4</td>
    <td>11.0</td>
    <td>21.7</td>
    <td>58.0</td>
    <td>20.0</td>
    <td>21.6</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>6.9</td>
    <td>38.3</td>
    <td>3.2</td>
    <td>16.2</td>
    <td>26.1</td>
    <td>36.0</td>
    <td>46.4</td>
    <td>33.2</td>
    <td>18.3</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>15.0</td>
    <td>64.2</td>
    <td>9.9</td>
    <td>28.4</td>
    <td>36.0</td>
    <td>41.0</td>
    <td>98.6</td>
    <td>59.4</td>
    <td>51.1</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (2024)</b></td>
    <td>15.9</td>
    <td>-</td>
    <td>-</td>
    <td>50.5</td>
    <td>60.0</td>
    <td>67.3</td>
    <td>97.8</td>
    <td>57.6</td>
    <td>50.7</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (theirs)</b></td>
    <td>17.8</td>
    <td>75.8</td>
    <td>11.3</td>
    <td>28.4</td>
    <td>85.1</td>
    <td>91.9</td>
    <td>99.6</td>
    <td>72.0</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours)<sup>†</sup></b></td>
    <td>22.4</td>
    <td>66.6</td>
    <td>12.2</td>
    <td>31.5</td>
    <td>65.5</td>
    <td>71.0</td>
    <td>99.6</td>
    <td>61.1</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td><b>29.9</b></td>
    <td><b>86.7</b></td>
    <td><b>39.2</b></td>
    <td>62.3</td>
    <td>74.5</td>
    <td><b>98.4</b></td>
    <td>99.3</td>
    <td><b>81.6</b></td>
    <td>54.0</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5)<sup>†</sup></td>
    <td>27.3</td>
    <td>81.0</td>
    <td>13.5</td>
    <td>61.5</td>
    <td>70.5</td>
    <td>73.6</td>
    <td><b>100.0</b></td>
    <td>75.4</td>
    <td>71.8</td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td>29.1</td>
    <td>86.6</td>
    <td>38.5</td>
    <td>62.4</td>
    <td>69.1</td>
    <td><b>100.0</b></td>
    <td>80.6</td>
    <td><b>80.1</b></td>
    <td>-</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>26.1</td>
    <td>81.3</td>
    <td>13.1</td>
    <td>63.3</td>
    <td>90.5</td>
    <td><b>100.0</b></td>
    <td>78.5</td>
    <td>75.7</td>
    <td>-</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td>28.2</td>
    <td>82.0</td>
    <td>13.1</td>
    <td><b>63.7</b></td>
    <td><b>91.4</b></td>
    <td><b>100.0</b></td>
    <td>81.0</td>
    <td>73.5</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="7">Prognosis</td>
    <td>random reranking</td>
    <td>6.3</td>
    <td>6.3</td>
    <td>1.4</td>
    <td>6.0</td>
    <td>11.0</td>
    <td>20.0</td>
    <td>50.8</td>
    <td>1.8</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>9.2</td>
    <td>15.3</td>
    <td>0.5</td>
    <td>7.8</td>
    <td>16.2</td>
    <td>38.0</td>
    <td>73.4</td>
    <td>10.2</td>
    <td>5.1</td>
  </tr>
  <tr>
    <td>monoT5 (T)</td>
    <td>18.5</td>
    <td>43.9</td>
    <td>5.7</td>
    <td>18.2</td>
    <td>31.2</td>
    <td>59.9</td>
    <td>92.7</td>
    <td>36.7</td>
    <td>16.6</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (2024)</b></td>
    <td>43.0</td>
    <td>-</td>
    <td>-</td>
    <td>40.0</td>
    <td>65.3</td>
    <td>80.0</td>
    <td>98.4</td>
    <td>54.3</td>
    <td>28.9</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (theirs)</b></td>
    <td>49.4</td>
    <td>68.3</td>
    <td>13.0</td>
    <td>43.8</td>
    <td>66.2</td>
    <td>82.8</td>
    <td>97.9</td>
    <td>60.7</td>
    <td>43.8</td>
  </tr>
  <tr>
    <td><b>Akinseloyin et al. (ours)<sup>†</sup></b></td>
    <td>57.5</td>
    <td>71.8</td>
    <td>14.1</td>
    <td>54.2</td>
    <td>74.5</td>
    <td>85.4</td>
    <td>99.0</td>
    <td>63.7</td>
    <td><b>44.6</b></td>
  </tr>
  <tr>
    <td>LGAR (T+R, random rerank)</td>
    <td>67.1</td>
    <td>81.1</td>
    <td><b>15.1</b></td>
    <td><b>62.1</b></td>
    <td>84.4</td>
    <td>93.3</td>
    <td><b>99.8</b></td>
    <td>72.8</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>LGAR (T, monoT5)<sup>†</sup></td>
    <td>67.7</td>
    <td>81.2</td>
    <td><b>15.1</b></td>
    <td>62.0</td>
    <td>83.8</td>
    <td>93.8</td>
    <td>99.5</td>
    <td>74.1</td>
    <td>29.8</td>
  </tr>
  <tr>
    <td>LGAR (T+R, monoT5)</td>
    <td><b>67.9</b></td>
    <td>83.6</td>
    <td><b>15.1</b></td>
    <td>61.5</td>
    <td>84.4</td>
    <td>93.8</td>
    <td>99.5</td>
    <td>74.3</td>
    <td>32.8</td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>65.2</td>
    <td>82.4</td>
    <td>14.6</td>
    <td>61.5</td>
    <td>84.9</td>
    <td>93.8</td>
    <td>99.5</td>
    <td>73.4</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>+ CoT (n=3)</td>
    <td>67.1</td>
    <td><b>84.9</b></td>
    <td><b>15.1</b></td>
    <td>60.4</td>
    <td><b>84.9</b></td>
    <td><b>95.3</b></td>
    <td>99.5</td>
    <td><b>75.5</b></td>
    <td>32.6</td>
  </tr>
</table>

Table 7: Main results (detailed). T = title of the SLR, R = research questions of the SLR. <b>Akinseloyin et al. (2024)</b>: ours = uses our annotated criteria, theirs = uses their criteria. In all combinations, monoT5 uses the title as query and the LLM the scale 0-19. <b>Wang et al. (2023c):</b> Os=zero-shot, ft=fine-tuned. † indicates that the approaches get the same information on SLR and paper.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">LLM</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>@95%</th>
    <th>@100%</th>
  </tr>
  <tr>
    <td rowspan="5">SYNERGY</td>
    <td>Llama3.1-8B</td>
    <td>31.9</td>
    <td>61.3</td>
    <td>22.9</td>
    <td>53.4</td>
    <td>67.0</td>
    <td>80.9</td>
    <td>93.3</td>
    <td>60.5</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td>40.7</td>
    <td>67.0</td>
    <td><b>34.1</b></td>
    <td>59.3</td>
    <td>72.0</td>
    <td>81.9</td>
    <td><b>94.4</b></td>
    <td><b>65.2</b></td>
    <td><b>49.3</b></td>
  </tr>
  <tr>
    <td>Mistral-123B</td>
    <td>38.8</td>
    <td>65.8</td>
    <td>33.1</td>
    <td><b>61.6</b></td>
    <td><b>73.3</b></td>
    <td>82.0</td>
    <td>93.6</td>
    <td>63.4</td>
    <td>47.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>38.7</td>
    <td>63.2</td>
    <td>33.4</td>
    <td>57.0</td>
    <td>68.1</td>
    <td>78.6</td>
    <td>93.7</td>
    <td>61.4</td>
    <td>45.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td><b>41.3</b></td>
    <td><b>68.5</b></td>
    <td>32.7</td>
    <td>60.0</td>
    <td>72.0</td>
    <td><b>83.0</b></td>
    <td>94.1</td>
    <td>64.8</td>
    <td>47.7</td>
  </tr>
  <tr>
    <td rowspan="5">TAR2019</td>
    <td>Llama3.1-8B</td>
    <td>42.3</td>
    <td>69.6</td>
    <td>24.7</td>
    <td>57.5</td>
    <td>71.5</td>
    <td>86.1</td>
    <td>95.2</td>
    <td>65.3</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>Llama3.3-70B</td>
    <td><b>50.6</b></td>
    <td><b>76.5</b></td>
    <td>29.3</td>
    <td><b>63.6</b></td>
    <td><b>76.7</b></td>
    <td>88.3</td>
    <td>95.8</td>
    <td><b>71.2</b></td>
    <td><b>60.9</b></td>
  </tr>
  <tr>
    <td>Mistral-123B</td>
    <td>48.4</td>
    <td>70.8</td>
    <td>28.5</td>
    <td>62.7</td>
    <td>75.3</td>
    <td>86.4</td>
    <td>95.9</td>
    <td>68.1</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>47.6</td>
    <td>73.4</td>
    <td>28.6</td>
    <td>60.4</td>
    <td>74.0</td>
    <td>84.6</td>
    <td><b>96.2</b></td>
    <td>68.6</td>
    <td>58.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>50.2</td>
    <td>73.0</td>
    <td><b>29.8</b></td>
    <td>63.1</td>
    <td>76.1</td>
    <td><b>89.0</b></td>
    <td>95.8</td>
    <td>68.9</td>
    <td>56.0</td>
  </tr>
</table>

Table 8: Comparison of LLMs. Scale: 0-19. Re-ranker: monoT5 with title-only as query. 0-shot prompting.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Scale</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>@95%</th>
    <th>@100%</th>
  </tr>
  <tr>
    <td rowspan="8">SYNERGY</td>
    <td>0-1</td>
    <td>33.7</td>
    <td>59.4</td>
    <td>29.2</td>
    <td>55.1</td>
    <td>67.6</td>
    <td>79.3</td>
    <td>93.2</td>
    <td>58.2</td>
    <td>41.2</td>
  </tr>
  <tr>
    <td>0-2</td>
    <td>35.0</td>
    <td>62.9</td>
    <td>30.8</td>
    <td>56.0</td>
    <td>70.1</td>
    <td>82.1</td>
    <td>93.6</td>
    <td>63.6</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>0-4</td>
    <td>36.4</td>
    <td>63.2</td>
    <td>30.6</td>
    <td>56.6</td>
    <td>70.0</td>
    <td>80.6</td>
    <td>93.7</td>
    <td>63.8</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>0-9</td>
    <td>39.4</td>
    <td>67.4</td>
    <td>32.3</td>
    <td>57.6</td>
    <td>69.2</td>
    <td>82.2</td>
    <td><b>94.6</b></td>
    <td>64.8</td>
    <td>50.1</td>
  </tr>
  <tr>
    <td>0-14</td>
    <td>39.9</td>
    <td><b>69.2</b></td>
    <td>33.3</td>
    <td>57.4</td>
    <td>69.7</td>
    <td><b>82.3</b></td>
    <td>94.5</td>
    <td>65.5</td>
    <td><b>50.8</b></td>
  </tr>
  <tr>
    <td>0-19</td>
    <td><b>40.7</b></td>
    <td>67.0</td>
    <td><b>34.1</b></td>
    <td><b>59.3</b></td>
    <td><b>72.0</b></td>
    <td>81.9</td>
    <td>94.4</td>
    <td>65.2</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>0-24</td>
    <td>39.6</td>
    <td>66.0</td>
    <td>32.7</td>
    <td>57.8</td>
    <td>70.9</td>
    <td>81.5</td>
    <td>94.4</td>
    <td>64.5</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>0-29</td>
    <td>39.5</td>
    <td>66.4</td>
    <td>33.3</td>
    <td>59.1</td>
    <td>71.1</td>
    <td>82.2</td>
    <td>94.2</td>
    <td><b>66.2</b></td>
    <td>49.0</td>
  </tr>
  <tr>
    <td rowspan="6">TAR2019</td>
    <td>0-1</td>
    <td>43.2</td>
    <td>67.1</td>
    <td>25.2</td>
    <td>57.3</td>
    <td>73.4</td>
    <td>82.0</td>
    <td>95.1</td>
    <td>63.2</td>
    <td>52.9</td>
  </tr>
  <tr>
    <td>0-2</td>
    <td>45.5</td>
    <td>75.4</td>
    <td>25.0</td>
    <td>60.7</td>
    <td>75.2</td>
    <td>86.8</td>
    <td>95.6</td>
    <td>69.7</td>
    <td>59.2</td>
  </tr>
  <tr>
    <td>0-4</td>
    <td>47.1</td>
    <td>75.6</td>
    <td>27.0</td>
    <td>61.8</td>
    <td>76.5</td>
    <td>87.5</td>
    <td>95.6</td>
    <td>70.1</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>0-9</td>
    <td>48.1</td>
    <td><b>77.6</b></td>
    <td>27.3</td>
    <td>61.0</td>
    <td>76.2</td>
    <td>88.6</td>
    <td><b>96.4</b></td>
    <td>70.7</td>
    <td><b>63.3</b></td>
  </tr>
  <tr>
    <td>0-14</td>
    <td>48.0</td>
    <td>77.5</td>
    <td>28.9</td>
    <td>61.6</td>
    <td>75.6</td>
    <td>88.0</td>
    <td>96.2</td>
    <td>70.9</td>
    <td>63.2</td>
  </tr>
  <tr>
    <td>0-19</td>
    <td><b>50.6</b></td>
    <td>76.5</td>
    <td>29.3</td>
    <td><b>63.6</b></td>
    <td><b>76.7</b></td>
    <td>88.3</td>
    <td>95.8</td>
    <td><b>71.2</b></td>
    <td>60.9</td>
  </tr>
  <tr>
    <td>0-24</td>
    <td>48.3</td>
    <td>76.8</td>
    <td><b>29.6</b></td>
    <td>62.0</td>
    <td>76.5</td>
    <td><b>88.8</b></td>
    <td>96.3</td>
    <td>69.9</td>
    <td>62.4</td>
  </tr>
  <tr>
    <td>0-29</td>
    <td>48.9</td>
    <td>77.1</td>
    <td>27.1</td>
    <td>62.3</td>
    <td>76.3</td>
    <td>88.6</td>
    <td><b>96.4</b></td>
    <td>69.6</td>
    <td>62.8</td>
  </tr>
</table>

Table 9: Comparison of different Likert scale ranges in LLM. LLM: Llama3.3-70B. Scale: 0-19. Re-ranker: monoT5 with title-only as query.

A.6 Use of AI Assistants

Copilot and ChatGPT were used only as coding assistants. This was especially useful for debugging existing code and for understanding third-party code. We have not used AI assistants for writing this paper.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS @95% @100%</th>
  </tr>
  <tr>
    <th>MAP</th>
    <th>TNR@95%</th>
    <th>R@1%</th>
    <th>R@5%</th>
    <th>R@10%</th>
    <th>R@20%</th>
    <th>R@50%</th>
    <th>WSS @95%</th>
    <th>WSS @100%</th>
  </tr>
  <tr>
    <td rowspan="6">SYNERGY</td>
    <td>BM25 (T)</td>
    <td>10.3</td>
    <td>36.8</td>
    <td>7.3</td>
    <td>24.2</td>
    <td>35.4</td>
    <td>49.3</td>
    <td>77.8</td>
    <td>32.0</td>
    <td>22.9</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>12.5</td>
    <td>36.3</td>
    <td>9.0</td>
    <td>26.4</td>
    <td>40.0</td>
    <td>54.4</td>
    <td>80.3</td>
    <td>33.6</td>
    <td>21.8</td>
  </tr>
  <tr>
    <td>CoBERT (T)</td>
    <td>10.2</td>
    <td>32.4</td>
    <td>7.4</td>
    <td>22.9</td>
    <td>34.2</td>
    <td>50.7</td>
    <td>80.3</td>
    <td>32.9</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>CoBERT (T+R)</td>
    <td>8.9</td>
    <td>29.7</td>
    <td>5.1</td>
    <td>18.2</td>
    <td>31.7</td>
    <td>47.5</td>
    <td>79.4</td>
    <td>29.6</td>
    <td>17.0</td>
  </tr>
  <tr>
    <td>monoBERT (T)</td>
    <td>16.6</td>
    <td>44.6</td>
    <td>13.8</td>
    <td>33.6</td>
    <td>47.8</td>
    <td>64.1</td>
    <td>87.2</td>
    <td>42.6</td>
    <td>30.3</td>
  </tr>
  <tr>
    <td>monoBERT (T+R)</td>
    <td>17.8</td>
    <td>42.6</td>
    <td>14.6</td>
    <td>32.6</td>
    <td>48.4</td>
    <td>64.3</td>
    <td>87.1</td>
    <td>40.6</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td rowspan="2">monoT5 (T)</td>
    <td><b>18.8</b></td>
    <td><b>52.3</b></td>
    <td><b>14.1</b></td>
    <td><b>37.5</b></td>
    <td><b>54.5</b></td>
    <td><b>69.1</b></td>
    <td><b>91.1</b></td>
    <td><b>50.0</b></td>
    <td><b>35.4</b></td>
  </tr>
  <tr>
    <td>monoT5 (T+R)</td>
    <td>17.0</td>
    <td>35.0</td>
    <td>14.4</td>
    <td>30.3</td>
    <td>42.3</td>
    <td>58.6</td>
    <td>83.0</td>
    <td>32.5</td>
    <td>22.6</td>
  </tr>
  <tr>
    <td rowspan="6">TAR2019</td>
    <td>BM25 (T)</td>
    <td>18.1</td>
    <td>38.9</td>
    <td>9.5</td>
    <td>30.2</td>
    <td>40.5</td>
    <td>57.4</td>
    <td>81.4</td>
    <td>36.1</td>
    <td>28.4</td>
  </tr>
  <tr>
    <td>BM25 (T+R)</td>
    <td>21.1</td>
    <td>44.8</td>
    <td>12.6</td>
    <td>34.2</td>
    <td>46.2</td>
    <td>60.5</td>
    <td>82.5</td>
    <td>40.9</td>
    <td>30.4</td>
  </tr>
  <tr>
    <td>CoBERT (T)</td>
    <td>15.8</td>
    <td>45.1</td>
    <td>6.1</td>
    <td>21.2</td>
    <td>36.1</td>
    <td>54.5</td>
    <td>84.8</td>
    <td>40.8</td>
    <td>33.5</td>
  </tr>
  <tr>
    <td>CoBERT (T+R)</td>
    <td>14.5</td>
    <td>43.7</td>
    <td>4.7</td>
    <td>24.2</td>
    <td>34.2</td>
    <td>52.4</td>
    <td>83.7</td>
    <td>39.4</td>
    <td>33.7</td>
  </tr>
  <tr>
    <td>monoBERT (T)</td>
    <td>23.8</td>
    <td>50.8</td>
    <td>13.2</td>
    <td>38.5</td>
    <td>51.3</td>
    <td>64.8</td>
    <td>89.4</td>
    <td>47.4</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td>monoBERT (T+R)</td>
    <td>27.4</td>
    <td>54.4</td>
    <td>16.9</td>
    <td>40.9</td>
    <td>54.3</td>
    <td>71.2</td>
    <td>90.8</td>
    <td>49.9</td>
    <td>41.0</td>
  </tr>
  <tr>
    <td rowspan="2">monoT5 (T)</td>
    <td><b>30.1</b></td>
    <td><b>63.2</b></td>
    <td><b>17.7</b></td>
    <td><b>44.6</b></td>
    <td><b>59.3</b></td>
    <td><b>73.4</b></td>
    <td><b>94.0</b></td>
    <td><b>58.2</b></td>
    <td><b>50.7</b></td>
  </tr>
  <tr>
    <td>monoT5 (T+R)</td>
    <td>29.6</td>
    <td>59.9</td>
    <td>19.0</td>
    <td>42.6</td>
    <td>57.1</td>
    <td>72.8</td>
    <td>93.2</td>
    <td>56.1</td>
    <td>45.8</td>
  </tr>
</table>

Table 10: Comparison of dense ranking models. T = title of the SLR, R = research questions of the SLR used as query. All models have been trained on MS MACRO.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Prompt</th>
    <th colspan="7">MAP TNR@95% R@1% R@5% R@10% R@20% R@50%</th>
    <th colspan="2">WSS @95% @100%</th>
  </tr>
  <tr>
    <th>MAP</th>
    <th>TNR@95%</th>
    <th>R@1%</th>
    <th>R@5%</th>
    <th>R@10%</th>
    <th>R@20%</th>
    <th>R@50%</th>
    <th>WSS @95%</th>
    <th>WSS @100%</th>
  </tr>
  <tr>
    <td rowspan="4">SYNERGY</td>
    <td>0-shot</td>
    <td><b>40.7</b></td>
    <td><b>67.0</b></td>
    <td><b>34.1</b></td>
    <td><b>59.3</b></td>
    <td><b>72.0</b></td>
    <td><b>81.9</b></td>
    <td><b>94.4</b></td>
    <td><b>65.2</b></td>
    <td><b>49.3</b></td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>38.7</td>
    <td>60.5</td>
    <td>34.1</td>
    <td>60.6</td>
    <td>71.8</td>
    <td>80.7</td>
    <td>92.8</td>
    <td>59.5</td>
    <td>44.8</td>
  </tr>
  <tr>
    <td>2-shot CoT</td>
    <td>37.0</td>
    <td>65.1</td>
    <td>32.3</td>
    <td>59.0</td>
    <td>71.8</td>
    <td><b>83.4</b></td>
    <td>93.6</td>
    <td>62.8</td>
    <td>48.2</td>
  </tr>
  <tr>
    <td>2-shot CoT (n=3)</td>
    <td>39.2</td>
    <td>66.2</td>
    <td>32.5</td>
    <td>58.7</td>
    <td>71.0</td>
    <td>82.8</td>
    <td>94.1</td>
    <td>64.2</td>
    <td><b>49.9</b></td>
  </tr>
  <tr>
    <td rowspan="4">TAR2019</td>
    <td>0-shot</td>
    <td><b>50.6</b></td>
    <td><b>76.5</b></td>
    <td><b>29.3</b></td>
    <td><b>63.6</b></td>
    <td><b>76.7</b></td>
    <td><b>88.3</b></td>
    <td>95.8</td>
    <td><b>71.2</b></td>
    <td><b>60.9</b></td>
  </tr>
  <tr>
    <td>2-shot</td>
    <td>45.4</td>
    <td>72.5</td>
    <td>27.3</td>
    <td>57.8</td>
    <td>76.0</td>
    <td>85.2</td>
    <td>95.7</td>
    <td>68.2</td>
    <td>57.1</td>
  </tr>
  <tr>
    <td>2-shot CoT</td>
    <td>43.5</td>
    <td>72.0</td>
    <td>25.7</td>
    <td>57.5</td>
    <td>69.8</td>
    <td>86.0</td>
    <td><b>96.5</b></td>
    <td>67.0</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>2-shot CoT (n=3)</td>
    <td>44.8</td>
    <td>74.9</td>
    <td>26.1</td>
    <td>57.9</td>
    <td>74.2</td>
    <td>85.7</td>
    <td>95.8</td>
    <td>68.8</td>
    <td>59.4</td>
  </tr>
</table>

Table 11: Performance of 2-shot approaches. LLM: Llama3.3-70B. Scale: 0-19. Re-ranker: monoT5 with title-only as query.

Reviewer paper summaries: This paper examines an LLM approach to automating so called “Systematic Literature Reviews” (SLRs) which are a standardized way of investigating the current state-of-the-art in a specific field by following a strict protocol.  This paper examines a novel zero-shot LLM-Guided Abstract Ranker (LGAR) approach which provides graded relevance judgments. A dense ranker then ranks abstracts with the same score. Experiments show that the proposed approach is superior to what is presented as the current state of the art (Akinseloyin et al., 2024), (which is a question answering framework for abstract screening). Experiments are performed on an extended version of the SYNERGY dataset of De Bruin et al., (2023) and the test set of the CLEF Technological Assisted Review 2019 (TAR2019) dataset (Task 2).
Summary of strengths: This work provides extensions to the SYNERGY and CLEF datasets addressing the issue that the full set of criteria and research questions is not consistently spelled out comprehensively in the SLR’s of these datasets. Research questions are inferred from the inclusion and exclusion criteria used in the SRLs. This is a useful data resource for other researchers. Experiments are provided on pretty recent SOTA LLMs, including Llama 3.3 70B. Ablations of the dense reranking component are provided. CoT experiments are also provided. In general, the experimental work is well executed.
Summary of weaknesses: The datasets here are relatively small, with only 26 used for SYNERGY and 31 for CLEF. Most of the reviewers are in the medical domain. For other domains the amount of data here is really almost at the level of an anecdotal experiment. Specifically, there is only 1 math, 2 chemistry and 2 computer science papers being examine here. I somewhat suspect the results might hold pretty well going from medicine to these other domains with reasonable numbers of examples to measure the performance, but without data and experiments on it, one really cannot tell.
Comments, suggestions and typos: It is stated that experiments are run using Llama3.1-8B-Instruct, Qwen2.5-32B-Instruct,5 Qwen2.5-72B-Instruct, and Mistral-Large-Instruct-2411 (123B) using 16-bit quantization. But the key Table 2 is based only on the Llama3.3 70B. Maybe when the assertion about using all these models is made some pointers to where they are used for experiments could be provided.
Overall assessment scores: 3