# Overview of the CLEF 2024 SimpleText Task 1: Retrieve Passages to Include in a Simplified Summary

Eric Sanjuan, Stéphane Huet, Jaap Kamps, Liana Ermakova

<figure><img src="image_1.png" /></figure>

## To cite this version:

Eric Sanjuan, Stéphane Huet, Jaap Kamps, Liana Ermakova. Overview of the CLEF 2024 SimpleText Task 1: Retrieve Passages to Include in a Simplified Summary. Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Sep 2024, Grenoble, France. pp.3115-3128. hal-05181846

HAL Id: hal-05181846

https://hal.science/hal-05181846v1

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Overview of the CLEF 2024 SimpleText Task 1: Retrieve Passages to Include in a Simplified Summary

Éric SanJuan<fnref n="1" />, Stéphane Huet<fnref n="1" />, Jaap Kamps<fnref n="2" /> and Liana Ermakova<fnref n="3" />

**Abstract**

This paper presents an overview of the CLEF 2024 SimpleText Task 1 on *Content Selection*, asking systems to retrieve scientific abstracts in response to a query prompted by a popular science article. Overall, the SimpleText track provides an evaluation platform for the automatic simplification of scientific texts. We discuss the details of the task set-up. First, the SimpleText Corpus with over 4 million academic papers and abstracts. Second, the Topics based on 40 popular science articles in the news and the 114 Queries prompted by them. Third, the Formats of requests and results, the Evaluation labels and Evaluation measures used. Fourth, the Results of the runs submitted by our participants.

**Keywords**

information retrieval, scientific documents, text simplification, scientific information retrieval, non-expert queries, press outlets, query-document relationships (Q-rels), popularized science

## 1. Introduction

This paper presents an overview of the CLEF 2024 SimpleText Task 1 on *Content Selection*, asking systems to retrieve scientific abstracts in response to a query prompted by a popular science article. This task performs a key element of the overall approach of the CLEF 2024 SimpleText Track.

The track as a whole offers valuable data and benchmarks to facilitate discussions on the challenges associated with automatic text simplification. It presents an interconnected framework that encompasses various tasks, providing a comprehensive view of the complexities involved:

**Task 1** on *Content Selection*: retrieve passages to include in a simplified summary.

**Task 2** on *Complexity Spotting*: identify and explain difficult concepts.

**Task 3** on *Text Simplification*: simplify scientific text.

**Task 4** on *SOTA*?: tracking the state-of-the-art in scholarly publications.

This paper presents an overview of the first task in the SimpleText track on automatic simplification of scientific texts following up on the CLEF 2024 SimpleText Workshop. For a comprehensive overview of the other tasks, the task overview papers on Task 2 [1], Task 3 [2], and Task 4 [3], as well as the track overview paper [4], provide detailed information and further insights.

A total of 45 teams registered for our SimpleText track at CLEF 2024. A total of 20 teams submitted 207 runs in total for the Track, of which 11 teams submitted a total of 42 runs for Task 1. The statistics for the Task 1 runs submitted are presented in Table 1. However, some runs had problems that we could not resolve. We do not detail them in the paper as well as the 0-scored runs.

The rest of the paper is organized as follows. Section 2 provides details on the datasets utilized and the evaluation metrics employed in the study. Section 3 offers an overview of the retrieval approaches

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<footnote n="1">¹Avignon Université, LIA, France</footnote>

<footnote n="2">²University of Amsterdam, Amsterdam, The Netherlands</footnote>

<footnote n="3">³Université de Bretagne Occidentale, HCTI, France</footnote>
<table><caption>**Table 1**
CLEF 2024 Simpletext Task 1 official run submission statistics</caption><thead><tr><th>Task</th><th>AIIR Lab</th><th>AMATU</th><th>Arampatzis</th><th>Elsevier</th><th>L3S</th><th>LIA</th><th>PiTheory</th><th>Sharigans</th><th>SINAI</th><th>SONAR</th><th>AB/DPV</th><th>Dajana/Katya</th><th>Frane/Andrea</th><th>Petra/Regina</th><th>Ruby</th><th>Tomislav/Rowan</th><th>UAmsterdam</th><th>UBO</th><th>UniPD</th><th>UZH Pandas</th><th>Total</th></tr></thead><tbody><tr><td>1</td><td>5</td><td>9</td><td>10</td><td>5</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>6</td><td>1</td><td></td><td></td><td></td><td>42</td></tr></tbody></table>

adopted by the participants, specifically focusing on the scientific text. In Section 4, the official submissions' results are presented and discussed. Section 5 conducts a comprehensive analysis of the results, examining various significant aspects. Finally, Section 6 summarizes the findings and outlines potential directions for future research.

## 2. Task 1: Retrieve Passages to Include in a Simplified Summary

This section details *Task 1: Content Selection* on retrieve passages to include in a simplified summary.

### 2.1. Description

Given a popular science article targeted to a general audience, this task aims at retrieving passages, which can help to understand this article, from a large corpus of academic abstracts and bibliographic metadata. Relevant passages should relate to any of the topics in the source article.

### 2.2. Data

We use popular science articles as a source for the types of topics the general public is interested in and as a validation of the reading level that is suitable for them. The main corpus is a large set of scientific abstracts plus associated metadata covering the fields of computer science and engineering. We reuse the collection of academic abstracts from the Citation Network Dataset (12th version released in 2020)<fnref n="1" /> [5]. This collection was extracted from DBLP, ACM, MAG (Microsoft Academic Graph), and other sources. It includes 4,232,520 abstracts in English, published before 2020.

Search requests are based on popular press articles targeted to a general audience, based on *The Guardian* and *Tech Xplore*. Each of these popular science articles represents a general topic that has to be analyzed to retrieve relevant scientific information from the corpus.

We provide the URLs to original articles, the title, and the textual content of each popular science article as a general topic. Each general topic was also enriched with one or more specific keyword queries manually extracted from their content, creating a familiar information retrieval task ranking passages or abstracts in response to a query. Available training data from 2023 includes 29 (train) and 34 (test) queries, with the later set having an extensive recall base due to the large number of submissions in 2023 [6].

In 2024, we added between 2 and 5 new queries (with IDs of the form G*.C*) for each of the 20 articles from the Guardian. These topics were generated by ChatGPT 4, with a prompt asking to list the main subtopics related to computer science; they were manually inspected to check they are linked to the original article and are not redundant. They are longer, containing around ten words and focusing on a specific point related to the article. An example of a keyword query is "system on chip" (T06.1) and an example of a long query is "How AI systems, especially virtual assistants, can perpetuate gender stereotypes?" (G01.C1).

<footnote n="1">¹https://www.aminer.cn/citation</footnote>
The C1 queries were generated based on the following prompt: *In the attached article from the Guardian, list the main sub topics related to computer science and for each topic find at least five related references to scientific publications before 2019 that would have been relevant to be cited in this article. Just provide the references, don't try to get the full text.* We then considered as query the first sub topic. We also considered to use ChatGPT results as a complete run, but few references were returned, many were not indexed in computer science and some did not even exist. That emphasizes the real difficulty of the task of retrieving references to be included in a popular science article.

## 2.3. Baselines

An ElasticSearch index is provided to participants with access through an API. A JSON dump of the index is also available for participants. This index can be accessed online through queries, e.g. https://clef.termwatch.eu/dblp1/_search?q=biases&size=1000 for the query "Biases."

We additionally provided two supplementary baselines leveraging bag-of-words models and sparse vector document representations. The first baseline (denoted by "meili" in the results tables) was generated using the Meilisearch system<fnref n="2" /> and relies on a bucket sort approach. The second baseline (denoted by "boolean") was constructed using a simple boolean model powered by PostgreSQL GIN text indexing.

For each topic, organizers manually assessed each proposed keyword query retrieved by the baseline run powered by Elasticsearch, ensuring that it retrieved at least five relevant documents. As a consequence, this boolean system, which retrieves all abstracts containing all query keywords within the abstract, is expected to artificially achieve high recall levels at a depth of 5. However, this approach suffers from two limitations: it misses relevant abstracts that do not contain all keywords, and it retrieves irrelevant abstracts that happen to contain all query keywords.

In the case of the long C1 queries, we manually extracted the largest subset of terms that retrieved at least five relevant documents. For these queries, the boolean approach is essentially a manual run, which is indicated by an asterisk (*) in the results tables.

Despite their effectiveness, neural models are computationally expensive, requiring significant training data and processing power. Consequently, most participants rely on a hybrid document retrieval approach. This approach leverages a two-stage process:

1. Initial Retrieval: This phase employs a more traditional and less resource-intensive method, such as tf-idf vectorization. This initial retrieval identifies a set of potentially relevant documents.

2. Re-ranking: The documents retrieved in the first stage are then re-ranked using the more nuanced dense representations provided by neural models. This step refines the initial retrieval results based on the semantic understanding of the neural models.

In previous editions, participants relied on the provided Elasticsearch baseline for the initial retrieval phase. To enhance run diversity and address resource limitations, the organizers this year provided access to two vector databases containing pre-computed paragraph embeddings (for titles and abstracts). These vector databases enable to compare the efficiency of scientific document retrieval techniques using asymmetric sparse document retrieval (based on tf-idf) and symmetric dense passage retrieval (based on pre-computed embeddings).

Two embedding vectors were based on the paragraph cross-encoder MS MARCO Mini LM (all-MiniLM-L6-v2)<fnref n="3" />. These embeddings, along with a search API based on them, have been released to participants. Documents are ranked based on the dot product between the query and the abstract (vir仑 abstract) or the title (vir_仑 title) using the pg_vector<fnref n="4" /> PostgreSQL extension and an ivvflat dense vector index (k-means vector clustering with $\sqrt{|D|}$ centroids).

These dense vector and the boolean baselines can be accessed online through a CGI API<fnref n="5" /> with three parameters:

<footnote n="2">²https://www.meilisearch.com/</footnote>

<footnote n="3">³https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2</footnote>

<footnote n="4">⁴https://github.com/pgvector/pgvector</footnote>

<footnote n="5">⁵https://clef.termwatch.eu/stvir_test</footnote>
**corpus**: title, abstract or bool

**phrase**: text passage as query

**length**: number of results to be retrieved

In the case of non boolean query, this API generates the vector embedding of the query on the fly before retrieving results using SQL syntax. For example, for the query "Exploring the use of AI to improve success rates and speed in the pharmaceutical research field", the top 100 documents whose abstracts are most similar to the query (based on dot product) can be retrieved in JSON format using the following syntax:

https://clef.termwatch.eu/stvir\_test?
corpus=abstract\\
\&phrase=Exploring the use of AI to improve success rates and speed
in the pharmaceutical research field\\
\&length=100

In addition to the dot product similarity measure, we also experimented with cosine distance. However, this alternative approach yielded comparable results.

The Boolean and dense vector baselines are provided as a PostgreSQL database containing four tables:

1. Complete Documents (JSON): full documents in JSON format, enabling access to all content.

2. Textual Content (Boolean Search): title and abstracts of documents, facilitating efficient boolean search operations.

3. Title Embeddings: pre-computed dense vector representations (embeddings) of the document titles.

4. Truncated Abstract Embeddings: pre-computed dense vector representations (embeddings) of the first 110 tokens of each document's abstract.

## 2.4. Formats

**Ad-hoc passage retrieval** Participants should retrieve, for each topic and each query, DBLP abstracts related to the query and relevant to be inserted as a citation in the paper associated with the topic. We encourage participants to take into account passage complexity as well as its credibility/influentialness.

**Open passage retrieval (optional)** Participants are encouraged to extract supplementary relevant queries from the titles or content articles and to provide results based on these supplementary queries.

**Output format** Results should be provided in a TREC style JSON format with the following fields:

1. run_id: Run ID starting with <team_id>_<task_id>_<method_used>, e.g. UBO_Task1_TFIDF

2. manual: Whether the run is manual {0,1}

3. topic_id: Topic ID

4. query_id: Query ID used to retrieve the document (if one of the queries provided for the topic was used; 0 otherwise)

5. doc_id: ID of the retrieved document (to be extracted from the JSON output)

6. rel_score: Relevance score of the passage (in the [0-1] scale)

7. comb_score: General score that may combine relevance and other aspects: readability, citation measures...(in the [0-1] scale)

8. passage: Text of the selected passage
For each query, the maximum number of distinct DBLP references (doc_id field) must be 100 and the total length of passages should not exceed 1,000 tokens. The idea of taking into account complexity is to have passages easier to understand for non-experts, while the credibility score aims at guiding them on the expertise of authors and the value of publication w.r.t. the article topic. For example, complexity scores can be evaluated using readability scores and credibility scores using bibliometrics.

Here is an output format example:

```json
[
  {
    "run_id": "UBO_Task1_TFIDF",
    "manual": 0,
    "topic_id": "G01",
    "query_id": "G01.1",
    "doc_id": 1564531496,
    "rel_score": 0.97,
    "comb_score": 0.85,
    "passage": "A CDA is a mobile user device, similar to a Personal Digital Assistant (PDA).
      → It supports the citizen when dealing with public authorities and proves his rights -
      → if desired, even without revealing his identity."
  },
  {
    "run_id": "UBO_Task1_TFIDF",
    "manual": 0,
    "topic_id": "G01",
    "query_id": "G01.1",
    "doc_id": 3000234933,
    "rel_score": 0.9,
    "comb_score": 0.9,
    "passage": "People are becoming increasingly comfortable using Digital Assistants (DAs)
      → to interact with services or connected objects"
  },
  {
    "run_id": "UBO_Task1_TFIDF",
    "manual": 0,
    "topic_id": "G01",
    "query_id": "G01.2",
    "doc_id": 1448624402,
    "rel_score": 0.6,
    "comb_score": 0.3,
    "passage": "As extensive experimental research has shown individuals suffer from diverse
      → biases in decision-making."
  }
]
```

## 2.5. Evaluation

To assess topical relevance, we assigned a 0-2 score to each retrieved document based on its content alignment with the original article. To expand the training data for relevance judgments (qrels), we pooled all documents retrieved at depth 10 from all submitted systems. This approach significantly increased the size of the qrels by 9,990 documents, with a particular focus on newly introduced long queries for the Guardian corpus and T06-T11 queries that previously lacked relevance assessments.

Table 2 summarizes the test collection constructed for the CLEF 2024 SimpleText Task 1, in relation to the earlier years (note that earlier topics have been reused in the “train” data).

While generating the long C1 queries using state-of-the-art LLMs, we were surprised by the inability of these models, specifically ChatGPT4, to find relevant references in the computer science domain suitable for inclusion in large audience tech articles. This raises questions about the inherent difficulty of the task and the potential necessity of combining multiple retrieval systems to improve recall. This need was addressed by both participants and organizers this year.

Many participants employed multiple LLMs, not for initial retrieval, but as rerankers within their systems. Additionally, several participants utilized different implementations of BM25 compared to the one provided by the organizers for the retrieval stage. These novel end-to-end retrieval approaches,
<table><caption>**Table 2**
CLEF 2024 SimpleText Task 1 Test Collection Statistics.</caption><thead><tr><th rowspan="2">Qrels</th><th rowspan="2">Topics</th><th rowspan="2">#Queries</th><th colspan="3">#Assessed abstracts</th><th rowspan="2">#Avg Ass.</th></tr><tr><th>0</th><th>1</th><th>2</th></tr></thead><tbody><tr><td>2022 test</td><td>G1-G20, T2,4,5,10-12,15-16,T18-20</td><td>72</td><td>192</td><td>187</td><td>107</td><td>6.8</td></tr><tr><td>2023 train</td><td>G01-G15</td><td>29</td><td>728</td><td>338</td><td>237</td><td>44.9</td></tr><tr><td>2023 test</td><td>G16-G20, T01-T05</td><td>34</td><td>2260</td><td>357</td><td>1218</td><td>112.8</td></tr><tr><td>2024 train</td><td>G01-G20, T01-T05</td><td>64</td><td>3,675</td><td>768</td><td>1,655</td><td>95.5</td></tr><tr><td>2024 test</td><td>G1.C1-G10.C1, T06-T11</td><td>30</td><td>2,775</td><td>1,500</td><td>579</td><td>128.5</td></tr><tr><td>2024 test extended</td><td>G1-G10, T01-T20</td><td>96</td><td>6,463</td><td>2,491</td><td>1,036</td><td>104.1</td></tr></tbody></table>

coupled with the 4 new baselines provided, resulted in an unexpectedly high number of unassessed documents among the top ten retrieved documents per run. This phenomenon included queries from previous editions. For instance, among queries G01-G10, there were 3, 843 new documents not returned in the top ten of previous editions. Notably, 954 of these documents appeared relevant to at least one existing topic, and 576 were relevant to one of the newly introduced long C1 queries. This confirms the task's inherent difficulty but also demonstrates the potential to achieve high recall levels at depth 10.

In addition to topical relevance, we took into account other key aspects of the track, such as the text complexity and the credibility of the retrieved results. These evaluations were performed using automatic metrics.

## 3. Scientific Passage Retrieval Approaches

In this section, we discuss a range of scientific text retrieval approaches that have been applied by the participants of the track. A total of 11 teams submitted 42 runs in total.

**AB/DPV** Bartulović and Váradi [7] submitted 1 run for Task 1. They used our ElasticSearch API and took into account an FKGL readability score for their combined score.

**Sharingans** Ali et al. [8] also submitted 1 run. They experimented with the ColBERT neural ranker and used GPT 3.5 to select the most informative and concise passages for inclusion in the summary.

**Tomislav/Rowan** Mann and Mikulandric [9] submitted a total of 2 runs. They took the top 100 results retrieved by ElasticSearch. Then, they used cosine similarity on TF-IDF vectors as the relevance score and FKGL score as the combined score.

**Petra/Regina** Elagina and Vučić [10] submitted 1 run, for the first 3 queries only, with the same approach as the previous system.

**AllRLab** Largey et al. [11] submitted a total of 5 runs and proposed several models. First, since input queries are short keyword terms, they used query expansion with LLaMA 3 and reranked the top 5,000 results retrieved by TF-IDF with a bi-encoder or a cross-encoder. Second, they applied LLaMa3 as a pairwise re-ranker. Third, they leveraged ElasticSearch with fine-tuned cross-encoders.

**UBO** Vendeville et al. [12] submitted a total of 1 run. They used PyTerrier<fnref n="6" /> to retrieve documents from TF-IDF scores. Then, the MonoT5 reranker provided by PyTerrier was employed to reorder all extracted documents.

<footnote n="6">⁶https://pyterrier.readthedocs.io/</footnote>
**UAmsterdam** Bakker et al. [13] submitted a total of 6 runs for Task 1. First, they focused on regular information retrieval effectiveness with 2 vanilla baseline runs on an Anserini index, using either BM25 or BM25+RM3, and 2 other runs generated with neural cross-encoder rerankings of these runs by an MS MARCO-trained ranker. Second, 2 further runs filter out the most complex abstracts per request, using the median FKGL readability measure.

**Elsevier** Capari et al. [14] submitted a total of 10 runs. Their approaches mainly centered on creating a ranking model. They started by assessing the performance of several models on a proprietary test collection of scientific papers. Then, the top-performing model was fine-tuned on a large set of unlabeled documents using the Generative Pseudo Labeling approach. They also experimented with generating new search queries.

**LIA** submitted a total of 5 runs as baselines for Task 1. All five have been included in the pool of results for qrel evaluation.

**Ruby** This team (No paper received) submitted a total of 1 run for Task 1. Their approach relies on ElasticSearch and a TF-IDF score.

**Arampatzis** This team (No paper received) submitted a total of 9 runs for Task 1. As these reports are very close, the Tables below only report their evaluation made on their first run.

# 4. Results

This section details the results of the task, for both the train and test data.

## 4.1. Released database

All data and results have been organized within a relational database, which will be released to all active participants. This release will facilitate:

* Computation of Diverse Scores.

* Addressing qrel Issues.

* Easy Generation of Supplementary Runs.

One particular benefit of the relational database is the ability to easily extend the qrels based on dense vector similarity and similarity thresholds. This capability is especially relevant given the observation that seemingly identical abstracts in the DBLP dataset appear with different relevance labels.

## 4.2. Train results

Table 3 shows the results of the CLEF 2023 Simpletext Task 1 using the training qrels provided to participants. These evaluation data encompass 64 queries over 25 topics (G01-G20 and T01-T05). Runs presented in the table are sorted by the main measure of the task, i.e. NDCG@10. Let us note that for this table, we only considered top results retrieved according to relevance scores. The scores of two competitive runs taking into account combined scores are also provided in the two last lines.

## 4.3. Test results

Table 4 still shows relevance evaluation, with a ranking by NDCG@10 on queries absent from the training qrels. This time, we included two separated lines for a single run, when top retrieved documents differ using combined score (comb) or relevance score (rel).

We also inspect how systems behave on two different subsets of queries used in the test. Tables 5 and 6 distinguish the scores on the long G*.C1 queries related to general computer science topics,
<table><caption>**Table 3**
Results for CLEF 2024 SimpleText Task 1 on the Train qrels: G01-G20 and T01-T05.</caption><thead><tr><th rowspan="2">Run</th><th rowspan="2">MRR</th><th colspan="2">Precision</th><th colspan="2">NDCG</th><th rowspan="2">Bpref</th><th rowspan="2">MAP</th></tr><tr><th>10</th><th>20</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>AllRLab_Task1_LLaMABiEncoder</td><td>0.7570</td><td>0.6467</td><td>0.4133</td><td>0.4955</td><td>0.4206</td><td>0.3463</td><td>0.2227</td></tr><tr><td>AllRLab_Task1_LLaMAReranker2</td><td>0.7531</td><td>0.6200</td><td>0.4008</td><td>0.4708</td><td>0.4014</td><td>0.3364</td><td>0.2086</td></tr><tr><td>AllRLab_Task1_LLaMAReranker</td><td>0.7459</td><td>0.6083</td><td>0.4000</td><td>0.4643</td><td>0.3983</td><td>0.3405</td><td>0.2070</td></tr><tr><td>AllRLab_Task1_LLaMACrossEncoder</td><td>0.7849</td><td>0.5150</td><td>0.3675</td><td>0.4117</td><td>0.3732</td><td>0.3413</td><td>0.1985</td></tr><tr><td>LIA_vir_title</td><td>0.6680</td><td>0.4433</td><td>0.2758</td><td>0.3405</td><td>0.2766</td><td>0.2742</td><td>0.1191</td></tr><tr><td>AllRLAB_Task1_CERRF</td><td>0.6329</td><td>0.4033</td><td>0.3083</td><td>0.3246</td><td>0.3030</td><td>0.2113</td><td>0.1375</td></tr><tr><td>Arampatzis_1.GPT2_search_results</td><td>0.5732</td><td>0.3933</td><td>0.1967</td><td>0.2972</td><td>0.2184</td><td>0.0876</td><td>0.0676</td></tr><tr><td>UAmts_Task1_Anserini_rm3</td><td>0.5613</td><td>0.3817</td><td>0.2833</td><td>0.2805</td><td>0.2541</td><td>0.2842</td><td>0.1408</td></tr><tr><td>UAmts_Task1_Anserini_bm25</td><td>0.5493</td><td>0.3733</td><td>0.3208</td><td>0.2803</td><td>0.2827</td><td>0.3003</td><td>0.1536</td></tr><tr><td>Elsevier@SimpleText_task_1_run8</td><td>0.6173</td><td>0.3633</td><td>0.2458</td><td>0.2800</td><td>0.2406</td><td>0.1673</td><td>0.0993</td></tr><tr><td>LIA_vir_abSTRACT</td><td>0.6015</td><td>0.3867</td><td>0.2633</td><td>0.2795</td><td>0.2405</td><td>0.2738</td><td>0.1168</td></tr><tr><td>UAmts_Task1_CE100_CAR</td><td>0.4800</td><td>0.3683</td><td>0.2958</td><td>0.2637</td><td>0.2591</td><td>0.2048</td><td>0.1207</td></tr><tr><td>UAmts_Task1_CE100</td><td>0.4800</td><td>0.3683</td><td>0.2958</td><td>0.2637</td><td>0.2591</td><td>0.2048</td><td>0.1207</td></tr><tr><td>LIA_bool</td><td>0.5646</td><td>0.3517</td><td>0.2400</td><td>0.2552</td><td>0.2238</td><td>0.2134</td><td>0.1037</td></tr><tr><td>UAmts_Task1_CE1K_CAR</td><td>0.4408</td><td>0.3483</td><td>0.2833</td><td>0.2419</td><td>0.2418</td><td>0.2030</td><td>0.1147</td></tr><tr><td>UAmts_Task1_CE1K</td><td>0.4408</td><td>0.3483</td><td>0.2833</td><td>0.2419</td><td>0.2418</td><td>0.2778</td><td>0.1347</td></tr><tr><td>Ruby_Task_1</td><td>0.5231</td><td>0.3050</td><td>0.2425</td><td>0.2387</td><td>0.2281</td><td>0.1696</td><td>0.1018</td></tr><tr><td>Elsevier@SimpleText_task_1_run10</td><td>0.5072</td><td>0.2983</td><td>0.2000</td><td>0.2335</td><td>0.1983</td><td>0.1356</td><td>0.0815</td></tr><tr><td>Elsevier@SimpleText_task_1_run4</td><td>0.5360</td><td>0.3100</td><td>0.2292</td><td>0.2285</td><td>0.2081</td><td>0.1476</td><td>0.0874</td></tr><tr><td>LIA_elastic</td><td>0.4540</td><td>0.2817</td><td>0.2067</td><td>0.2213</td><td>0.1977</td><td>0.2275</td><td>0.1103</td></tr><tr><td>AB/DPV_SimpleText_task1_FKGL</td><td>0.4538</td><td>0.2817</td><td>0.2067</td><td>0.2213</td><td>0.1977</td><td>0.1623</td><td>0.0948</td></tr><tr><td>Elsevier@SimpleText_task_1_run6</td><td>0.4686</td><td>0.2900</td><td>0.2283</td><td>0.2145</td><td>0.2014</td><td>0.1699</td><td>0.0947</td></tr><tr><td>Tomislav/Rowan_SimpleText_T1_1</td><td>0.5023</td><td>0.2683</td><td>0.1933</td><td>0.2108</td><td>0.1910</td><td>0.0972</td><td>0.0650</td></tr><tr><td>Elsevier@SimpleText_task_1_run1</td><td>0.5263</td><td>0.2400</td><td>0.2267</td><td>0.2108</td><td>0.2150</td><td>0.1733</td><td>0.1011</td></tr><tr><td>Elsevier@SimpleText_task_1_run5</td><td>0.4464</td><td>0.2767</td><td>0.2200</td><td>0.2023</td><td>0.1919</td><td>0.1664</td><td>0.0913</td></tr><tr><td>LIA_meili</td><td>0.4372</td><td>0.2883</td><td>0.1792</td><td>0.1833</td><td>0.1570</td><td>0.2024</td><td>0.0691</td></tr><tr><td>Elsevier@SimpleText_task_1_run9</td><td>0.4149</td><td>0.2583</td><td>0.1775</td><td>0.1833</td><td>0.1622</td><td>0.1240</td><td>0.0645</td></tr><tr><td>Elsevier@SimpleText_task_1_run7</td><td>0.3731</td><td>0.2367</td><td>0.1717</td><td>0.1735</td><td>0.1577</td><td>0.1194</td><td>0.0588</td></tr><tr><td>UBO_Task1_TFIDFT5</td><td>0.4134</td><td>0.1933</td><td>0.1775</td><td>0.1621</td><td>0.1625</td><td>0.1647</td><td>0.0730</td></tr><tr><td>Elsevier@SimpleText_task_1_run3</td><td>0.3879</td><td>0.1700</td><td>0.1508</td><td>0.1498</td><td>0.1469</td><td>0.1246</td><td>0.0654</td></tr><tr><td>Elsevier@SimpleText_task_1_run2</td><td>0.3186</td><td>0.1500</td><td>0.1742</td><td>0.1279</td><td>0.1446</td><td>0.1416</td><td>0.0696</td></tr><tr><td>Sharingans_Task1.marco-GPT3</td><td>0.4167</td><td>0.0417</td><td>0.0208</td><td>0.0658</td><td>0.0466</td><td>0.0085</td><td>0.0085</td></tr><tr><td>Tomislav/Rowan_SimpleText_T1_2</td><td>0.0108</td><td>0.0100</td><td>0.0067</td><td>0.0057</td><td>0.0051</td><td>0.0030</td><td>0.0011</td></tr><tr><td>Petra/Regina_results_simpleText_task_1</td><td>0.0013</td><td>0.0000</td><td>0.0025</td><td>0.0000</td><td>0.0018</td><td>0.0016</td><td>0.0004</td></tr><tr><td>UAmts_Task1_CE100_CAR†</td><td>0.6709</td><td>0.4687</td><td>0.3937</td><td>0.4530</td><td>0.3972</td><td>0.3144</td><td>0.1922</td></tr><tr><td>UAmts_Task1_CE1K_CAR†</td><td>0.6403</td><td>0.4219</td><td>0.3672</td><td>0.4032</td><td>0.3646</td><td>0.3411</td><td>0.1904</td></tr></tbody></table>

† Evaluated on comb_score.

which can be a source of public debates (like privacy, quantum computing, bitcoins...) from those established on the short Tech Xplore queries, which are more specific and related with a scientific paper in peer-reviewed venues (indoor positioning system, RISC-V architecture for space computing, underwater WiFi developed using LEDs and lasers...). Rankings on these two subsets are very similar, which shows the consistency of relevance results across queries.
<table><caption>**Table 4**

Results for CLEF 2024 SimpleText Task 1 on the Test qrels (G01.C1-G10.C1 and T06-T11).</caption><thead><tr><th rowspan="2">Run</th><th rowspan="2">MRR</th><th colspan="2">Precision</th><th colspan="2">NDCG</th><th rowspan="2">Bpref</th><th rowspan="2">MAP</th></tr><tr><th>10</th><th>20</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>AllRLab_Task1_LLaMABiEncoder<sup>rel</sup></td><td>0.9444</td><td>0.8167</td><td>0.5517</td><td>0.6311</td><td>0.5240</td><td>0.3559</td><td>0.2304</td></tr><tr><td>AllRLab_Task1_LLaMAReranker2<sup>rel</sup></td><td>0.9300</td><td>0.7933</td><td>0.5417</td><td>0.6092</td><td>0.5082</td><td>0.3495</td><td>0.2177</td></tr><tr><td>AllRLab_Task1_LLaMAReranker2<sup>comb</sup></td><td>0.9361</td><td>0.7833</td><td>0.5450</td><td>0.6063</td><td>0.5107</td><td>0.3494</td><td>0.2192</td></tr><tr><td>AllRLab_Task1_LLaMAReranker<sup>rel</sup></td><td>0.8944</td><td>0.7967</td><td>0.5583</td><td>0.5991</td><td>0.5070</td><td>0.3541</td><td>0.2200</td></tr><tr><td>AllRLab_Task1_LLaMAReranker<sup>comb</sup></td><td>0.9111</td><td>0.7833</td><td>0.5600</td><td>0.5982</td><td>0.5112</td><td>0.3543</td><td>0.2217</td></tr><tr><td>LIA_vir_title</td><td>0.8454</td><td>0.6933</td><td>0.4383</td><td>0.5090</td><td>0.4010</td><td>0.3594</td><td>0.1534</td></tr><tr><td>AllRLab_Task1_LLaMACrossEncoder<sup>rel</sup></td><td>0.7975</td><td>0.6933</td><td>0.5100</td><td>0.4879</td><td>0.4335</td><td>0.3404</td><td>0.1970</td></tr><tr><td>LIA_vir护文</td><td>0.7683</td><td>0.6000</td><td>0.4067</td><td>0.4269</td><td>0.3539</td><td>0.3857</td><td>0.1603</td></tr><tr><td>Uams_Task1_Anserini_rm3</td><td>0.7878</td><td>0.5700</td><td>0.4350</td><td>0.3945</td><td>0.3506</td><td>0.4010</td><td>0.1824</td></tr><tr><td>Uams_Task1_Anserini_bm25</td><td>0.7187</td><td>0.5500</td><td>0.4883</td><td>0.3774</td><td>0.3721</td><td>0.3994</td><td>0.1972</td></tr><tr><td>Uams_Task1_CE1K<sup>comb</sup></td><td>0.5950</td><td>0.5333</td><td>0.4583</td><td>0.3726</td><td>0.3659</td><td>0.4032</td><td>0.1939</td></tr><tr><td>Uams_Task1_CE1K CAR<sup>rel</sup></td><td>0.5950</td><td>0.5333</td><td>0.4583</td><td>0.3726</td><td>0.3659</td><td>0.2701</td><td>0.1605</td></tr><tr><td>Uams_Task1_CE1K<sup>rel</sup></td><td>0.5950</td><td>0.5333</td><td>0.4583</td><td>0.3726</td><td>0.3659</td><td>0.4032</td><td>0.1939</td></tr><tr><td>Uams_Task1_CE100<sup>rel</sup></td><td>0.6618</td><td>0.5300</td><td>0.4567</td><td>0.3705</td><td>0.3579</td><td>0.2657</td><td>0.1579</td></tr><tr><td>Uams_Task1_CE100 CAR<sup>rel</sup></td><td>0.6618</td><td>0.5300</td><td>0.4567</td><td>0.3705</td><td>0.3579</td><td>0.2657</td><td>0.1579</td></tr><tr><td>Uams_Task1_CE100<sup>comb</sup></td><td>0.6618</td><td>0.5300</td><td>0.4567</td><td>0.3705</td><td>0.3579</td><td>0.2657</td><td>0.1579</td></tr><tr><td>Uams_Task1_CE1K CAR<sup>comb</sup></td><td>0.6611</td><td>0.5133</td><td>0.3400</td><td>0.3654</td><td>0.2998</td><td>0.2676</td><td>0.1348</td></tr><tr><td>AllRLAB_Task1_CERRF</td><td>0.7264</td><td>0.5033</td><td>0.4000</td><td>0.3584</td><td>0.3239</td><td>0.2204</td><td>0.1309</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>rel</sup></td><td>0.6986</td><td>0.5100</td><td>0.2550</td><td>0.3522</td><td>0.2465</td><td>0.0742</td><td>0.0577</td></tr><tr><td>UBO_Task1_TFIDFT5</td><td>0.7132</td><td>0.4833</td><td>0.3817</td><td>0.3506</td><td>0.3215</td><td>0.2354</td><td>0.1274</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>comb</sup></td><td>0.6814</td><td>0.5100</td><td>0.2550</td><td>0.3449</td><td>0.2423</td><td>0.0741</td><td>0.0563</td></tr><tr><td>LIA_bool*</td><td>0.7242</td><td>0.5233</td><td>0.3633</td><td>0.3409</td><td>0.2906</td><td>0.2661</td><td>0.1199</td></tr><tr><td>AllRLab_Task1_LLaMACrossEncoder<sup>comb</sup></td><td>0.6609</td><td>0.4867</td><td>0.4100</td><td>0.3363</td><td>0.3281</td><td>0.3364</td><td>0.1579</td></tr><tr><td>AllRLab_Task1_LLaMABiEncoder<sup>comb</sup></td><td>0.6078</td><td>0.4867</td><td>0.3783</td><td>0.3246</td><td>0.3024</td><td>0.3393</td><td>0.1474</td></tr><tr><td>Uams_Task1_CE100 CAR<sup>comb</sup></td><td>0.6420</td><td>0.4700</td><td>0.3433</td><td>0.3236</td><td>0.2790</td><td>0.2657</td><td>0.1321</td></tr><tr><td>Elsevier@SimpleText_task_1_run8</td><td>0.7123</td><td>0.4533</td><td>0.3367</td><td>0.3152</td><td>0.2755</td><td>0.1582</td><td>0.0906</td></tr><tr><td>Elsevier@SimpleText_task_1_run4</td><td>0.6162</td><td>0.4300</td><td>0.3217</td><td>0.3075</td><td>0.2692</td><td>0.1642</td><td>0.1005</td></tr><tr><td>Elsevier@SimpleText_task_1_run10</td><td>0.5117</td><td>0.4067</td><td>0.2767</td><td>0.2949</td><td>0.2401</td><td>0.1236</td><td>0.0729</td></tr><tr><td>LIA_elastic</td><td>0.6173</td><td>0.3733</td><td>0.2900</td><td>0.2818</td><td>0.2442</td><td>0.3016</td><td>0.1325</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>rel</sup></td><td>0.6173</td><td>0.3733</td><td>0.2900</td><td>0.2818</td><td>0.2442</td><td>0.1966</td><td>0.1078</td></tr><tr><td>Ruby_Task_1<sup>rel</sup></td><td>0.5470</td><td>0.4233</td><td>0.3533</td><td>0.2790</td><td>0.2688</td><td>0.1980</td><td>0.1110</td></tr><tr><td>LIA_meili</td><td>0.6386</td><td>0.4700</td><td>0.2867</td><td>0.2736</td><td>0.2242</td><td>0.2377</td><td>0.0833</td></tr><tr><td>Elsevier@SimpleText_task_1_run6</td><td>0.5333</td><td>0.3833</td><td>0.3117</td><td>0.2654</td><td>0.2445</td><td>0.1841</td><td>0.0973</td></tr><tr><td>Ruby_Task_1<sup>comb</sup></td><td>0.5910</td><td>0.3767</td><td>0.3000</td><td>0.2641</td><td>0.2407</td><td>0.1961</td><td>0.0980</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>rel</sup></td><td>0.5444</td><td>0.3733</td><td>0.2750</td><td>0.2477</td><td>0.2201</td><td>0.0963</td><td>0.0601</td></tr><tr><td>Elsevier@SimpleText_task_1_run5</td><td>0.4867</td><td>0.3533</td><td>0.2883</td><td>0.2415</td><td>0.2238</td><td>0.1834</td><td>0.0943</td></tr><tr><td>Elsevier@SimpleText_task_1_run1</td><td>0.5589</td><td>0.3000</td><td>0.3300</td><td>0.2262</td><td>0.2407</td><td>0.1978</td><td>0.1018</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>comb</sup></td><td>0.5309</td><td>0.3200</td><td>0.2333</td><td>0.2044</td><td>0.1790</td><td>0.0938</td><td>0.0509</td></tr><tr><td>Elsevier@SimpleText_task_1_run7</td><td>0.4026</td><td>0.3200</td><td>0.2250</td><td>0.2039</td><td>0.1733</td><td>0.1085</td><td>0.0565</td></tr><tr><td>Elsevier@SimpleText_task_1_run9</td><td>0.3868</td><td>0.3300</td><td>0.2283</td><td>0.1971</td><td>0.1710</td><td>0.1103</td><td>0.0590</td></tr><tr><td>Elsevier@SimpleText_task_1_run3</td><td>0.4733</td><td>0.2367</td><td>0.2033</td><td>0.1872</td><td>0.1712</td><td>0.1587</td><td>0.0714</td></tr><tr><td>Elsevier@SimpleText_task_1_run2</td><td>0.4193</td><td>0.2233</td><td>0.2433</td><td>0.1812</td><td>0.1878</td><td>0.1768</td><td>0.0820</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>comb</sup></td><td>0.4380</td><td>0.2333</td><td>0.2100</td><td>0.1476</td><td>0.1496</td><td>0.1909</td><td>0.0667</td></tr><tr><td>Sharingans_Task1.marco-GPT3</td><td>0.6667</td><td>0.0667</td><td>0.0333</td><td>0.1167</td><td>0.0807</td><td>0.0107</td><td>0.0107</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_2</td><td>0.0217</td><td>0.0233</td><td>0.0150</td><td>0.0156</td><td>0.0124</td><td>0.0062</td><td>0.0025</td></tr><tr><td>Petra&Regina_simpleText_task_1</td><td>0.0026</td><td>0.0000</td><td>0.0050</td><td>0.0000</td><td>0.0035</td><td>0.0031</td><td>0.0007</td></tr></tbody></table>
<table><caption>**Table 5**
Results for CLEF 2024 SimpleText Task 1 on the Test qrels limited to G01.C1-G10.C1.</caption><thead><tr><th rowspan="2">Run</th><th rowspan="2">MRR</th><th colspan="2">Precision</th><th colspan="2">NDCG</th><th rowspan="2">Bpref</th><th rowspan="2">MAP</th></tr><tr><th>10</th><th>20</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>AIIRLab_Task1_LLaMABiEncoder<sup>rel</sup></td><td>0.9500</td><td>0.7600</td><td>0.5125</td><td>0.5546</td><td>0.4777</td><td>0.3150</td><td>0.1919</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker2<sup>rel</sup></td><td>0.9350</td><td>0.7450</td><td>0.5200</td><td>0.5366</td><td>0.4733</td><td>0.3115</td><td>0.1868</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker2<sup>comb</sup></td><td>0.9375</td><td>0.7400</td><td>0.5225</td><td>0.5334</td><td>0.4740</td><td>0.3111</td><td>0.1887</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker<sup>rel</sup></td><td>0.8667</td><td>0.7500</td><td>0.5300</td><td>0.5292</td><td>0.4657</td><td>0.3127</td><td>0.1845</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker<sup>comb</sup></td><td>0.8917</td><td>0.7400</td><td>0.5300</td><td>0.5260</td><td>0.4678</td><td>0.3126</td><td>0.1854</td></tr><tr><td>AIIRLab_Task1_LLaMACrossEncoder<sup>rel</sup></td><td>0.7892</td><td>0.6650</td><td>0.4750</td><td>0.4399</td><td>0.3957</td><td>0.3032</td><td>0.1667</td></tr><tr><td>LIA_vir_title</td><td>0.8014</td><td>0.6100</td><td>0.3750</td><td>0.4043</td><td>0.3307</td><td>0.2793</td><td>0.0985</td></tr><tr><td>LIA_bool*</td><td>0.7613</td><td>0.5800</td><td>0.4175</td><td>0.3531</td><td>0.3194</td><td>0.3384</td><td>0.1452</td></tr><tr><td>AIIRLab_Task1_LLaMABiEncoder<sup>comb</sup></td><td>0.6500</td><td>0.5750</td><td>0.4225</td><td>0.3526</td><td>0.3268</td><td>0.2988</td><td>0.1433</td></tr><tr><td>AIIRLab_Task1_LLaMACrossEncoder<sup>comb</sup></td><td>0.6767</td><td>0.5650</td><td>0.4500</td><td>0.3503</td><td>0.3448</td><td>0.2968</td><td>0.1504</td></tr><tr><td>LIA_meili</td><td>0.7017</td><td>0.6100</td><td>0.3800</td><td>0.3477</td><td>0.2929</td><td>0.3175</td><td>0.1145</td></tr><tr><td>UAmts_Task1_Anserini_rm3</td><td>0.7150</td><td>0.5250</td><td>0.4075</td><td>0.3248</td><td>0.3078</td><td>0.3486</td><td>0.1463</td></tr><tr><td>AIIRLAB_Task1_CERRF</td><td>0.6800</td><td>0.4950</td><td>0.3975</td><td>0.3159</td><td>0.3047</td><td>0.1943</td><td>0.1104</td></tr><tr><td>UAmts_Task1_Anserini_bm25</td><td>0.6364</td><td>0.5050</td><td>0.4600</td><td>0.3060</td><td>0.3269</td><td>0.3549</td><td>0.1651</td></tr><tr><td>LIA_vir_abSTRACT</td><td>0.6774</td><td>0.4900</td><td>0.3025</td><td>0.3053</td><td>0.2537</td><td>0.3020</td><td>0.0906</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>comb</sup></td><td>0.6588</td><td>0.4900</td><td>0.2450</td><td>0.3050</td><td>0.2237</td><td>0.0651</td><td>0.0476</td></tr><tr><td>UAmts_Task1_CE1K_CAR<sup>comb</sup></td><td>0.5583</td><td>0.5000</td><td>0.3300</td><td>0.3042</td><td>0.2630</td><td>0.2297</td><td>0.1091</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>rel</sup></td><td>0.6333</td><td>0.4900</td><td>0.2450</td><td>0.2993</td><td>0.2193</td><td>0.0646</td><td>0.0453</td></tr><tr><td>Elsevier@SimpleText_task_1_run8</td><td>0.6780</td><td>0.4400</td><td>0.2950</td><td>0.2847</td><td>0.2424</td><td>0.1131</td><td>0.0614</td></tr><tr><td>UBO_Task1_TFIDFT5</td><td>0.6198</td><td>0.4500</td><td>0.3425</td><td>0.2774</td><td>0.2610</td><td>0.1911</td><td>0.0903</td></tr><tr><td>UAmts_Task1_CE1K_CAR<sup>rel</sup></td><td>0.4592</td><td>0.4700</td><td>0.4200</td><td>0.2642</td><td>0.2931</td><td>0.2289</td><td>0.1221</td></tr><tr><td>UAmts_Task1_CE1K</td><td>0.4592</td><td>0.4700</td><td>0.4200</td><td>0.2642</td><td>0.2931</td><td>0.3573</td><td>0.1539</td></tr><tr><td>UAmts_Task1_CE100_CAR<sup>comb</sup></td><td>0.5880</td><td>0.4450</td><td>0.3225</td><td>0.2574</td><td>0.2383</td><td>0.2341</td><td>0.1074</td></tr><tr><td>Ruby_Task_1<sup>rel</sup></td><td>0.5550</td><td>0.4100</td><td>0.3600</td><td>0.2546</td><td>0.2587</td><td>0.1677</td><td>0.0966</td></tr><tr><td>UAmts_Task1_CE100<sup>comb</sup></td><td>0.5260</td><td>0.4550</td><td>0.4000</td><td>0.2515</td><td>0.2750</td><td>0.2328</td><td>0.1217</td></tr><tr><td>UAmts_Task1_CE100</td><td>0.5260</td><td>0.4550</td><td>0.4000</td><td>0.2515</td><td>0.2750</td><td>0.2328</td><td>0.1217</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>rel</sup></td><td>0.5550</td><td>0.4000</td><td>0.3200</td><td>0.2467</td><td>0.2380</td><td>0.1125</td><td>0.0675</td></tr><tr><td>Ruby_Task_1<sup>comb</sup></td><td>0.5510</td><td>0.3850</td><td>0.3150</td><td>0.2409</td><td>0.2310</td><td>0.1649</td><td>0.0840</td></tr><tr><td>Elsevier@SimpleText_task_1_run4</td><td>0.5297</td><td>0.3350</td><td>0.2400</td><td>0.2153</td><td>0.1933</td><td>0.0923</td><td>0.0452</td></tr><tr><td>LIA_elastic</td><td>0.5163</td><td>0.3000</td><td>0.2325</td><td>0.2010</td><td>0.1851</td><td>0.2540</td><td>0.0988</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>rel</sup></td><td>0.5163</td><td>0.3000</td><td>0.2325</td><td>0.2010</td><td>0.1851</td><td>0.1589</td><td>0.0762</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>comb</sup></td><td>0.4839</td><td>0.3300</td><td>0.2575</td><td>0.1977</td><td>0.1846</td><td>0.1088</td><td>0.0560</td></tr><tr><td>Elsevier@SimpleText_task_1_run10</td><td>0.3786</td><td>0.3150</td><td>0.2225</td><td>0.1906</td><td>0.1694</td><td>0.0891</td><td>0.0407</td></tr><tr><td>Elsevier@SimpleText_task_1_run6</td><td>0.4898</td><td>0.2950</td><td>0.2325</td><td>0.1790</td><td>0.1712</td><td>0.1343</td><td>0.0539</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>comb</sup></td><td>0.4048</td><td>0.2600</td><td>0.2200</td><td>0.1446</td><td>0.1481</td><td>0.1560</td><td>0.0608</td></tr><tr><td>Elsevier@SimpleText_task_1_run9</td><td>0.2848</td><td>0.2900</td><td>0.1925</td><td>0.1356</td><td>0.1226</td><td>0.0838</td><td>0.0366</td></tr><tr><td>Elsevier@SimpleText_task_1_run5</td><td>0.3961</td><td>0.2400</td><td>0.1950</td><td>0.1330</td><td>0.1359</td><td>0.1294</td><td>0.0482</td></tr><tr><td>Elsevier@SimpleText_task_1_run1</td><td>0.4550</td><td>0.1700</td><td>0.2800</td><td>0.1306</td><td>0.1884</td><td>0.1622</td><td>0.0722</td></tr><tr><td>Elsevier@SimpleText_task_1_run7</td><td>0.3148</td><td>0.2500</td><td>0.1750</td><td>0.1250</td><td>0.1164</td><td>0.0815</td><td>0.0311</td></tr><tr><td>Elsevier@SimpleText_task_1_run3</td><td>0.3725</td><td>0.1200</td><td>0.1325</td><td>0.1022</td><td>0.1098</td><td>0.1306</td><td>0.0435</td></tr><tr><td>Sharingans_Task1.marco-GPT3</td><td>0.5000</td><td>0.0500</td><td>0.0250</td><td>0.0816</td><td>0.0589</td><td>0.0070</td><td>0.0070</td></tr><tr><td>Elsevier@SimpleText_task_1_run2</td><td>0.2995</td><td>0.0800</td><td>0.1600</td><td>0.0679</td><td>0.1091</td><td>0.1209</td><td>0.0384</td></tr></tbody></table>
<table><caption>**Table 6**
Results for CLEF 2024 SimpleText Task 1 on the Test qrels limited to T06-T11.</caption><thead><tr><th rowspan="2">Run</th><th rowspan="2">MRR</th><th colspan="2">Precision</th><th colspan="2">NDCG</th><th rowspan="2">Bpref</th><th rowspan="2">MAP</th></tr><tr><th>10</th><th>20</th><th>10</th><th>20</th></tr></thead><tbody><tr><td>AIIRLab_Task1_LLaMABiEncoder<sup>rel</sup></td><td>0.9500</td><td>0.7600</td><td>0.5125</td><td>0.5546</td><td>0.4777</td><td>0.3150</td><td>0.1919</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker2<sup>rel</sup></td><td>0.9350</td><td>0.7450</td><td>0.5200</td><td>0.5366</td><td>0.4733</td><td>0.3115</td><td>0.1868</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker2<sup>comb</sup></td><td>0.9375</td><td>0.7400</td><td>0.5225</td><td>0.5334</td><td>0.4740</td><td>0.3111</td><td>0.1887</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker<sup>rel</sup></td><td>0.8667</td><td>0.7500</td><td>0.5300</td><td>0.5292</td><td>0.4657</td><td>0.3127</td><td>0.1845</td></tr><tr><td>AIIRLab_Task1_LLaMAReranker<sup>comb</sup></td><td>0.8917</td><td>0.7400</td><td>0.5300</td><td>0.5260</td><td>0.4678</td><td>0.3126</td><td>0.1854</td></tr><tr><td>AIIRLab_Task1_LLaMACrossEncoder<sup>rel</sup></td><td>0.7892</td><td>0.6650</td><td>0.4750</td><td>0.4399</td><td>0.3957</td><td>0.3032</td><td>0.1667</td></tr><tr><td>LIA_vir_title</td><td>0.8014</td><td>0.6100</td><td>0.3750</td><td>0.4043</td><td>0.3307</td><td>0.2793</td><td>0.0985</td></tr><tr><td>LIA_bool</td><td>0.7613</td><td>0.5800</td><td>0.4175</td><td>0.3531</td><td>0.3194</td><td>0.3384</td><td>0.1452</td></tr><tr><td>AIIRLab_Task1_LLaMABiEncoder<sup>comb</sup></td><td>0.6500</td><td>0.5750</td><td>0.4225</td><td>0.3526</td><td>0.3268</td><td>0.2988</td><td>0.1433</td></tr><tr><td>AIIRLab_Task1_LLaMACrossEncoder<sup>comb</sup></td><td>0.6767</td><td>0.5650</td><td>0.4500</td><td>0.3503</td><td>0.3448</td><td>0.2968</td><td>0.1504</td></tr><tr><td>LIA_meili</td><td>0.7017</td><td>0.6100</td><td>0.3800</td><td>0.3477</td><td>0.2929</td><td>0.3175</td><td>0.1145</td></tr><tr><td>UAmts_Task1_Anserini_rm3</td><td>0.7150</td><td>0.5250</td><td>0.4075</td><td>0.3248</td><td>0.3078</td><td>0.3486</td><td>0.1463</td></tr><tr><td>AIIRLAB_Task1_CERRF</td><td>0.6800</td><td>0.4950</td><td>0.3975</td><td>0.3159</td><td>0.3047</td><td>0.1943</td><td>0.1104</td></tr><tr><td>UAmts_Task1_Anserini_bm25</td><td>0.6364</td><td>0.5050</td><td>0.4600</td><td>0.3060</td><td>0.3269</td><td>0.3549</td><td>0.1651</td></tr><tr><td>LIA_vir_abSTRACT</td><td>0.6774</td><td>0.4900</td><td>0.3025</td><td>0.3053</td><td>0.2537</td><td>0.3020</td><td>0.0906</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>comb</sup></td><td>0.6588</td><td>0.4900</td><td>0.2450</td><td>0.3050</td><td>0.2237</td><td>0.0651</td><td>0.0476</td></tr><tr><td>UAmts_Task1_CE1K CAR<sup>comb</sup></td><td>0.5583</td><td>0.5000</td><td>0.3300</td><td>0.3042</td><td>0.2630</td><td>0.2297</td><td>0.1091</td></tr><tr><td>Arampatzis_1.GPT2_search<sup>rel</sup></td><td>0.6333</td><td>0.4900</td><td>0.2450</td><td>0.2993</td><td>0.2193</td><td>0.0646</td><td>0.0453</td></tr><tr><td>Elsevier@SimpleText_task_1_run8</td><td>0.6780</td><td>0.4400</td><td>0.2950</td><td>0.2847</td><td>0.2424</td><td>0.1131</td><td>0.0614</td></tr><tr><td>UBO_Task1_TFIDFT5</td><td>0.6198</td><td>0.4500</td><td>0.3425</td><td>0.2774</td><td>0.2610</td><td>0.1911</td><td>0.0903</td></tr><tr><td>UAmts_Task1_CE1K CAR</td><td>0.4592</td><td>0.4700</td><td>0.4200</td><td>0.2642</td><td>0.2931</td><td>0.2289</td><td>0.1221</td></tr><tr><td>UAmts_Task1_CE1K</td><td>0.4592</td><td>0.4700</td><td>0.4200</td><td>0.2642</td><td>0.2931</td><td>0.3573</td><td>0.1539</td></tr><tr><td>UAmts_Task1_CE100 CAR<sup>comb</sup></td><td>0.5880</td><td>0.4450</td><td>0.3225</td><td>0.2574</td><td>0.2383</td><td>0.2341</td><td>0.1074</td></tr><tr><td>Ruby_Task_1<sup>rel</sup></td><td>0.5550</td><td>0.4100</td><td>0.3600</td><td>0.2546</td><td>0.2587</td><td>0.1677</td><td>0.0966</td></tr><tr><td>UAmts_Task1_CE100 CAR<sup>rel</sup></td><td>0.5260</td><td>0.4550</td><td>0.4000</td><td>0.2515</td><td>0.2750</td><td>0.2328</td><td>0.1217</td></tr><tr><td>UAmts_Task1_CE100</td><td>0.5260</td><td>0.4550</td><td>0.4000</td><td>0.2515</td><td>0.2750</td><td>0.2328</td><td>0.1217</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>rel</sup></td><td>0.5550</td><td>0.4000</td><td>0.3200</td><td>0.2467</td><td>0.2380</td><td>0.1125</td><td>0.0675</td></tr><tr><td>Ruby_Task_1<sup>comb</sup></td><td>0.5510</td><td>0.3850</td><td>0.3150</td><td>0.2409</td><td>0.2310</td><td>0.1649</td><td>0.0840</td></tr><tr><td>Elsevier@SimpleText_task_1_run4</td><td>0.5297</td><td>0.3350</td><td>0.2400</td><td>0.2153</td><td>0.1933</td><td>0.0923</td><td>0.0452</td></tr><tr><td>LIA_elastic</td><td>0.5163</td><td>0.3000</td><td>0.2325</td><td>0.2010</td><td>0.1851</td><td>0.2540</td><td>0.0988</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>rel</sup></td><td>0.5163</td><td>0.3000</td><td>0.2325</td><td>0.2010</td><td>0.1851</td><td>0.1589</td><td>0.0762</td></tr><tr><td>Tomislav/Rowan&Rowan_SimpleText_T1_1<sup>comb</sup></td><td>0.4839</td><td>0.3300</td><td>0.2575</td><td>0.1977</td><td>0.1846</td><td>0.1088</td><td>0.0560</td></tr><tr><td>Elsevier@SimpleText_task_1_run10</td><td>0.3786</td><td>0.3150</td><td>0.2225</td><td>0.1906</td><td>0.1694</td><td>0.0891</td><td>0.0407</td></tr><tr><td>Elsevier@SimpleText_task_1_run6</td><td>0.4898</td><td>0.2950</td><td>0.2325</td><td>0.1790</td><td>0.1712</td><td>0.1343</td><td>0.0539</td></tr><tr><td>AB&DPV_SimpleText_task1_FKGL<sup>comb</sup></td><td>0.4048</td><td>0.2600</td><td>0.2200</td><td>0.1446</td><td>0.1481</td><td>0.1560</td><td>0.0608</td></tr><tr><td>Elsevier@SimpleText_task_1_run9</td><td>0.2848</td><td>0.2900</td><td>0.1925</td><td>0.1356</td><td>0.1226</td><td>0.0838</td><td>0.0366</td></tr><tr><td>Elsevier@SimpleText_task_1_run5</td><td>0.3961</td><td>0.2400</td><td>0.1950</td><td>0.1330</td><td>0.1359</td><td>0.1294</td><td>0.0482</td></tr><tr><td>Elsevier@SimpleText_task_1_run1</td><td>0.4550</td><td>0.1700</td><td>0.2800</td><td>0.1306</td><td>0.1884</td><td>0.1622</td><td>0.0722</td></tr><tr><td>Elsevier@SimpleText_task_1_run7</td><td>0.3148</td><td>0.2500</td><td>0.1750</td><td>0.1250</td><td>0.1164</td><td>0.0815</td><td>0.0311</td></tr><tr><td>Elsevier@SimpleText_task_1_run3</td><td>0.3725</td><td>0.1200</td><td>0.1325</td><td>0.1022</td><td>0.1098</td><td>0.1306</td><td>0.0435</td></tr><tr><td>Sharingans_Task1.marco-GPT3</td><td>0.5000</td><td>0.0500</td><td>0.0250</td><td>0.0816</td><td>0.0589</td><td>0.0070</td><td>0.0070</td></tr><tr><td>Elsevier@SimpleText_task_1_run2</td><td>0.2995</td><td>0.0800</td><td>0.1600</td><td>0.0679</td><td>0.1091</td><td>0.1209</td><td>0.0384</td></tr></tbody></table>
<table><caption>**Table 7**

Evaluation of complexity and credibility for SimpleText Task 1 (over all 176 queries).</caption><thead><tr><th rowspan="2">Run</th><th rowspan="2">Avg #Refs</th><th rowspan="2">Avg size of vocabulary</th><th colspan="2">Ratio of</th><th colspan="2">FKGL</th></tr><tr><th>long words</th><th>complex words</th><th>avg</th><th>median</th></tr></thead><tbody><tr><td>AB/DPV_SimpleText_task1_FKGL<sup>comb</sup></td><td>9.7</td><td>91.6</td><td>0.421</td><td>0.533</td><td>20.4</td><td>19.5</td></tr><tr><td>AB/DPV_SimpleText_task1_FKGL<sup>rel</sup></td><td>9.2</td><td>92.9</td><td>0.384</td><td>0.505</td><td>15.3</td><td>15.1</td></tr><tr><td>AIIRLABherm1_CERRF<sup>comb</sup></td><td>10.6</td><td>96.4</td><td>0.386</td><td>0.503</td><td>15.3</td><td>15.1</td></tr><tr><td>AIIRLab herm1_LLaMABiEncoder<sup>comb</sup></td><td>9.5</td><td>98.1</td><td>0.411</td><td>0.515</td><td>20.7</td><td>20</td></tr><tr><td>AIIRLab herm1_LLaMABiEncoder<sup>rel</sup></td><td>8.7</td><td>95.8</td><td>0.375</td><td>0.485</td><td>15.3</td><td>15.1</td></tr><tr><td>AIIRLab herm1_LLaMACrossEncoder<sup>comb</sup></td><td>10.0</td><td>99.4</td><td>0.411</td><td>0.513</td><td>20.4</td><td>19.7</td></tr><tr><td>AIIRLab herm1_LLaMACrossEncoder<sup>rel</sup></td><td>10.7</td><td>104.3</td><td>0.378</td><td>0.485</td><td>15.5</td><td>15.3</td></tr><tr><td>AIIRLab herm1_LLaMAReranker<sup>comb</sup></td><td>8.8</td><td>96.1</td><td>0.377</td><td>0.487</td><td>15.7</td><td>15.4</td></tr><tr><td>AIIRLab herm1_LLaMAReranker<sup>rel</sup></td><td>8.8</td><td>95.8</td><td>0.376</td><td>0.486</td><td>15.5</td><td>15.2</td></tr><tr><td>AIIRLab herm1_LLaMAReranker<sup>2</sup><sup>comb</sup></td><td>8.6</td><td>93.9</td><td>0.378</td><td>0.489</td><td>15.5</td><td>15.3</td></tr><tr><td>AIIRLab herm1_LLaMAReranker<sup>2</sup><sup>rel</sup></td><td>8.6</td><td>94</td><td>0.376</td><td>0.487</td><td>15.3</td><td>15.1</td></tr><tr><td>Arampatzis_1.GPT2_searchs</td><td>10.5</td><td>91.9</td><td>0.392</td><td>0.511</td><td>15.7</td><td>15.1</td></tr><tr><td>Elsevier@SimpleText_task_1_run1</td><td>10.0</td><td>92.6</td><td>0.385</td><td>0.498</td><td>15.4</td><td>15</td></tr><tr><td>Elsevier@SimpleText_task_1_run10</td><td>10.2</td><td>92.1</td><td>0.38</td><td>0.499</td><td>15.2</td><td>14.8</td></tr><tr><td>Elsevier@SimpleText_task_1_run2</td><td>11.2</td><td>99.1</td><td>0.38</td><td>0.495</td><td>15.2</td><td>15.1</td></tr><tr><td>Elsevier@SimpleText_task_1_run3</td><td>9.7</td><td>90.4</td><td>0.384</td><td>0.494</td><td>15.3</td><td>15</td></tr><tr><td>Elsevier@SimpleText_task_1_run4</td><td>10.7</td><td>99.1</td><td>0.375</td><td>0.495</td><td>15.1</td><td>14.9</td></tr><tr><td>Elsevier@SimpleText_task_1_run5</td><td>11.1</td><td>98</td><td>0.377</td><td>0.492</td><td>15</td><td>14.9</td></tr><tr><td>Elsevier@SimpleText_task_1_run6</td><td>11.2</td><td>96.7</td><td>0.378</td><td>0.492</td><td>15.2</td><td>15</td></tr><tr><td>Elsevier@SimpleText_task_1_run7</td><td>9.8</td><td>100.4</td><td>0.368</td><td>0.492</td><td>14.8</td><td>14.8</td></tr><tr><td>Elsevier@SimpleText_task_1_run8</td><td>10.3</td><td>94.4</td><td>0.387</td><td>0.504</td><td>15.5</td><td>15.3</td></tr><tr><td>Elsevier@SimpleText_task_1_run9</td><td>10.6</td><td>98.2</td><td>0.364</td><td>0.487</td><td>14.7</td><td>14.5</td></tr><tr><td>LIA_bool</td><td>13.0</td><td>139.3</td><td>0.388</td><td>0.513</td><td>20.9</td><td>17</td></tr><tr><td>LIA_elastic</td><td>9.2</td><td>92.9</td><td>0.384</td><td>0.505</td><td>15.3</td><td>15.1</td></tr><tr><td>LIA_meili</td><td>9.6</td><td>115.1</td><td>0.383</td><td>0.502</td><td>17.1</td><td>15.5</td></tr><tr><td>LIA_vir_abSTRACT</td><td>7.2</td><td>69.8</td><td>0.378</td><td>0.484</td><td>14.6</td><td>14.3</td></tr><tr><td>LIA_vir_title</td><td>9.8</td><td>90.4</td><td>0.372</td><td>0.483</td><td>15</td><td>14.7</td></tr><tr><td>Petra/Reginas_simpleText_task1</td><td>5.5</td><td>86.1</td><td>0.386</td><td>0.509</td><td>15.4</td><td>15.3</td></tr><tr><td>Ruby herm1<sup>comb</sup></td><td>9.6</td><td>101.2</td><td>0.36</td><td>0.484</td><td>14</td><td>13.7</td></tr><tr><td>Ruby herm1<sup>rel</sup></td><td>9.7</td><td>92.9</td><td>0.389</td><td>0.503</td><td>15.9</td><td>15.2</td></tr><tr><td>Sharingans herm1_herma<sup>GPT3</sup></td><td>9.8</td><td>59.8</td><td>0.373</td><td>0.436</td><td>15.5</td><td>15.5</td></tr><tr><td>Tomislav/Rowan_simpleText_T1_1<sup>comb</sup></td><td>11.3</td><td>97.6</td><td>0.408</td><td>0.519</td><td>17.2</td><td>16.6</td></tr><tr><td>Tomislav/Rowan_simpleText_T1_1<sup>rel</sup></td><td>9.9</td><td>93.2</td><td>0.391</td><td>0.505</td><td>15.9</td><td>15.4</td></tr><tr><td>Tomislav/Rowan_simpleText_T1_2<sup>comb</sup></td><td>11.8</td><td>97.1</td><td>0.397</td><td>0.501</td><td>16.4</td><td>16.1</td></tr><tr><td>Tomislav/Rowan_simpleText_T1_2<sup>rel</sup></td><td>10.9</td><td>91.1</td><td>0.392</td><td>0.496</td><td>15.8</td><td>15.7</td></tr><tr><td>Uams herm1>Anserini_bm25</td><td>11.8</td><td>111.4</td><td>0.385</td><td>0.506</td><td>16.2</td><td>15.3</td></tr><tr><td>Uams herm1>Anserini.rm3</td><td>11.9</td><td>112.9</td><td>0.387</td><td>0.508</td><td>16.8</td><td>16</td></tr><tr><td>Uams herm1 herm1<sup>CE100</sup><sup>comb</sup></td><td>10.6</td><td>102.5</td><td>0.363</td><td>0.485</td><td>13.5</td><td>13.5</td></tr><tr><td>Uams herm1 herm1<sup>CE100</sup></td><td>11.1</td><td>103.1</td><td>0.388</td><td>0.501</td><td>15.6</td><td>15.3</td></tr><tr><td>Uams herm1 herm1<sup>CE1K</sup><sup>comb</sup></td><td>10.2</td><td>98.5</td><td>0.363</td><td>0.483</td><td>13.8</td><td>13.5</td></tr><tr><td>Uams herm1 herm1<sup>CE1K</sup></td><td>10.8</td><td>101.4</td><td>0.387</td><td>0.499</td><td>15.9</td><td>15.4</td></tr><tr><td>UBO herm1_TFIDFT5</td><td>10.3</td><td>99.2</td><td>0.386</td><td>0.498</td><td>15.4</td><td>15.2</td></tr></tbody></table>

## 5. Analysis

This section provides further analysis of the submitted runs, and the task as whole.

We complement the evaluation above by taking into consideration other aspects essential for Task 1. Table 7 highlights credibility and text complexity. We used simple automatic metrics to provide an overview of the importance and the complexity of the article. First, the average number of bibliographic references among the top 10 results of each query is provided. Second, we provide several metrics provided by the Python library readability<fnref n="7" />: the average size of vocabulary per abstract, the average ratio of words considered as long (i.e., with at least 7 characters), the average ratio of words considered

<footnote n="7">⁷https://pypi.org/project/readability/.</footnote>
as complex (i.e., absent from the Dale-Chall word list of 3,000 words recognized by 80% of fifth graders) and the averaged and median FKGL readability metrics.

A large majority of runs have a similar FKGL of 15, corresponding to university level texts, which can be expected since the document deals with advanced scientific topics. However, AIIRLab runs obtained with bi- or cross-encoders and ordered according to comb scores exhibit a significant higher FKGL readability scores. This difference is related to longer sentences retrieved with this score that with relevance score (average length of 31 words vs 23 words).

Only one run (Sharinganshere Task1, marco-GPT3) provided a rephrased extract from the retrieved abstracts, while other runs gave the abstracts in full. This feature translates in the Table in a lower size of vocabulary in their passages.

# 6. Discussion and Conclusions

This concludes the results for the CLEF 2024 SimpleText Task 1: Content Selection on retrieve passages to include in a simplified summary. Our main findings are the following: First, the Tables on relevance are dominated by neural rankers, in particular, cross-encoders and LLaMA 3 used as a pairwise re-ranker. Second, a majority of participants relied on ElasticSearch search results. If neural models used in processing steps leveraged these results, other IR systems turned out to be competitive. For instance, LIA_vir_title operating with embedding sentences or UAmSTask1_Anserini_r3, using an Anserini index have high relevance evaluations. Third, as expected, ranking over systems differs according to the considered criterion. Runs filtered against readability measures tend to have shorter sentences with a more or less drop in relevance. Remarkably, LLaMA 3 used as a reranker seems to not only help to select more relevant documents but also with more concise sentences.

# Acknowledgments

This track would not have been possible without the great support of numerous individuals. We want to thank in particular the colleagues and the students who participated in data construction and evaluation. Please visit the SimpleText website for more details on the track.<fnref n="8" />

Liana Ermakova is funded by the French National Research Agency (ANR) Automatic Simplification of Scientific Texts project (ANR-22-CE23-0019-01),<fnref n="9" /> and the MaDICS research group.<fnref n="10" />

# References

[1] G. M. Di Nunzio, F. Vezzani, V. Bonato, H. Azarbonyad, J. Kamps, L. Ermakova, Overview of the CLEF 2024 SimpleText Task 2: Identify and Explain Difficult Concepts, in: [15], 2024, pp. 3129–3146.

[2] L. Ermakova, V. Laimé, H. McCombie, J. Kamps, Overview of the CLEF 2024 SimpleText Task 3: Simplify Scientific Text, in: [15], 2024, pp. 3147–3162.

[3] J. D'Souza, S. Kabongo, H. B. Giglou, Y. Zhang, Overview of the CLEF 2024 SimpleText Task 4: SOTA? Tracking the State-of-the-Art in Scholarly Publications, in: [15], 2024, pp. 3163–3173.

[4] L. Ermakova, E. SanJuan, S. Huet, H. Azarbonyad, G. M. Di Nunzio, F. Vezzani, J. D'Souza, J. Kamps, Overview of the CLEF 2024 SimpleText track: Improving access to scientific texts for everyone, in: L. Goeuriot, G. Q. Philippe Mulhem, D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, 2024.

<footnote n="8">⁸https://simpletext-project.com/</footnote>

<footnote n="9">⁹https://anr.fr/Project-ANR-22-CE23-0019</footnote>

<footnote n="10">¹⁰https://www.madics.fr/ateliers/simpletext/</footnote>
[5] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, Z. Su, Arnetminer: Extraction and mining of academic social networks, in: KDD'08, 2008, pp. 990-998.

[6] E. SanJuan, S. Huet, J. Kamps, L. Ermakova, Overview of the CLEF 2023 simpletext task 1: Passage selection for a simplified summary, in: M. Aliannejadi, G. Faggioli, N. Ferro, M. Vlachos (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023, volume 3497 of *CEUR Workshop Proceedings*, CEUR-WS.org, 2023, pp. 2823-2834. URL: https://ceur-ws.org/Vol-3497/paper-238.pdf.

[7] A. Bartulović, D. P. Váradi, University of Split and University of Malta (Team AB&DPV) at the CLEF 2024 SimpleText Track: Scientific Text Made Simpler Through the Use of Artificial Intelligence, in: [15], 2024, pp. 3195-3205.

[8] S. M. Ali, H. Sajid, O. Aijaz, O. Waheed, F. Alvi, A. Samad, Team Sharingans at SimpleText: Fine-tuned LLM based approach to Scientific Text Simplification, in: [15], 2024, pp. 3174-3181.

[9] R. Mann, T. Mikulandric, CLEF 2024 SimpleText Tasks 1-3: Use of Llama-2 for Text Simplification, in: [15], 2024, pp. 3274-3283.

[10] R. Elagina, P. Vučić, AI Contributions to Simplifying Scientific Discourse in SimpleText 2024, in: [15], 2024, pp. 3238-3245.

[11] N. Largey, R. Maarefdoust, S. Durgin, B. Mansouri, AIIR Lab Systems for CLEF 2024 SimpleText: Large Language Models for Text Simplification, in: [15], 2024, pp. 3261-3273.

[12] B. Vendeville, L. Ermakova, P. D. Loor, UBO NLP Report on the SimpleText track at CLEF 2024, in: [15], 2024, pp. 3317-3340.

[13] J. Bakker, G. Yüksel, J. Kamps, University of Amsterdam at the CLEF 2024 SimpleText Track, in: [15], 2024, pp. 3182-3194.

[14] A. Capari, H. Azarbonyad, Z. Afzal, G. Tsatsaronis, Enhancing Scientific Document Simplification through Adaptive Retrieval and Generative Models, in: [15], 2024, pp. 3206-3229.

[15] G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024: Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, CEUR-WS.org, 2024.