# Extended overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance

Philippe Mulhem, Rabab Alkhalifa, Hsuvas Borkakoty, Romain Deveaud, Alaa El-Ebshihy, Luis Espinosa Anke, Tobias Fink, Petra Galuscáková, Gabriela González Sáez, Lorraine Goeuriot, et al.

► To cite this version:

Philippe Mulhem, Rabab Alkhalifa, Hsuvas Borkakoty, Romain Deveaud, Alaa El-Ebshihy, et al. Extended overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. CLEF 2024: Conference and Labs of the Evaluation Forum, September 09-12, 2024, Grenoble, France, Philippe Mulhem; Lorraine Goeuriot; Georges Quénot; Didier Schwab, Sep 2024, Grenoble, France. hal-04738324

HAL Id: hal-04738324

https://hal.science/hal-04738324v1

Submitted on 8 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Extended overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance

## Notebook for the LongEval Lab at CLEF 2024

Rabab Alkhalifa<fnref n="1" />,<fnref n="2" />,†, Hsuvas Borkakoty<fnref n="3" />,†, Romain Deveaud<fnref n="4" />,†, Alaa El-Ebshihy<fnref n="5" />,<fnref n="6" />,†, Luis Espinosa-Anke<fnref n="3" />,<fnref n="7" />,†, Tobias Fink<fnref n="5" />,<fnref n="6" />,†, Petra Galuščáková<fnref n="9" />,†, Gabriela Gonzalez-Saez<fnref n="8" />,†, Lorraine Goeuriot<fnref n="8" />,†, David Iommi<fnref n="5" />,†, Maria Liakata<fnref n="1" />,<fnref n="10" />,<fnref n="11" />,†, Harish Tayyar Madabushi<fnref n="12" />,†, Pablo Medina-Alias<fnref n="12" />,†, Philippe Mulhem<fnref n="8" />,†, Florina Piroi<fnref n="5" />,<fnref n="6" />,†, Martin Popel<fnref n="13" />,† and Arkaitz Zubiaga<fnref n="1" />,†

## Abstract

We describe the second edition of the LongEval CLEF 2024 shared task. This lab evaluates the temporal persistence of Information Retrieval (IR) systems and Text Classifiers. Task 1 requires IR systems to run on corpora acquired at several timestamps, and evaluates the drop in system quality (NDCG) along these timestamps. Task 2 tackles binary sentiment classification at different points in time, and evaluates the performance drop for different temporal gaps. Overall, 37 teams registered for Task 1 and 25 for Task 2. Ultimately, 14 and 4 teams participated in Task 1 and Task 2, respectively.

## Keywords

Evaluation, Temporal Persistence, Temporal Generalisability, Information Retrieval, Text Classification

<figure><img src="image_2.png" /></figure>

<footnote n="1">¹Queen Mary University of London, UK</footnote>

<footnote n="2">²Imam Abdulrahman Bin Faisal University, SA</footnote>

<footnote n="3">³Cardiff University, UK</footnote>

<footnote n="4">⁴Qwant, France</footnote>

<footnote n="5">⁵Research Studios Austria, Data Science Studio, Vienna, AT</footnote>

<footnote n="6">⁶TU Wien, Austria</footnote>

<footnote n="7">⁷AMPLYFI, UK</footnote>

<footnote n="8">⁸Univ. Grenoble Alpes, CNRS, Grenoble INP¹, LIG, Grenoble, France</footnote>

<footnote n="9">⁹University of Stavanger, Stavanger, Norway</footnote>

<footnote n="10">¹⁰Alan Turing Institute, UK</footnote>

<footnote n="11">¹¹University of Warwick, UK</footnote>

<footnote n="12">¹²University of Bath, UK</footnote>

<footnote n="13">¹³Charles University, Prague, Czech Republic</footnote>

<footnote n="1">¹Institute of Engineering Univ. Grenoble Alpes.
CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>† These authors contributed equally.</footnote>

<footnote>Raalkhalifa@iau.edu.sa (R. Alkhalifa); borkakotyh@cardiff.ac.uk (H. Borkakoty); r.deveaud@qwant.com (R. Deveaud); alaa.el-ebshihy@researchstudio.at (A. El-Ebshihy); espinosa-ankel@cardiff.ac.uk (L. Espinosa-Anke); tobias.fink@researchstudio.at (T. Fink); petra.galuscakova@uis.no (P. Galuščáková); gabriela-nicole.gonzalez-saez@univ-grenoble-alpes.fr (G. Gonzalez-Saez); lorraine.goeuriot@univ-grenoble-alpes.fr (L. Goeuriot); david.iommi@researchstudio.at (D. Iommi); m.liakata@qmul.ac.uk (M. Liakata); htm43@bath.ac.uk (H. T. Madabushi); Philippe.Mulhem@imag.fr (P. Mulhem); florina.piroi@researchstudio.at (F. Piroi); popel@ufal.mff.cuni.cz (M. Popel); a.zubiaga@qmul.ac.uk (A. Zubiaga)</footnote>

<footnote>ID 0000-0002-2875-5400 (R. Alkhalifa); 0000-0003-3262-0127 (H. Borkakoty); 0000-0003-2676-7405 (R. Deveaud); 0000-0001-6644-2360 (A. El-Ebshihy); 0000-0001-6830-9176 (L. Espinosa-Anke); 0000-0002-1045-8352 (T. Fink); 0000-0001-6328-7131 (P. Galuščáková); 0000-0003-0878-5263 (G. Gonzalez-Saez); 0000-0001-7491-1980 (L. Goeuriot); 0000-0002-4270-5709 (D. Iommi); 0000-0001-5765-0416 (M. Liakata); 0000-0001-5260-3653 (H. T. Madabushi); 0009-0001-4202-8664 (P. Medina-Alias); 0000-0002-3245-6462 (P. Mulhem); 0000-0001-7584-6439 (F. Piroi); 0000-0002-3628-8419 (M. Popel); 0000-0003-4583-3623 (A. Zubiaga)</footnote>
# 1. Introduction

Outside the strict scientific context, the European Artificial Intelligence Act<fnref n="1" />, adopted by European Commission in 2024, stresses in Article 17, section (d), that providers must comply with “examination, test and validation procedures to be carried out before, during and after the development of the high-risk AI system, and the frequency with which they have to be carried out”. Without focusing here on the degree of risk of Information Retrieval or Classification systems, this Act clearly states that AI systems must tackle evolution. Time is a dimension that is often overlooked when conducting Information Retrieval (IR) experiments, especially when static data sets are utilized. The advantages of such datasets are that they are easily used to evaluate and test systems. Some data sets, like CORD19, contain documents collected at different points in time, showing differences in the set of documents from one collection time to another. Recent research [1] has demonstrated that models trained on data pertaining to a particular time period struggle to keep their performance levels when applied on test data that is distant in time. On the other side, [2] showed that neural systems, especially transformers-based ones, are not always very sensitive to corpus evolution.

With the aim of tackling this challenge of making models have persistent quality over time, the objective of the LongEval lab is twofold: (i) to explore the extent to which temporal differences over time, as reflected in the evolution of evaluation datasets, results in the deterioration of the performance of information retrieval and classification systems, and (ii) to propose improved methods that mitigate performance drop by making models more robust over time.

The LongEval lab [3] took place as part of the Conference and Labs of the Evaluation Forum (CLEF) 2024, and consisted in two separate tasks: (i) Task 1, described in Section 2, focused on information retrieval, and (ii) Task 2, described in Section 3, focused on text classification for sentiment analysis. Both tasks provided labeled datasets enabling analysis and evaluation of models over data evolving in time (what we call “longitudinally evolving data”). In this paper, we add details to [4], by focusing on the datasets statistics, and on analysing in details the overall participant runs and results for each task.

# 2. Task 1 - Retrieval

The retrieval task of LongEval 2024 explores the effect of changes in datasets on retrieval of text documents. More specifically, we focus on a setup in which the datasets are evolving, as in the LongEval 2023 Retrieval Task data [3]. This means, that one dataset can be acquired from another by adding, removing (and replacing) a limited number of documents and queries. The two main scenarios considered focus on one single system or on several ones, as detailed below:

## A single system in an evolving setup

We explore how one selected system behaves when evaluated on several collections, which evolve along the time. The context in which this task taked place is retrieval performances for **Web search**. When considering evolution of Web data along time, we are facing a case when the documents, the queries and also the relevance continuously evolves. We are then studying how Web search engines deal with this situation. The considered scenario is then similar to classical *ad-hoc* search, in the case of evolving data sets. The evaluation in this scenario consider both the Web search case in which the top documents are the most important elements considered, and should take into account the evolving nature of the data. Evaluation should ideally reflect the changes in the collection and especially signal substantial changes that could lead to performance drop. This would allow to re-train the search engine model then and only when it is really necessary, and enable much more efficient overall training.

As described earlier, there is no consensus about the stability of the performance of the neural networks IR systems along time, but it seems to be lower than in the case of statistical models. Moreover, the performance strongly depends on the data used for training the neural model. One

<footnote n="1">¹https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.html</footnote>
objective of the task is to explore the behavior of the neural system in the evolving data scenario.

## Comparison of multiple systems in an evolving setup

While in the first point, we explore a single system, comparison of this systems with multiple systems across evolving collections, should provide more information about systems stability and robustness.

## 2.1. Description of the task

Compared to the LongEval 2023 Dataset [3], in 2024 we take larger lags between the training and the test sets. More precisely, the task is composed of:

* One training set, that contains Web documents, actual user's queries, and assessments, acquired at timestamp *t*;

* Two test sets, acquired later than *t* at time *t'* and *t''*, composed of Web documents and user's queries.

The task datasets were created over sequential time periods, which allows doing observations at different time stamps *t*, and most importantly, comparing the performance across different time stamps *t* and *t'*. So, the IR task aims to assess the performance difference between *t'* and *t''* when *t'* occurs after *t'*, according to the fact that training set acquired at *t*, takes place few months before *t'*.

## 2.2. Dataset

As for LongEval 2023, in 2024 the data for this task were provided by the French search engine Qwant. They consist of the queries issued by the users of this search engine, cleaned Web documents, which were 1) selected to correspond to the queries, and 2) to add additional noise, and relevance judgments, which were created using a click model. The dataset is fully described in [5]. We provided training data, which included 599 train queries, with corresponding 9,785 relevance assessments and 2,049,729 Web pages. All training data were collected during January 2023. The test set corpus is composed of two subsets: Lag6 acquired in June 2023 (i.e., 6 months later than the training set), and Lag8 acquired in August 2024 (i.e. acquired 8 months later than the training set). The test dataset contains 4,321,642 documents (June: 1,790,028; August: 2,531,614) and 1,925 test queries (June: 407; August: 1,518). The datasets are accessible through the lab's webpage<fnref n="2" /> and from the TU Wien Research Data Repository<fnref n="3" />.

The data collected from the Qwant search engine is in French. In a way to help participants, the LongEval data set for the Retrieval task also contains automatic translations into English of both queries and documents. We mention however that the translations provided by LongEval are only applied to the first 500 characters of each sentence of the initial French documents downloaded.

The document and query overlap ratios between the collections is given by Table 1 and Table 2. We see from these tables that there is a substantial overlap between the Train and the Test collection documents and (due to the larger size of the August query set) a substantial overlap between the Train / June queries and the August queries.

<caption>**Table 1**</caption>

Ratio of documents shared between the LongEval 2024 train and test collections, row vs. column, i.e. 0.93 means that 93% of documents in the row collection are also included in the column collection.

<table><thead><tr><th></th><th>Train 2024</th><th>June (Lag6)</th><th>August (Lag8)</th></tr></thead><tbody><tr><th>Train 2024</th><td>1.00</td><td>0.67</td><td>0.93</td></tr><tr><th>June (Lag6)</th><td>0.77</td><td>1.00</td><td>0.97</td></tr><tr><th>August (Lag8)</th><td>0.75</td><td>0.69</td><td>1.00</td></tr></tbody></table>

To evaluate the submissions we use one set of relevance judgments: the judgments acquired by the Qwant click model. For the evaluation, we use the NDCG measure (calculated for each dataset) at 10, as

<footnote n="2">²https://clef-longeval.github.io/</footnote>

<footnote n="3">³https://doi.org/10.48436/xr350-79683</footnote>
**Table 2**

Ratio of the queries shared between the LongEval 2024 train and test collections, rows vs. columns, i.e. 0.99 means that 99% of queries in the row collection are also included in the column collection.

<table><thead><tr><th></th><th>Train 2024</th><th>June (Lag6)</th><th>August (Lag8)</th></tr></thead><tbody><tr><th>Train 2024</th><td>1.00</td><td>0.22</td><td>0.42</td></tr><tr><th>June (Lag6)</th><td>0.32</td><td>1.00</td><td>0.56</td></tr><tr><th>August (Lag8)</th><td>0.17</td><td>0.15</td><td>1.00</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>(a) Lag6 Dataset</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) Lag8 Dataset</figcaption></figure>

**Figure 1:** Overview of the systems using a neural approach (green) vs. other (yellow).

well as the drop between the Lag8 and Lag6 collection. This allows us to check to which extend the IR system face the evolution of the data. We also plan to use manual assessments, acquired through the interface described in section 2.8.

## 2.3. Submissions

14 teams submitted their systems to the Retrieval task. Each team was allowed to submit up to 10 systems. Together, this a overall of 73 runs submitted. Two teams submitted their runs on the wrong test data set, so we do not include their submission results in our further analysis.

## 2.4. Absolute Scores

For the Retrieval task of the LongEval lab, we computed two sets of scores for each of the lags in the test collection, namely NDCG and MAP. Table 3 gives the overview of them for each run on the Lag6 and Lag8 datasets. For each run, the columns of the table indicate which language was used (English, French, or both), whether neural approaches were involved (values yes/no), and whether a single or a combination of several approaches was used (values yes/no). In addition, we show NDCG score histograms for these runs, in decreasing order, for each dataset, showing whether a run uses any neural approach (green for yes, yellow for no) in Figure 1, and whether the run uses a combination of more than a single approach (orange for yes, cyan for no) in Figure 2. This information was acquired from the participants through a questionnaire the participants had to fill for each submitted run. Figure 3 shows which language each made use of.

From Table 3 we see that the systems which did best for the Lag6 data are also among the top for the Lag8, where the first ranked nine systems scores are comparable to each other. For instance, the best system on Lag6, according to the NDCG measure, (dam_run_4), is ranked the second best also on Lag8. Similarity, the best system on Lag8, according to the NDCG measure, (mouse_run_8), is ranked the second best also on Lag6. This finding holds for the MAP measure as well.

Here, we describe the methods used in the top-3 runs, according to the NDCG evaluation measure, for both Lag6 and Lag8 datasets.
<figure><img src="image_5.png" /><figcaption>(a) Lag6 Dataset</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Lag8 Dataset</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(a) Lag6 Dataset</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Lag8 Dataset</figcaption></figure>

<caption>**Figure 3:** Overview of the systems which use French (blue), which use English translations (red), and which use both (purple).</caption>

1. dam_run_4 from the DAM team: This system uses BM25 as a first stage retrieval model, enhanced with proximity search, query expansion via synonyms, and the MBNET model [6], which combines BERT and XLNET, for re-ranking the results.

2. mouse_run_8 from MOUSE team: This system also uses BM25 as a first stage retrieval model, enhanced with an LLM-based re-ranking model using the Cohere API<fnref n="4" />. It utilizes the Llama 3 model [7] for query expansion.

3. mouse_run_10 from MOUSE team: Similar to mouse_run_8, this system uses BM25 as first stage retrieval model, but it is enhanced with a deep neural-based re-ranking model using PyGaggle. It also employs the Llama 3 model for query expansion.

For the Lag8 dataset, the top-3 systems are:

1. mouse_run_9 from MOUSE team: This system uses BM25 as a first stage retrieval model, enhanced with a deep neural-based re-ranking model using PyGaggle⁵. It uses the Mixtral model [8] for query expansion.

2. mouse_run_8 from MOUSE team: Described above.

3. mouse_run_10 from MOUSE team: Described above.

<footnote n="4">⁴https://docs.cohere.com/docs/rerank-2
⁵https://github.com/castorini/pygaggle</footnote>
Generally, most of the solutions chosen by the participants to the LongEval Retrieval task apply a multi-stage retrieval approach. Often, the first stage involves a lexical-based retrieval (e.g., BM25), and query expansion methods like PL2 or BO1. Query expansion is also done by employing Large Language Models, like Mistral or Llama 3. Reranking is done either using neural-based methods or sentence based transformers. Listwise rerankers and fusing have also been used in reranking of retrieved results. Notably, the temporal aspect of the LongEval test collection has been used by some participants to include past query relevance information into query reformulation either from clicklogs or from the documents deemed relevant in the previous

Considering the Figures 1, 2 and 3, we see that the shape of the distribution of the NDCG values are similar for the Lag6 and Lag8 datasets. However, the systems have higher performances on Lag6 than on Lag8, with maximum 0.4 value for the NDCG on the Lag6 versus 0.3 for the Lag8.

<table><caption>Table 3: NDCG and MAP scores for Lag6, Lag8. Results are sorted according to the NDCG scores on the Lag6.</caption><thead><tr><th rowspan="2">Run Id</th><th rowspan="2">Neural</th><th rowspan="2">Comb.</th><th rowspan="2">Language</th><th colspan="2">NDCG</th><th colspan="2">MAP</th></tr><tr><th>Lag6</th><th>Lag8</th><th>Lag6</th><th>Lag8</th></tr></thead><tbody><tr><td>dam_run_4 [9]</td><td>yes</td><td>no</td><td>French</td><td>0.396</td><td>0.294</td><td>0.249</td><td>0.171</td></tr><tr><td>mouse_run_8 [10]</td><td>yes</td><td>yes</td><td>French</td><td>0.395</td><td>0.298</td><td>0.248</td><td>0.174</td></tr><tr><td>mouse_run_10 [10]</td><td>yes</td><td>yes</td><td>French</td><td>0.393</td><td>0.298</td><td>0.246</td><td>0.175</td></tr><tr><td>iris_run_4 [11]</td><td>yes</td><td>yes</td><td>French</td><td>0.392</td><td>0.293</td><td>0.244</td><td>0.171</td></tr><tr><td>mouse_run_9 [10]</td><td>yes</td><td>yes</td><td>French</td><td>0.392</td><td>0.298</td><td>0.245</td><td>0.175</td></tr><tr><td>iris_run_1 [11]</td><td>yes</td><td>yes</td><td>French</td><td>0.392</td><td>0.294</td><td>0.244</td><td>0.171</td></tr><tr><td>iris_run_2 [11]</td><td>yes</td><td>yes</td><td>French</td><td>0.392</td><td>0.293</td><td>0.242</td><td>0.170</td></tr><tr><td>iris_run_3 [11]</td><td>yes</td><td>yes</td><td>French</td><td>0.391</td><td>0.293</td><td>0.243</td><td>0.171</td></tr><tr><td>iris_run_5 [11]</td><td>yes</td><td>yes</td><td>French</td><td>0.390</td><td>0.294</td><td>0.240</td><td>0.171</td></tr><tr><td>mouse_run_7 [10]</td><td>yes</td><td>no</td><td>French</td><td>0.386</td><td>0.288</td><td>0.236</td><td>0.163</td></tr><tr><td>dam_run_3 [9]</td><td>no</td><td>no</td><td>French</td><td>0.385</td><td>0.285</td><td>0.235</td><td>0.162</td></tr><tr><td>quokkas_run_2</td><td>no</td><td>no</td><td>French</td><td>0.379</td><td>0.276</td><td>0.225</td><td>0.150</td></tr><tr><td>quokkas_run_1</td><td>no</td><td>no</td><td>French</td><td>0.374</td><td>0.274</td><td>0.221</td><td>0.148</td></tr><tr><td>lfzzo_run_7</td><td>no</td><td>no</td><td>French</td><td>0.373</td><td>0.269</td><td>0.221</td><td>0.145</td></tr><tr><td>lfzzo_run_7</td><td>no</td><td>no</td><td>French</td><td>0.373</td><td>0.269</td><td>0.221</td><td>0.145</td></tr><tr><td>lfzzo_run_8</td><td>no</td><td>no</td><td>French</td><td>0.372</td><td>0.269</td><td>0.221</td><td>0.144</td></tr><tr><td>lfzzo_run_9</td><td>no</td><td>no</td><td>French</td><td>0.372</td><td>0.268</td><td>0.221</td><td>0.143</td></tr><tr><td>lfzzo_run_10</td><td>no</td><td>no</td><td>French</td><td>0.372</td><td>0.269</td><td>0.219</td><td>0.145</td></tr><tr><td>lfzzo_run_6</td><td>no</td><td>no</td><td>French</td><td>0.371</td><td>0.270</td><td>0.218</td><td>0.145</td></tr><tr><td>dam_run_5 [9]</td><td>yes</td><td>no</td><td>French</td><td>0.370</td><td>0.279</td><td>0.220</td><td>0.156</td></tr><tr><td>mouse_run_6 [10]</td><td>yes</td><td>no</td><td>French</td><td>0.367</td><td>0.286</td><td>0.215</td><td>0.162</td></tr><tr><td>cir_run_3 [12]</td><td>no</td><td>no</td><td>English</td><td>0.354</td><td>0.242</td><td>0.226</td><td>0.136</td></tr><tr><td>snu_run_1 [13]</td><td>yes</td><td>yes</td><td>English</td><td>0.334</td><td>0.251</td><td>0.197</td><td>0.142</td></tr><tr><td>ows_run_1 [13]</td><td>no</td><td>no</td><td>English</td><td>0.333</td><td>0.243</td><td>0.199</td><td>0.139</td></tr><tr><td>kalu_run_2 [14]</td><td>yes</td><td>no</td><td>French</td><td>0.330</td><td>0.254</td><td>0.192</td><td>0.143</td></tr><tr><td>kalu_run_3 [14]</td><td>yes</td><td>no</td><td>French</td><td>0.330</td><td>0.254</td><td>0.192</td><td>0.143</td></tr><tr><td>kalu_run_5 [14]</td><td>yes</td><td>no</td><td>French</td><td>0.324</td><td>0.249</td><td>0.188</td><td>0.140</td></tr><tr><td>kalu_run_4 [14]</td><td>yes</td><td>no</td><td>French</td><td>0.323</td><td>0.250</td><td>0.186</td><td>0.140</td></tr><tr><td>cir_run_4 [12]</td><td>no</td><td>no</td><td>English</td><td>0.320</td><td>0.229</td><td>0.172</td><td>0.117</td></tr><tr><td>wonder_run_3</td><td>no</td><td>no</td><td>French,English</td><td>0.313</td><td>0.235</td><td>0.163</td><td>0.116</td></tr><tr><td>cir_run_2 [12]</td><td>yes</td><td>no</td><td>English</td><td>0.308</td><td>0.230</td><td>0.173</td><td>0.123</td></tr><tr><td>mouse_run_3 [10]</td><td>yes</td><td>yes</td><td>English</td><td>0.306</td><td>0.235</td><td>0.171</td><td>0.126</td></tr><tr><td>ows_run_2 [15]</td><td>no</td><td>no</td><td>English</td><td>0.306</td><td>0.229</td><td>0.197</td><td>0.140</td></tr><tr><td>dam_run_2 [9]</td><td>yes</td><td>no</td><td>English</td><td>0.304</td><td>0.231</td><td>0.169</td><td>0.121</td></tr></tbody></table>
<table><tr><td>mouse_run_4 [10]</td><td>yes</td><td>yes</td><td>English</td><td>0.304</td><td>0.232</td><td>0.167</td><td>0.124</td></tr><tr><td>mouse_run_5 [10]</td><td>yes</td><td>yes</td><td>English</td><td>0.304</td><td>0.232</td><td>0.166</td><td>0.124</td></tr><tr><td>wonder_run_4</td><td>no</td><td>no</td><td>French</td><td>0.299</td><td>0.223</td><td>0.155</td><td>0.107</td></tr><tr><td>kalu_run_1 [14]</td><td>no</td><td>no</td><td>French</td><td>0.298</td><td>0.219</td><td>0.158</td><td>0.107</td></tr><tr><td>galapagos_run_4 [16]</td><td>yes</td><td>yes</td><td>English</td><td>0.295</td><td>0.220</td><td>0.189</td><td>0.131</td></tr><tr><td>ows_run_3 [15]</td><td>yes</td><td>yes</td><td>English</td><td>0.294</td><td>0.224</td><td>0.188</td><td>0.135</td></tr><tr><td>dam_run_1 [9]</td><td>no</td><td>no</td><td>English</td><td>0.294</td><td>0.221</td><td>0.156</td><td>0.112</td></tr><tr><td>galapagos_run_5 [16]</td><td>yes</td><td>yes</td><td>English</td><td>0.293</td><td>0.221</td><td>0.187</td><td>0.132</td></tr><tr><td>mouse_run_2 [10]</td><td>yes</td><td>no</td><td>English</td><td>0.291</td><td>0.225</td><td>0.152</td><td>0.115</td></tr><tr><td>mouse_run_1 [10]</td><td>yes</td><td>no</td><td>English</td><td>0.291</td><td>0.225</td><td>0.153</td><td>0.114</td></tr><tr><td>ows_run_7 [15]</td><td>yes</td><td>yes</td><td>English</td><td>0.290</td><td>0.213</td><td>0.180</td><td>0.123</td></tr><tr><td>cir_run_5 [12]</td><td>no</td><td>no</td><td>English</td><td>0.285</td><td>0.212</td><td>0.148</td><td>0.104</td></tr><tr><td>ows_run_6 [15]</td><td>yes</td><td>yes</td><td>English</td><td>0.284</td><td>0.216</td><td>0.173</td><td>0.126</td></tr><tr><td>cir_run_1 [12]</td><td>no</td><td>no</td><td>English</td><td>0.282</td><td>0.211</td><td>0.145</td><td>0.103</td></tr><tr><td>snu_run_2 [13]</td><td>yes</td><td>yes</td><td>English</td><td>0.282</td><td>0.213</td><td>0.177</td><td>0.127</td></tr><tr><td>lfzzo_run_4</td><td>no</td><td>no</td><td>English</td><td>0.280</td><td>0.209</td><td>0.142</td><td>0.102</td></tr><tr><td>lfzzo_run_2</td><td>no</td><td>no</td><td>English</td><td>0.280</td><td>0.207</td><td>0.142</td><td>0.099</td></tr><tr><td>wonder_run_2</td><td>no</td><td>no</td><td>English</td><td>0.279</td><td>0.207</td><td>0.137</td><td>0.099</td></tr><tr><td>lfzzo_run_3</td><td>no</td><td>no</td><td>English</td><td>0.277</td><td>0.209</td><td>0.139</td><td>0.102</td></tr><tr><td>lfzzo_run_1</td><td>no</td><td>no</td><td>English</td><td>0.276</td><td>0.207</td><td>0.140</td><td>0.100</td></tr><tr><td>lfzzo_run_5</td><td>no</td><td>no</td><td>English</td><td>0.274</td><td>0.207</td><td>0.137</td><td>0.101</td></tr><tr><td>seekx_run_1</td><td>no</td><td>no</td><td>French</td><td>0.274</td><td>0.201</td><td>0.145</td><td>0.095</td></tr><tr><td>seekx_run_2</td><td>no</td><td>no</td><td>French</td><td>0.274</td><td>0.202</td><td>0.144</td><td>0.096</td></tr><tr><td>seekx_run_4</td><td>no</td><td>no</td><td>English</td><td>0.273</td><td>0.202</td><td>0.139</td><td>0.098</td></tr><tr><td>wonder_run_5</td><td>no</td><td>no</td><td>English</td><td>0.273</td><td>0.203</td><td>0.137</td><td>0.098</td></tr><tr><td>wonder_run_1</td><td>no</td><td>no</td><td>English</td><td>0.272</td><td>0.203</td><td>0.136</td><td>0.098</td></tr><tr><td>seekx_run_5</td><td>no</td><td>no</td><td>English</td><td>0.264</td><td>0.193</td><td>0.133</td><td>0.091</td></tr><tr><td>galapagos_run_2 [16]</td><td>yes</td><td>yes</td><td>English</td><td>0.261</td><td>0.198</td><td>0.162</td><td>0.115</td></tr><tr><td>galapagos_run_1 [16]</td><td>yes</td><td>yes</td><td>English</td><td>0.258</td><td>0.196</td><td>0.157</td><td>0.111</td></tr><tr><td>galapagos_run_3 [16]</td><td>yes</td><td>yes</td><td>English</td><td>0.253</td><td>0.192</td><td>0.151</td><td>0.107</td></tr><tr><td>ows_run_4 [15]</td><td>yes</td><td>yes</td><td>English</td><td>0.246</td><td>0.204</td><td>0.128</td><td>0.114</td></tr><tr><td>ows_run_5 [15]</td><td>no</td><td>yes</td><td>English</td><td>0.240</td><td>0.177</td><td>0.124</td><td>0.085</td></tr><tr><td>seekx_run_3</td><td>no</td><td>no</td><td>French</td><td>0.236</td><td>0.174</td><td>0.120</td><td>0.079</td></tr><tr><td>AVERAGE</td><td></td><td></td><td></td><td>0.318</td><td>0.238</td><td>0.183</td><td>0.129</td></tr></table>

## 2.5. Changes in the Scores

The main part of the retrieval task is to study the changes in the performance scores between the collections. The collections were created using the same approach and procedure have a relatively high overlap in terms of both queries and documents (see Tables 1 and 2), we thus provide the Relative NDCG Drop (RND) values of systems between the collections Lag6 and Lag8. RnD(r) for a system r, is defined as as:

$$RND(r) = \frac{\text{NDCG}_{\text{Lag6}}(r) - \text{NDCG}_{\text{Lag8}}(r)}{\text{NDCG}_{\text{Lag6}}(r)}$$

With such definition, small RND values man more robust systems against changes, and large RND values mean that the systems are not able to generalize well between lag6 and lag8. What we see in Table 4 is that the systems which are more robust to the evolution of the test collections (low values on RND) are not the best ones: for instance, ows_run_4 is the more robust system but the third worse one in table 3. The best systems in term of NDCG values in lag6, dam_run_4 and mouse_run_8, have an RND of 0.245, which means that they quite robust, but much less than the most robust ones. This shows
that the very best systems do cope with some extend to the evolution of the corpus, but that their is room for improving best systems against robustness. We also see that the worse robust system against changes, cir_run_3, is a system that does not rely on neural IR models: such finding shows that neural models are also likely to be more robust against changes than non-neural ones.

<table><caption>Table 4: Changes in the NDCG scores. Lines are ordered by descending RND values.</caption><thead><tr><th rowspan="2">System</th><th colspan="2">NDCG</th><th>RND</th></tr><tr><th>Lag6</th><th>Lag8</th><th></th></tr></thead><tbody><tr><td>ows_run_4</td><td>0.246</td><td>0.204</td><td>0.169</td></tr><tr><td>mouse_run_6</td><td>0.367</td><td>0.286</td><td>0.220</td></tr><tr><td>kalu_run_4</td><td>0.323</td><td>0.250</td><td>0.224</td></tr><tr><td>mouse_run_1</td><td>0.291</td><td>0.225</td><td>0.226</td></tr><tr><td>mouse_run_2</td><td>0.291</td><td>0.225</td><td>0.229</td></tr><tr><td>kalu_run_2</td><td>0.330</td><td>0.254</td><td>0.230</td></tr><tr><td>kalu_run_5</td><td>0.324</td><td>0.249</td><td>0.230</td></tr><tr><td>mouse_run_3</td><td>0.306</td><td>0.235</td><td>0.231</td></tr><tr><td>kalu_run_3</td><td>0.330</td><td>0.254</td><td>0.232</td></tr><tr><td>mouse_run_5</td><td>0.304</td><td>0.232</td><td>0.235</td></tr><tr><td>mouse_run_4</td><td>0.304</td><td>0.232</td><td>0.235</td></tr><tr><td>ows_run_6</td><td>0.284</td><td>0.216</td><td>0.238</td></tr><tr><td>galapagos_run_1</td><td>0.258</td><td>0.196</td><td>0.239</td></tr><tr><td>ows_run_3</td><td>0.294</td><td>0.224</td><td>0.239</td></tr><tr><td>mouse_run_9</td><td>0.392</td><td>0.298</td><td>0.240</td></tr><tr><td>galapagos_run_2</td><td>0.261</td><td>0.198</td><td>0.241</td></tr><tr><td>dam_run_2</td><td>0.304</td><td>0.231</td><td>0.241</td></tr><tr><td>mouse_run_10</td><td>0.393</td><td>0.298</td><td>0.243</td></tr><tr><td>galapagos_run_3</td><td>0.253</td><td>0.192</td><td>0.243</td></tr><tr><td>lfzzo_run_3</td><td>0.277</td><td>0.209</td><td>0.243</td></tr><tr><td>snu_run_2</td><td>0.282</td><td>0.213</td><td>0.245</td></tr><tr><td>mouse_run_8</td><td>0.395</td><td>0.298</td><td>0.245</td></tr><tr><td>dam_run_5</td><td>0.370</td><td>0.279</td><td>0.245</td></tr><tr><td>lfzzo_run_5</td><td>0.274</td><td>0.207</td><td>0.245</td></tr><tr><td>wonder_run_3</td><td>0.313</td><td>0.235</td><td>0.247</td></tr><tr><td>iris_run_5</td><td>0.390</td><td>0.294</td><td>0.248</td></tr><tr><td>galapagos_run_5</td><td>0.293</td><td>0.221</td><td>0.248</td></tr><tr><td>dam_run_1</td><td>0.294</td><td>0.221</td><td>0.249</td></tr><tr><td>snu_run_1</td><td>0.334</td><td>0.251</td><td>0.250</td></tr><tr><td>iris_run_3</td><td>0.391</td><td>0.293</td><td>0.251</td></tr><tr><td>lfzzo_run_1</td><td>0.276</td><td>0.207</td><td>0.251</td></tr><tr><td>ows_run_2</td><td>0.306</td><td>0.229</td><td>0.251</td></tr><tr><td>iris_run_2</td><td>0.392</td><td>0.293</td><td>0.251</td></tr><tr><td>iris_run_1</td><td>0.392</td><td>0.294</td><td>0.251</td></tr><tr><td>lfzzo_run_4</td><td>0.280</td><td>0.209</td><td>0.252</td></tr><tr><td>iris_run_4</td><td>0.392</td><td>0.293</td><td>0.252</td></tr><tr><td>cir_run_2</td><td>0.308</td><td>0.230</td><td>0.252</td></tr><tr><td>cir_run_1</td><td>0.282</td><td>0.211</td><td>0.252</td></tr><tr><td>wonder_run_1</td><td>0.272</td><td>0.203</td><td>0.253</td></tr><tr><td>wonder_run_4</td><td>0.299</td><td>0.223</td><td>0.253</td></tr><tr><td>mouse_run_7</td><td>0.386</td><td>0.288</td><td>0.255</td></tr><tr><td>galapagos_run_4</td><td>0.295</td><td>0.220</td><td>0.256</td></tr></tbody></table>
<table><tr><td>wonder_run_5</td><td>0.273</td><td>0.203</td><td>0.257</td></tr><tr><td>cir_run_5</td><td>0.285</td><td>0.212</td><td>0.257</td></tr><tr><td>dam_run_4</td><td>0.396</td><td>0.294</td><td>0.258</td></tr><tr><td>wonder_run_2</td><td>0.279</td><td>0.207</td><td>0.258</td></tr><tr><td>dam_run_3</td><td>0.385</td><td>0.285</td><td>0.258</td></tr><tr><td>seekx_run_4</td><td>0.273</td><td>0.202</td><td>0.260</td></tr><tr><td>ows_run_5</td><td>0.240</td><td>0.177</td><td>0.261</td></tr><tr><td>lfzzo_run_2</td><td>0.280</td><td>0.207</td><td>0.261</td></tr><tr><td>seekx_run_2</td><td>0.274</td><td>0.202</td><td>0.263</td></tr><tr><td>seekx_run_1</td><td>0.274</td><td>0.201</td><td>0.264</td></tr><tr><td>ows_run_7</td><td>0.290</td><td>0.213</td><td>0.264</td></tr><tr><td>seekx_run_3</td><td>0.236</td><td>0.174</td><td>0.265</td></tr><tr><td>kalu_run_1</td><td>0.298</td><td>0.219</td><td>0.265</td></tr><tr><td>seekx_run_5</td><td>0.264</td><td>0.193</td><td>0.267</td></tr><tr><td>quokkas_run_1</td><td>0.374</td><td>0.274</td><td>0.268</td></tr><tr><td>quokkas_run_2</td><td>0.379</td><td>0.276</td><td>0.271</td></tr><tr><td>ows_run_1</td><td>0.333</td><td>0.243</td><td>0.272</td></tr><tr><td>lfzzo_run_6</td><td>0.371</td><td>0.270</td><td>0.273</td></tr><tr><td>lfzzo_run_10</td><td>0.372</td><td>0.269</td><td>0.277</td></tr><tr><td>lfzzo_run_8</td><td>0.372</td><td>0.269</td><td>0.277</td></tr><tr><td>lfzzo_run_7</td><td>0.373</td><td>0.269</td><td>0.280</td></tr><tr><td>lfzzo_run_9</td><td>0.372</td><td>0.268</td><td>0.281</td></tr><tr><td>cir_run_4</td><td>0.320</td><td>0.229</td><td>0.284</td></tr><tr><td>cir_run_3</td><td>0.354</td><td>0.242</td><td>0.316</td></tr><tr><td>AVERAGE</td><td>0.305</td><td>0.228</td><td>0.251</td></tr></table>

## 2.6. Run Rankings

Another point of view studied is how the submitted runs compare to each other, either in terms of the absolute NDCG scores achieved on the collections, or in terms of NDCG changes between the collections. We also calculated the Pearson correlation between the runs (now shown here), with high correlation in terms of NDCG scores, 0.99, and similarly high, 0.98, with respect to ranking order. This corresponds to the relatively high overlaps of the documents and also the queries between Lag6 and Lag8 collections (Table 1 and Table 2). This observation does not hold for the correlation between the ranking according to the NDCG score achieved and the ranking of the performance change, which is relatively low. The Pearson correlation is 0.07 for the Lag6 dataset and -0.05 on the Lag8 dataset.

Last, we calculated a combination of both rankings (ranking in terms of absolute values and ranking in terms of change). For this, we first calculated a Borda count of the ranking in terms of absolute values and Borda count of the ranking in terms of relative change and then we simply summed these two Borda counts: this result is displayed in the last column in the Table 5. We see that in terms of this measure the top performing systems (on Lag6 and Lag8 datasets) are ranked higher, although they have lower rank in terms of the rank of the NDCG change.

<table><caption>Table 5: Ranking of the submitted systems by NDCG scores (columns 2-3), changes in NDCG scores between Lag6 and Lag8 dataset (column 4). Column 4 shows the sum of the Borda count applied to ranking on Lag6 and Lag8 datasets and Borda count of ranking change between Lag8 and Lag6 dataset. The darker color means better performance.</caption><thead><tr><th>System</th><th>NDCG Lag6</th><th>NDCG Lag8</th><th>RND</th><th>Borda</th></tr></thead><tbody><tr><td>dam_run_4</td><td>1</td><td>4</td><td>45</td><td>151</td></tr></tbody></table>
<table><tr><td>mouse_run_8</td><td>2</td><td>2</td><td>22</td><td>175</td></tr><tr><td>mouse_run_10</td><td>3</td><td>3</td><td>18</td><td>177</td></tr><tr><td>iris_run_4</td><td>4</td><td>7</td><td>36</td><td>154</td></tr><tr><td>mouse_run_9</td><td>5</td><td>1</td><td>15</td><td>180</td></tr><tr><td>iris_run_1</td><td>6</td><td>5</td><td>34</td><td>156</td></tr><tr><td>iris_run_2</td><td>7</td><td>8</td><td>33</td><td>153</td></tr><tr><td>iris_run_3</td><td>8</td><td>9</td><td>30</td><td>154</td></tr><tr><td>iris_run_5</td><td>9</td><td>6</td><td>26</td><td>160</td></tr><tr><td>mouse_run_7</td><td>10</td><td>10</td><td>41</td><td>140</td></tr><tr><td>dam_run_3</td><td>11</td><td>12</td><td>47</td><td>131</td></tr><tr><td>quokkas_run_2</td><td>12</td><td>14</td><td>58</td><td>117</td></tr><tr><td>quokkas_run_1</td><td>13</td><td>15</td><td>57</td><td>116</td></tr><tr><td>lfzzo_run_7</td><td>14</td><td>19</td><td>63</td><td>105</td></tr><tr><td>lfzzo_run_8</td><td>15</td><td>17</td><td>62</td><td>107</td></tr><tr><td>lfzzo_run_9</td><td>16</td><td>20</td><td>64</td><td>101</td></tr><tr><td>lfzzo_run_10</td><td>17</td><td>18</td><td>61</td><td>105</td></tr><tr><td>lfzzo_run_6</td><td>18</td><td>16</td><td>60</td><td>107</td></tr><tr><td>dam_run_5</td><td>19</td><td>13</td><td>23</td><td>146</td></tr><tr><td>mouse_run_6</td><td>20</td><td>11</td><td>2</td><td>168</td></tr><tr><td>cir_run_3</td><td>21</td><td>27</td><td>66</td><td>87</td></tr><tr><td>snu_run_1</td><td>22</td><td>23</td><td>29</td><td>127</td></tr><tr><td>ows_run_1</td><td>23</td><td>26</td><td>59</td><td>93</td></tr><tr><td>kalu_run_2</td><td>24</td><td>21</td><td>9</td><td>147</td></tr><tr><td>kalu_run_3</td><td>24</td><td>22</td><td>6</td><td>149</td></tr><tr><td>kalu_run_5</td><td>26</td><td>25</td><td>7</td><td>143</td></tr><tr><td>kalu_run_4</td><td>27</td><td>24</td><td>3</td><td>147</td></tr><tr><td>cir_run_4</td><td>28</td><td>34</td><td>65</td><td>74</td></tr><tr><td>wonder_run_3</td><td>29</td><td>29</td><td>25</td><td>118</td></tr><tr><td>cir_run_2</td><td>30</td><td>33</td><td>37</td><td>101</td></tr><tr><td>mouse_run_3</td><td>31</td><td>28</td><td>8</td><td>134</td></tr><tr><td>ows_run_2</td><td>32</td><td>35</td><td>32</td><td>102</td></tr><tr><td>dam_run_2</td><td>33</td><td>32</td><td>17</td><td>119</td></tr><tr><td>mouse_run_4</td><td>34</td><td>31</td><td>11</td><td>125</td></tr><tr><td>mouse_run_5</td><td>35</td><td>30</td><td>10</td><td>126</td></tr><tr><td>wonder_run_4</td><td>36</td><td>39</td><td>40</td><td>86</td></tr><tr><td>kalu_run_1</td><td>37</td><td>43</td><td>55</td><td>66</td></tr><tr><td>galapagos_run_4</td><td>38</td><td>42</td><td>42</td><td>79</td></tr><tr><td>ows_run_3</td><td>39</td><td>38</td><td>14</td><td>110</td></tr><tr><td>dam_run_1</td><td>40</td><td>41</td><td>28</td><td>92</td></tr><tr><td>galapagos_run_5</td><td>41</td><td>40</td><td>27</td><td>93</td></tr><tr><td>mouse_run_2</td><td>42</td><td>37</td><td>5</td><td>117</td></tr><tr><td>mouse_run_1</td><td>43</td><td>36</td><td>4</td><td>118</td></tr><tr><td>ows_run_7</td><td>44</td><td>45</td><td>53</td><td>59</td></tr><tr><td>cir_run_5</td><td>45</td><td>47</td><td>44</td><td>65</td></tr><tr><td>ows_run_6</td><td>46</td><td>44</td><td>12</td><td>99</td></tr><tr><td>cir_run_1</td><td>47</td><td>48</td><td>38</td><td>68</td></tr><tr><td>snu_run_2</td><td>48</td><td>46</td><td>21</td><td>86</td></tr><tr><td>lfzzo_run_4</td><td>49</td><td>49</td><td>35</td><td>68</td></tr><tr><td>lfzzo_run_2</td><td>50</td><td>54</td><td>50</td><td>47</td></tr><tr><td>wonder_run_2</td><td>51</td><td>52</td><td>46</td><td>52</td></tr><tr><td>lfzzo_run_3</td><td>52</td><td>50</td><td>20</td><td>79</td></tr><tr><td>lfzzo_run_1</td><td>53</td><td>53</td><td>31</td><td>64</td></tr></table>
<table><tr><td>lfzzo_run_5</td><td>54</td><td>51</td><td>24</td><td>72</td></tr><tr><td>seekx_run_1</td><td>55</td><td>60</td><td>52</td><td>34</td></tr><tr><td>seekx_run_2</td><td>56</td><td>59</td><td>51</td><td>35</td></tr><tr><td>seekx_run_4</td><td>57</td><td>58</td><td>48</td><td>38</td></tr><tr><td>wonder_run_5</td><td>58</td><td>57</td><td>43</td><td>43</td></tr><tr><td>wonder_run_1</td><td>59</td><td>56</td><td>39</td><td>47</td></tr><tr><td>seekx_run_5</td><td>60</td><td>63</td><td>56</td><td>22</td></tr><tr><td>galapagos_run_2</td><td>61</td><td>61</td><td>16</td><td>63</td></tr><tr><td>galapagos_run_1</td><td>62</td><td>62</td><td>13</td><td>64</td></tr><tr><td>galapagos_run_3</td><td>63</td><td>64</td><td>19</td><td>55</td></tr><tr><td>ows_run_4</td><td>64</td><td>55</td><td>1</td><td>81</td></tr><tr><td>ows_run_5</td><td>65</td><td>65</td><td>49</td><td>22</td></tr><tr><td>seekx_run_3</td><td>66</td><td>66</td><td>54</td><td>15</td></tr></table>

## 2.7. Queries Overview

We further investigate performance on the provided queries. Due to the space reason, we only investigate a selected subset of queries from each collection. We used a pooling strategy to select these queries to be used for the manual assessment process (described in Section 2.8). We first selected the top five performing runs on the average NDCG performance on both collections. We then calculated the performance of these runs per queries for each collection (i.e. Lag6 and Lag8) and sorted the queries based on their NDCG performance for the five runs. Then, we divided the query set in each collection to four sets and randomly selected from each set: five and 10 queries from Lag 6 and Lag8, respectively. We selected in total 20 queries from Lag6 collection and 40 Lag8 collection. We selected more queries from Lag8 collection since, as shown in Table 2, the number of Lag8 collection is higher than Lag6 collection.

Overview of the scores achieved for the selected queries in each collection is displayed in Figure 4. The figure shows minimum performance (by any submitted run), 25%, quantile, 75% quantile and the maximum achieved NDCG score. Due to a relatively large number of runs, the range of the scores achieved is typically quite large and for some of the queries it even ranges between 0 and 0.8. It can be also noticed that the variation (corresponding to the size of the boxplot) of the query performance for the Lag8 collection is higher than Lag6 collection.

Some of the worst performing queries are very general (“birdsong”, “taxes”, and “used car” for instance) and can thus be expected to be ambiguous. This is in contrast with the top performing queries (e.g. “camping concarneau”, “Prune rabbit”, and “point bordeaux vision”) which refer to more specific information need. Some other top performing queries have high variation in the results, e.g. the query “origami bird” for which it is not specified if the user focuses about about "origami bird" or looks for tutorials to make them.

## 2.8. Manual relevance judgments acquisition

The evaluation results of LongEval IR task presented above rely on automatic assessments generated from click models [5]. In addition to these click-based relevance assessments, we have set up an annotation tool to acquire further relevance assessments by humans. For that, we used the open source annotation tool, Doctag [17], on a sample of the queries selected in section 2.7 (60 queries in total).

Doctag provides a customizable and portable platform specifically designed for Information Retrieval (IR) evaluation. To perform manual relevance judgments using Doctag, annotators utilize its web-based interface. They access the tool and interact with its annotation functionalities, including the assignment of labels to indicate document relevance to specific queries. Annotators view the documents and associate appropriate relevance labels (Fig. 5). The documents to be annotated were selected through pooling the participants runs [18]. For the annotation to remain tractable, we conducted a stratified
<figure><img src="image_9.png" /><figcaption>**Figure 4:** Selected queries performance from Lag6 and Lag8 datasets.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 5:** Screenshot from Doctag main page. Labels annotation is done associating to each document one label that expresses the relevance of that document for that topic.</figcaption></figure>

sampling and selected 60 queries for evaluation (Section 2.7). We set up dedicated online servers where Doctag is deployed, through their use we have acquired over 25K manual assessments. 2900 documents from the original dataset were then assessed. The average number of assessments per query is around 429. To perform the manual annotation and assess document relevance for the corresponding queries, we assigned subsets of the document dataset to a team of 25 annotators. We set up dedicated online servers where Doctag was deployed. Each annotator was assigned to a specific server to perform the annotation tasks. This distributed setup allowed for parallel processing, enabling annotators to work simultaneously and collaborate effectively within their assigned subsets.

We have recorded an aggregate of 25,759 judgments. These judgments span across four distinct categories: 'Relevant', 'Not Relevant', 'Partially Relevant', and 'I Don't Know'.

Preliminary analysis of the data indicates a more balanced approach among annotators in categorizing the query-document pairs. Figure 6 presents the judgment distribution for the top 30 queries in terms of document count. What we observe in Figure 6 is a more evenly distributed number of relevant (green) and non-relevant (red) documents for many queries. While some queries still show a high number of relevant documents (with peaks exceeding 300 relevant documents), the number of non-relevant documents is also significant, indicating no single dominant category. This balanced distribution of relevant and non-relevant documents is much more equitable than previous analyses, where non-relevant judgments predominated.
<figure><img src="image_11.png" /><figcaption>**Figure 6:** The distribution of judgment votes for the top 30 queries based on document count. Resulting counts of 'Relevant' (green), 'Not Relevant' (red), and 'Partially Relevant' (orange) votes are shown.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Figure 7:** Violin plots showing the distribution of judgment counts across different categories for all queries. The plots reveal that the distributions for relevant and not relevant judgments are similar, both with wide ranges and high densities around the median values.</figcaption></figure>
Additionally, Figure 7 provides a detailed view of the distribution of judgment counts across all queries using violin plots. The violin plots reveal that the distributions for relevant and non-relevant judgments are quite similar, with both categories showing a wide range of counts and high densities around the median values. The partially relevant category, while also having a substantial number of judgments, shows a narrower distribution, indicating less variability. The "I don't know" category has a very narrow distribution, reflecting its infrequent use among annotators.

Further evaluation rounds utilizing the collected data are in progress. We will utilize the annotated documents and relevance annotations from the queries to construct an aggregated *Qrel* file. With this *Qrel* file, we will run the evaluation using *trec_eval*<sup>6</sup> on the participants' runs. *Trec_eval* will compare the system's retrieved results against the ground truth relevance judgments defined in the *Qrel* file. This evaluation process will provide valuable insights by comparing the results of the clic model with the manual annotations, thereby assessing the effectiveness and performance of the information retrieval system in relation to the specified queries.

## 2.9. Discussion and conclusion

This task was the second attempt to collectively investigate the impact of the evolution of the data on search system's performances. Having 14 participating teams submitting runs confirmed that this topic was of interest to the community.

The dataset released for this task consisted in a sequence of test collections corresponding to different times. The collections were composed of documents and queries coming from Qwant, and relevance judgment coming from a click model and manual assessment. While the manual assessment is ongoing at the time of the paper's publication, performances of participants' submitted runs were measured using the click logs.

Most of submitted runs rely on multi-stage retrieval approaches. In addition to the usage of Large Language Models in Query expansion. The effect of the translation of the documents and queries provided by the lab has a clear impact: the best results were obtained on the original French data.

Since each subset had substantial overlaps, the correlations between systems rankings was pretty high. As for the robustness of the systems towards dataset changes, we observed that the systems that are the more robust to the evolution of test collection were not the best performing ones.

Further evaluations will be carried out in the near future with the manual assessment of the pooled sets. A thorough analysis of the results will be necessary to study the impact of queries on the results (their nature, topic, difficulty, etc.). Further analysis work will be necessary to fully establish the robustness of the systems and the specific impact of dataset evolution on the performances.

## 3. Task 2 - Classification

Stance detection, an essential task in natural language processing (NLP), involves identifying an author's position or attitude towards a particular topic or statement. This task goes beyond simple sentiment analysis by requiring models to discern not just positive or negative sentiments but also the specific stance (supporting/believer, opposing/denier, or neutral) towards a given target [19, 20].

Comprehending the evolution of social media stances over time poses a significant challenge, a topic that has gained recent interest in the AI and NLP communities but remains relatively unexplored. The performance of social media stance classifiers is intricately linked to temporal shifts in language and evolving societal attitudes toward the subject matter [21].

In LongEval 2024, social media stance detection, a multi-label English classification task, takes center stage, surpassing the complexity of the binary sentiment task in LongEval 2023 [22]. Our primary goal is to assess the persistence of stance detection models in the dynamic landscape of social media posts.

The evolving nature of language and social opinions adds an additional layer of complexity to the challenges faced by text classifiers. Language undergoes continuous changes, reflecting shifts in

<footnote n="6">⁶https://trec.nist.gov/trec_eval/</footnote>
societal norms and opinions and the emergence of novel concepts and words. For instance, consider the
evolution of public opinion on climate change over the past two decades:

*   **Sentence from 2000:** "Global warming is a theory that needs more proof; it's not urgent."

*   **Sentence from 2010:** "Evidence for climate change is mounting, and we need to start taking action."

*   **Sentence from 2020:** "Climate change is an undeniable crisis that requires immediate global action."

The context over two decades in the above example shows that language and urgency surrounding climate change have evolved from skepticism to an accepted crisis. Models not updated with recent discussions and policy changes might fail to accurately capture the critical tone and terminology used in current dialogues about the environment. Similarly, the rapid emergence of new vocabulary, as witnessed with terms like COVID-19 [23], highlights the dynamic nature of language, presenting unique challenges for text classifiers.

**3.1. Description of the task**

To assess the extent of the performance drop of models over shorter and longer temporal gaps, we provided a comprehensive training dataset along with five testing sets. These testing sets include two practice sets and three development sets. The shared competition aimed to stimulate the development of classifiers that can effectively handle temporal variations and maintain performance persistence over different time distances. Participants were expected to submit solutions for two sub-tasks, showcasing their ability to address the challenges of temporal variations in performance. The shared task was in turn divided into two sub-tasks:

**Sub-Task 1: Short-Term Persistence:** In this sub-task, participants were tasked with developing models that demonstrated performance persistence over short periods. Specifically, the models needed to maintain their performance over a temporal gap between the within datasets and the short-term datasets. This involved comparing the performance from the **within-practice** data (January 2010 to December 2010) to the **short-practice** data (January 2014 to December 2014), a time gap of 4 years, and from the **within-dev** data (January 2011 to December 2011) to the **short-dev** data (January 2015 to December 2015), a time gap of 4 years

**Sub-Task 2: Long-Term Persistence:** This sub-task required participants to develop models that maintained performance persistence over a longer period of time. The classifiers were expected to mitigate performance drops over a temporal gap between the within time datasets and the long-term datasets. This involved comparing the performance from the **within-dev** data (January 2011 to December 2011) to the **long-dev** data (January 2018 to September 2019), a time gap of approximately 7 to 8 years.

In addition to the main sub-tasks, participants were also asked to work on models that maintained
performance within the same temporal year of the training set, with the **practice-within** data covering
January 2010 to December 2010 and the **within-dev** data covering January 2011 to December 2011,
with no gap between them and the training set (time gap 0).

**3.2. Dataset**

In this section, we present the process of constructing our final annotated corpus for the task. The large-scale Climate Change Twitter dataset was originally described in [24]. Our primary focus will be on climate change stance, time of the post (created at), and the textual content of the tweets, which we will refer to as the **CC-SD** dataset. This **CC-SD** is large-scale, covering a span of 13 years and containing a diverse set of more than 15 million tweets from various years. Using the BERT model to annotated tweets, the **CC-SD** stance labels fall into three categories: those that express support for
the belief in man-made climate change (believer), those that dispute it (denier), and those that remain neutral on the topic.

The total sum of the categorized tweets over the entire time span are as follows: 11,292,424 tweets as believers, 1,191,386 as deniers, and 3,305,601 as neutral, distributed across the timeline. The annotation is performed using transfer learning with BERT as distant supervision based on another sentiment climate change dataset <fnref n="7" /> and, thus, can be easily manually annotated to improve its precision using human in the loop.

**Data sampling.** The dataset is first downsampled to ensure an equal number of instances for each stance (neutral, denier, believer) within a specified date range, using the minimum stance count across all selected months and years to avoid bias. This involves randomly sampling the same number of rows for each stance, year, and month combination, ensuring balanced representation. The downsampled data is then shuffled and split into training, development, and practice sets, including short- and long-term coverage, with any intersecting IDs between these sets being removed to maintain data integrity and prevent data leakage. Finally, a summary of the downsampled data is generated, detailing the number of rows, date and time of sampling, and statistics per year and month.

**Test set annotation.** We annotate our test data using Prolific<fnref n="8" />, which is a high quality data collection and annotation platform. The forms that contain data to annotate are created using Qualtrics<fnref n="9" />. We run the annotation in several batches, and provide the annotation guideline stating the task details and guidelines for the participants to follow. We add several filters, automatic and manual to select the optimal demographic and qualified annotators. Additionally, a manual annotation is also enforced which contains 5 tweets from the training set, which the organisers first annotate and then using the majority annotation is released as qualification task. The participant have to correctly answer 4 out of 5 questions to access the actual annotation task. We also provide fields in our form for every annotator to give their feedback and to point out if any tweet is inappropriate or contains explicit content in it. We collect responses from 5 annotators for each tweet, and select the majority annotation from the five annotation. In some cases, we find equal agreement among the annotators, and for those cases, we run an extra round of annotation to finalise the agreement. Finally after cleanup and majority annotation finding process, we manually check the data and divide into their respective splits.

The resulting distribution of data is shown in Table 6. Table Dataset statistics summary of training, practice and testing sets.

<table><caption>**Table 6**
Dataset statistics summary of training, practice and testing sets.</caption><thead><tr><th>Dataset</th><th>Time Period</th><th>Size</th></tr></thead><tbody><tr><td>train</td><td>January 2009 to December 2011</td><td>35739</td></tr><tr><td>within-practice</td><td>January 2010 to December 2010</td><td>450</td></tr><tr><td>short-practice</td><td>January 2014 to December 2014</td><td>450</td></tr><tr><td>dev-within</td><td>January 2011 to December 2011</td><td>1074</td></tr><tr><td>dev-short</td><td>January 2015 to December 2015</td><td>1074</td></tr><tr><td>dev-long</td><td>January 2018 to September 2019</td><td>1074</td></tr></tbody></table>

In the Practice phase, participants undertake Pre-Evaluation tasks with datasets from 2010 and 2014, sampled from CC-SD, allowing them to practice within a recent time frame and over a short duration. These datasets are manually verified. Additionally, human-annotated "within time" and "short time" practice sets are provided, also sampled from CC-SD, to refine model development before formal evaluation.

Subsequently, the Evaluation phase assesses models using datasets from 2011, 2015, and the longer period of 2018-2019, all sampled from CC-SD. These datasets undergo manual verification and encompass within-timeframe assessments, short-term predictions, and long-term predictions, offering a

<footnote n="7">⁷https://www.kaggle.com/datasets/edqian/twitter-climate-change-sentiment-dataset</footnote>

<footnote n="8">⁸https://www.prolific.com/</footnote>

<footnote n="9">⁹https://www.qualtrics.com/</footnote>
holistic evaluation of model performance across various temporal contexts. By incorporating datasets covering different years, the evaluation ensures thorough testing and understanding of models' temporal persistence and performance.

## 3.3. Evaluation

Evaluation metrics for this edition of the task remain consistent with the previous version [3, 25]. All submissions were assessed using two key metrics: the **macro-averaged F1-score** on the corresponding sub-task's development set and the **Relative Performance Drop (RPD)**, calculated by comparing performance on "within time" data against results from short- or long-term distant development sets. Submissions for each sub-task were ranked primarily based on the macro-averaged F1-score. Additionally, a unified score, **the weighted-F1**, was computed between the two sub-tasks, encouraging participants to contribute to both for accurate placement on a collective leaderboard and a deeper analysis of their system's performance in various settings.

Participants were expected to design an experimental architecture to enhance a text classifier's temporal performance. In such, the performance of the submissions was evaluated in two ways:

1. **Macro-averaged F1-score:** This metric measured the overall F1-score on the testing set for the sentiment classification sub-task. The F1-score combines precision and recall to provide a balanced measure of model performance. A higher F1-score indicated better performance in terms of both positive and negative sentiment classification.

$$F_{\text{macro}} = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \qquad (1)$$

2. **Relative Performance Drop (RPD):** This metric quantified the difference in performance between the "within-period" data and the short- or long-term distant testing sets. RPD was computed as the difference in performance scores between two sets. A negative RPD value indicated a drop in performance compared to the "within-period" data, while a positive value suggested an improvement.

$$RPD = \frac{f_{\text{score}_{t_j}} - f_{\text{score}_{t_0}}}{f_{\text{score}_{t_0}}} \qquad (2)$$

Where $t_0$ represents performance when the time gap is 0, and $t_j$ represents performance when the time gap is short or long, as introduced in previous work [26].

The submissions were ranked primarily based on the macro-averaged F1-score, emphasizing the overall performance of the stance detection model on the testing sets. The higher the macro-averaged F1-score, the higher the ranking of the submission.

## 3.4. Models

In our study, we evaluated several baseline classifiers to assess their performance and temporal persistence when exposed to evolving data. The models we focused on include **bert-base-uncased**, **roberta-base**, and their respective variations with additional continual incremental pretraining from the climate change corpus.

To address the challenges posed by evolving data, we implemented continual incremental pretraining for both **bert-base-uncased** and **roberta-base** models. These variations, referred to as ++MLM 2019, were further pretrained on a climate change corpus that covers data from the initial training year up to 2019 using masked language modeling. This approach aimed to incorporate recent linguistic trends and contextual information, enhancing the models' ability to adapt to new and evolving data.
The dataset is segmented by years, starting from 2006 to various end years (2011, 2013, 2015, 2017, 2019). For each end year, data from all preceding years up to that point is aggregated and preprocessed. Preprocessing includes filling missing values with the most frequent value in each column, removing rows with missing values in the 'text' or 'stance' columns, and eliminating duplicate entries. Text data is normalized to lowercase, and entries with fewer than six words are excluded. Post-processing, the data is merged into a single dataset for each end year, resulting in five datasets representing different temporal spans. These datasets are subsequently balanced by downsampling to ensure uniform representation for incremental training.

Using a masked language modeling strategy, the textual data without its label is fed into the models incrementally in their chronological order, starting with the 2011 sample and ending with the 2019 sample. This approach ensures a balanced and clean dataset, facilitating robust analysis and model training. Each model was incrementally tested to evaluate its persistence over time, and the best performance was reported in the results section.

* **bert-base-uncased** (Bidirectional Encoder Representations from Transformers) [27] is a foundational model in NLP that introduced the concept of bidirectional training of transformers for language modeling. The bert-base-uncased model is a version of BERT that ignores case sensitivity, which helps in learning case-independent features. It also consists of 12 transformer layers, 768 hidden units, and 12 attention heads. BERT uses a static masked language modeling objective during pretraining, which involves predicting masked words in a sentence based on their context.

* **roberta-base** (Robustly optimized BERT approach) [28] is a variant of the BERT model designed to improve performance by optimizing the pretraining process. It uses dynamic masking, a larger batch size, and more data to enhance the training of transformer-based models. The roberta-base model consists of 12 transformer layers, 768 hidden units, and 12 attention heads. It is pretrained on a diverse range of data to capture rich contextual representations, making it effective for various NLP tasks.

* **++MLM 2019**: A masked language modeling strategy used to adapt a language model to new data by incrementally pretraining with an unlabeled corpus up to 2019. This method leverages recent linguistic trends and contextual updates to improve model adaptation and performance over time.

This systematic approach allowed us to evaluate and enhance the models' temporal persistence and robustness baselines, ensuring they remain effective in the face of evolving language patterns.

## 3.5. Results

This section presents the results obtained during both the practice and evaluation phases of task 2.

## 3.6. Practice phase

In this subsection, we present the results of the practice phase of task 2. This practice dataset was provided to participants to allow them to practice and initiate their text classifiers. Since we did not get any submissions and to understand the initial performance of our practice sets, we compared several baseline classifiers. The models evaluated include **roberta-base**, **bert-base-uncased**, and their respective variations with additional continual incremental pretraining from the climate change corpus from the initial year of training up to 2019 using masked lanague modeling. The results are summarized in Table 7.

As it can be seen from Table 7, the results indicate that the ++MLM 2019 variations of both **roberta-base** and **bert-base-uncased** demonstrate improved f-Within and f-Avg scores compared to their original counterparts. This suggests that additional continual pretraining based on recent data, incrementally over time, contributes to better performance persistence. Notably, **bert-base-uncased++MLM 2019** achieved the lowest RPD, highlighting its resilience to temporal changes.
**Table 7**

Performance of baseline models on practice data. The columns represent: **f-Within** - performance within the same time period, **f-Short** - performance over short temporal gaps, **f-Avg** - average performance across all temporal gaps, and **RPD** - relative performance drop when applied to temporally distant data.

<table><thead><tr><th>Model</th><th>f-Within</th><th>f-Short</th><th>f-Avg</th><th>RPD</th></tr></thead><tbody><tr><td>roberta-base</td><td>0.586</td><td>0.523</td><td>0.555</td><td>-10.80%</td></tr><tr><td>++MLM 2019</td><td>0.612</td><td>0.525</td><td>0.569</td><td>-14.36%</td></tr><tr><td>bert-base-uncased</td><td>0.577</td><td>0.536</td><td>0.557</td><td>-7.19%</td></tr><tr><td>++MLM 2019</td><td>0.586</td><td>0.542</td><td>0.564</td><td>-7.59%</td></tr></tbody></table>

## 3.7. Evaluation phase

In this subsection, we present the results of the evaluation phase of task 2. Using the development dataset provided to participants, we evaluated the final performance of the text classifier models. To understand the performance of our development sets, we compared several baseline classifiers due to the lack of submissions. The models evaluated include **roberta-base**, **bert-base-uncased**, and their respective variations with additional continual incremental pretraining from the climate change corpus up to 2019 using masked language modeling. The results are summarized in Table 8.

**Table 8**

Performance of baseline models on development sets. The columns represent: **f-Within** - performance within the same time period, **f-Short** - performance over short temporal gaps, **f-Long** - performance over long temporal gaps, **f-Avg** - average performance across all temporal gaps, **RPD-Short** - relative performance drop over short temporal gaps, **RPD-Long** - relative performance drop over long temporal gaps, and **RPD-Avg** - average relative performance drop.

<table><thead><tr><th>Model</th><th>f-Within</th><th>f-Short</th><th>f-Long</th><th>f-Avg</th><th>RPD-Short</th><th>RPD-Long</th><th>RPD-Avg</th></tr></thead><tbody><tr><td>roberta-base</td><td>0.626</td><td>0.558</td><td>0.529</td><td>0.571</td><td>-10.81%</td><td>-15.46%</td><td>-26.26%</td></tr><tr><td>++MLM 2019</td><td>0.623</td><td>0.594</td><td>0.552</td><td>0.590</td><td>-4.74%</td><td>-11.46%</td><td>-16.20%</td></tr><tr><td>bert-base-uncased</td><td>0.614</td><td>0.569</td><td>0.536</td><td>0.573</td><td>-7.26%</td><td>-12.64%</td><td>-19.89%</td></tr><tr><td>++MLM 2019</td><td>0.600</td><td>0.571</td><td>0.540</td><td>0.570</td><td>-4.94%</td><td>-10.01%</td><td>-14.94%</td></tr></tbody></table>

As shown in Table 8, the ++MLM 2019 variations of both **roberta-base** and **bert-base-uncased** models exhibit notable improvements in the **f-Short** and **f-Long** scores, as well as reduced **RPD** values compared to their standard counterparts. The ++MLM 2019 variation of **roberta-base** achieved an f-Avg score of (0.590), an improvement over the original model's score of (0.571). It also showed a significantly lower RPD-Short of (-4.74%) and RPD-Long of (-11.46%), indicating better resilience to temporal changes over both short and long gaps. Similarly, the ++MLM 2019 variation of **bert-base-uncased** achieved an f-Avg score of (0.570), slightly lower than the original model's 0.573. However, it exhibited a lower **RPD-Long** of (-10.01%) and **RPD-Avg** of (-14.94%), demonstrating improved performance persistence over time.

These results reinforce the value of continual incremental pretraining with recent data to maintain and improve model performance in dynamic environments. The ++MLM 2019 variations consistently showed enhanced performance metrics and reduced performance degradation over time, validating the effectiveness of this approach in enhancing temporal persistence.

## 3.8. Discussion and conclusion

This section discusses the results of our study on temporally adaptive classification methods, highlighting the significance of incorporating temporal information into text classification models to mitigate performance drops over time and the use of an outdated language model. These results reveal that classifiers trained on older data exhibit significant performance drops when applied to newer data. This is evident from the relative performance drops (RPD) reported, where the ++MLM 2019 variations
showed a marked improvement in mitigating this drop.

Previous work by Alkhalifa et al. [26] introduced the *Incremental Temporal Alignment (ITA)* method as a superior approach for enhancing temporal persistence of static word embedding. This method aligns closely with the continual incremental pretraining approach evaluated in our results, where **++MLM 2019** variations of both **roberta-base** and **bert-base-uncased** demonstrated improved **f-Within**, **f-Avg** scores, and lower **RPD** values. The *ITA* method's emphasis on leveraging incremental updates to word embeddings aligns with the improvements seen in the **++MLM 2019** models, showcasing their resilience to evolving data and enhancing their persistence as text classifiers as context updated overtime.

The results reinforce several best practices for designing temporally robust and persistent text classifiers. Methods relying on incremental updates generally outperform static embeddings, as corroborated by the superior performance of the **++MLM 2019** models. Additionally, it is crucial to select robust baseline models and incrementally update them to accommodate evolving language patterns over time.

The practical implications of our findings are significant for real-world NLP applications. In dynamic environments such as stance posts on social media, language evolves rapidly, making temporal adaptation through an incremental pretraining approach substantially enhance the longevity and persistence of text classifiers. These results provide empirical evidence supporting the implementation of temporally adaptive classification methods in real-world scenarios.

## Acknowledgments

This work is supported by the ANR Kodicare bi-lateral project, grant ANR-19-CE23-0029 of the French Agence Nationale de la Recherche, and by the Austrian Science Fund (FWF, grant I4471-N). This work is also supported by a UKRI/EPSRC Turing AI Fellowship to Maria Liakata (grant no. EP/V030302/1). This work has been using services provided by the LINDAT/CLARIAH-CZ Research Infrastructure (https://lindat.cz), supported by the Ministry of Education, Youth and Sports of the Czech Republic (Project No. LM2023062) and has been also supported by the Ministry of Education, Youth and Sports of the Czech Republic, Project No. LM2023062 LINDAT/CLARIAH-CZ.

## References

[1] R. Gangi Reddy, B. Iyer, M. A. Sultan, R. Zhang, A. Sil, V. Castelli, R. Florian, S. Roukos, Synthetic target domain supervision for open retrieval qa, in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, Association for Computing Machinery, New York, NY, USA, 2021, p. 1793-1797. URL: https://doi.org/10.1145/3404835.3463085. doi:10.1145/3404835.3463085.

[2] J. Lovón-Melgarejo, L. Soulier, K. Pinel-Sauvagnat, L. Tamine, Studying catastrophic forgetting in neural ranking models, Springer-Verlag, Berlin, Heidelberg, 2021, p. 375-390. URL: https://doi.org/10.1007/978-3-030-72113-8_25. doi:10.1007/978-3-030-72113-8_25.

[3] R. Alkhalifa, I. Bilal, H. Borkakoty, J. Camacho-Collados, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, G. Gonzalez-Saez, P. Galuščáková, L. Goeuriot, E. Kochkina, M. Liakata, D. Loureiro, H. T. Madabushi, P. Mulhem, F. Piroi, M. Popel, C. Servan, A. Zubiaga, Overview of the clef-2023 longeval lab on longitudinal evaluation of model performance, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the CLEF Association (CLEF 2023), Lecture Notes in Computer Science (LNCS), Springer, Thessaliniki, Greece, 2023.

[4] R. Alkhalifa, H. Borkakoty, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, T. Fink, P. Galuščáková, G. Gonzalez-Saez, L. Goeuriot, D. Iommi, M. Liakata, H. T. Madabushi, P. Medina-Alias, P. Mulhem, F. Piroi, M. Popel, A. Zubiaga, Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. D. Nunzio, P. Galuščáková, A. G. S. de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International
Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science (LNCS),
Springer, Heidelberg, Germany, 2024.

[5] P. Galuščáková, R. Deveaud, G. Gonzalez-Saez, P. Mulhem, L. Goeuriot, F. Piroi, M. Popel, Longeval-retrieval: French-english dynamic test collection for continuous web search evaluation, 2023. arXiv:2303.03229.

[6] K. Song, X. Tan, T. Qin, J. Lu, T.-Y. Liu, Mpnet: Masked and permuted pre-training for language understanding, Advances in neural information processing systems 33 (2020) 16857-16867.

[7] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023).

[8] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, F. Bressand, et al., Mixtral of experts, arXiv preprint arXiv:2401.04088 (2024).

[9] A. Basaglia, A. Stocco, M. Popović, N. Ferro, Seupd@clef: Team dam on reranking using sentence embedders, in: [29], 2024.

[10] L. Cazzador, F. L. D. Faveri, F. Franceschini, L. Pamio, S. Piron, N. Ferro, Seupd@clef: Team mouse on enhancing search engines effectiveness with large language models, in: [29], 2024.

[11] F. Galli, M. Rigobello, M. Schibuola, R. Zuech, N. Ferro, Seupd@clef: Team iris on temporal evolution of query expansion and rank fusion techniques applied to cross-encoder re-rankers, in: [29], 2024.

[12] J. Keller, T. Breuer, P. Schaer, Leveraging prior relevance signals in web search, in: [29], 2024.

[13] S. Yoon, J. Kim, S. won Hwang, Analyzing the effectiveness of listwise reranking with positional invariance on temporal generalizability, in: [29], 2024.

[14] A. Kimia, A. Akan, F. Arwa, N. Ferro, Seupd@clef: Team kalu on improving search engine performance with query expansion and re-ranking approach, in: [29], 2024.

[15] D. Alexander, M. Fröbe, G. Hendriksen, F. Schlatt, M. Hagen, D. Hiemstra, M. Potthast, A. P. de Vries, Team openwebsearch at clef 2024: Longeval, in: [29], 2024.

[16] M. Gründel, M. Weber, J. Franke, J. H. Reimer, Team galápagos tortoise at longeval 2024: Neural re-ranking and rank fusion for temporal stability, in: [29], 2024.

[17] F. Giachelle, O. Irrera, G. Silvello, Doctag: A customizable annotation tool for ground truth creation, in: Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part II, volume 13186 of *Lecture Notes in Computer Science*, Springer, 2022, pp. 288-293.

[18] D. Harman, TREC-Style Evaluations, Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, pp. 97-115. URL: https://doi.org/10.1007/978-3-642-36415-0_7. doi:10.1007/978-3-642-36415-0_7.

[19] D. Küçük, F. Can, Stance detection: A survey, ACM Comput. Surv. 53 (2020). URL: https://doi.org/10.1145/3369026. doi:10.1145/3369026.

[20] S. M. Mohammad, P. Sobhani, S. Kirchenko, Stance and sentiment in Tweets, ACM Transactions on Internet Technology 17 (2017). URL: http://alt.qcri.org/semeval2016/task6/. doi:10.1145/3003433. arXiv:1605.01655.

[21] R. Alkhalifa, A. Zubiaga, Capturing stance dynamics in social media: open challenges and research directions, International Journal of Digital Humanities (2022) 1-21.

[22] R. Alkhalifa, I. Bilal, H. Borkakoty, J. Camacho-Collados, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, G. Gonzalez-Saez, P. Galuščáková, L. Goeuriot, E. Kochkina, M. Liakata, D. Loureiro, H. Tayyar Madabushi, P. Mulhem, F. Piroi, M. Popel, C. Servan, A. Zubiaga, Longeval: Longitudinal evaluation of model performance at clef 2023, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2023.

[23] R. Alkhalifa, T. Yoong, E. Kochkina, A. Zubiaga, M. Liakata, QMUL-SDS at checkthat! 2020: Determining COVID-19 tweet check-worthiness using an enhanced CT-BERT with numeric expressions, CoRR abs/2008.13160 (2020). URL: https://arxiv.org/abs/2008.13160. arXiv:2008.13160.

[24] D. Effrosynidis, A. I. Karasakalidis, G. Sylaios, A. Arampatzis, The climate change twitter dataset, Expert Systems with Applications 204 (2022) 117541. URL: https://www.sciencedirect.com/science/
article/pii/S0957417422008624. doi:https://doi.org/10.1016/j.eswa.2022.117541.

[25] R. Alkhalifa, I. M. Bilal, H. Borkakoty, Romain, Deveaud, A. El-Ebshihy, Luis, Espinosa-Anke, Gabriela, Gonzalez-Saez, P. Galuscáková, L. Goeuriot, E. Kochkina, M. Liakata, D. Loureiro, P. Mulhem, F. Piroi, M. Popel, C. Servan, H. T. Madabushi, Arkaitz, Zubiaga, Extended overview of the clef-2023 longeval lab on longitudinal evaluation of model performance, 2023. URL: https://api.semanticscholar.org/CorpusID:259953335.

[26] R. Alkhalifa, E. Kochkina, A. Zubiaga, Opinions are made to be changed: Temporally adaptive stance classification, in: Proceedings of the 2021 Workshop on Open Challenges in Online Social Networks, 2021, pp. 27–32.

[27] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.

[28] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019).

[29] G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Proceedings of Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, Aachen, 2024.
A. Runs submitted to the IR Task

<caption>Table 9</caption>

The original name of the submitted runs for the IR task are shown in the second column while the Runs Ids used
assigned to the systems and used in the paper are shown in the first column.

<table><thead><tr><th>Run Id</th><th>Submitted System</th></tr></thead><tbody><tr><td>abyss_run_1</td><td>ABYSS_BM25-French-Stop50_40FR_10EN-SnowStem-Dict-Fuzzy-Phrase-Start-Synonyms-RR</td></tr><tr><td>abyss_run_2</td><td>ABYSS_BM25-French-Stop50_40FR_10EN-SnowStem-Fuzzy-Phrase-Start</td></tr><tr><td>abyss_run_3</td><td>ABYSS_BM25-French-Stop50_40FR_10EN-SnowStem-Fuzzy-Phrase-Start-RR</td></tr><tr><td>cir_run_1</td><td>CIR_BM25</td></tr><tr><td>cir_run_2</td><td>CIR_BM25+monoT5</td></tr><tr><td>cir_run_3</td><td>CIR_BM25+qrel</td></tr><tr><td>cir_run_4</td><td>CIR_BM25+RF</td></tr><tr><td>cir_run_5</td><td>CIR_BM25+time boost</td></tr><tr><td>galapagos_run_1</td><td>galapagos-tortoise-bm25-bo1-pl2-monot5-kmax-avg-k-4</td></tr><tr><td>galapagos_run_2</td><td>galapagos-tortoise-bm25-bo1-pl2-monot5-max</td></tr><tr><td>galapagos_run_3</td><td>galapagos-tortoise-bm25-bo1-pl2-monot5-mean</td></tr><tr><td>galapagos_run_4</td><td>galapagos-tortoise-rank-zephyr</td></tr><tr><td>galapagos_run_5</td><td>galapagos-tortoise-wsum</td></tr><tr><td>kalu_run_1</td><td>KALU_MISTRAL_FRENCH</td></tr><tr><td>kalu_run_2</td><td>KALU_RERANK_HARMONIC_MISTRAL_FRENCH</td></tr><tr><td>kalu_run_3</td><td>KALU_RERANK_HARMONIC_MISTRAL_FRENCH_SHOULD</td></tr><tr><td>kalu_run_4</td><td>KALU_RERANK_simple_fRENCH_LLAMA</td></tr><tr><td>kalu_run_5</td><td>KALU_RERANK_simple_MISTRAL_FRENCH</td></tr><tr><td>ows_run_1</td><td>ows_bm25-bo1_keyqueries</td></tr><tr><td>ows_run_2</td><td>ows_bm25_reverted_index</td></tr><tr><td>ows_run_3</td><td>ows_ltr_all</td></tr><tr><td>ows_run_4</td><td>ows_ltr_wows_all_rerank</td></tr><tr><td>ows_run_5</td><td>ows_ltr_wows_base_rerank</td></tr><tr><td>ows_run_6</td><td>ows_ltr_wows_rerank_and_keyquery</td></tr><tr><td>ows_run_7</td><td>ows_ltr_wows_rerank_and_reverted_index</td></tr><tr><td>quokkas_run_1</td><td>Quokkas_french-letter-lightstem</td></tr><tr><td>quokkas_run_2</td><td>Quokkas_french-standard-lightstem</td></tr><tr><td>dam_run_1</td><td>seupd2324-dam_EN-Stop-SnowBall-Poss-Prox(50)</td></tr><tr><td>dam_run_2</td><td>seupd2324-dam_EN-Stop-SnowBall-Poss-Prox(50)-Reranking(200)</td></tr><tr><td>dam_run_3</td><td>seupd2324-dam_FR-Stop-FrenchLight-Elision-ICU-Proxy(50)</td></tr><tr><td>dam_run_4</td><td>seupd2324-dam_FR-Stop-FrenchLight-Elision-ICU-Proxy(50)-Reranking(150)</td></tr><tr><td>dam_run_5</td><td>seupd2324-dam_FR-Stop-FrenchLight-Elision-ICU-Shingles-Prox(50)-Reranking(150)</td></tr><tr><td>iris_run_1</td><td>seupd2324-iris_FR_GFF@12_w0.162_MMARCO@1000_ADD_w5</td></tr><tr><td>iris_run_2</td><td>seupd2324-iris_FR_GFF@12_w0.162_MMARCO@1000_MAXMIN_ADD_w5</td></tr><tr><td>iris_run_3</td><td>seupd2324-iris_FR_MMARCO@1000_ADD_w5</td></tr><tr><td>iris_run_4</td><td>seupd2324-iris_FR_url_w1.4_GFF@12_w0.162_MMARCO@1000_ADD_w5</td></tr><tr><td>iris_run_5</td><td>seupd2324-iris_FR_Q2K@1_w0.16_MMARCO@1000_MAXMIN_ADD_w5</td></tr><tr><td>lfzzo_run_1</td><td>seupd2324-lfzzo-englishSystem1</td></tr><tr><td>lfzzo_run_2</td><td>seupd2324-lfzzo-englishSystem2</td></tr><tr><td>lfzzo_run_3</td><td>seupd2324-lfzzo-englishSystem3</td></tr><tr><td>lfzzo_run_4</td><td>seupd2324-lfzzo-englishSystem4</td></tr><tr><td>lfzzo_run_5</td><td>seupd2324-lfzzo-englishSystem5</td></tr><tr><td>lfzzo_run_6</td><td>seupd2324-lfzzo-frenchSystem1</td></tr><tr><td>lfzzo_run_7</td><td>seupd2324-lfzzo-frenchSystem2</td></tr><tr><td>lfzzo_run_8</td><td>seupd2324-lfzzo-frenchSystem3</td></tr><tr><td>lfzzo_run_9</td><td>seupd2324-lfzzo-frenchSystem4</td></tr><tr><td>lfzzo_run_10</td><td>seupd2324-lfzzo-frenchSystem5</td></tr><tr><td>mouse_run_1</td><td>seupd2324-mouse_English_Porter_Standard_NoStop_Mixtral-8x7b_NoRerank</td></tr><tr><td>mouse_run_2</td><td>seupd2324-mouse_English_Porter_Standard_stopwords-en_Llama3-70b_NoRerank</td></tr><tr><td>mouse_run_3</td><td>seupd2324-mouse_English_Porter_Standard_top125_Llama3-70b_Cohere-100-w06</td></tr><tr><td>mouse_run_4</td><td>seupd2324-mouse_English_Porter_Standard_top125_Llama3-70b_Pygaggle-Luyu-20-w06</td></tr><tr><td>mouse_run_5</td><td>seupd2324-mouse_English_Porter_Standard_top125_Mixtral-8x7b_Pygaggle-Luyu-20-w06</td></tr><tr><td>mouse_run_6</td><td>seupd2324-mouse_French_FrenchLight_Standard_NoStop_Mixtral-8x7b_NoRerank</td></tr><tr><td>mouse_run_7</td><td>seupd2324-mouse_French_FrenchLight_Standard_stopwords-fr_Llama3-70b_NoRerank</td></tr><tr><td>mouse_run_8</td><td>seupd2324-mouse_French_FrenchLight_Standard_top125_Llama3-70b_Cohere-100-w06</td></tr><tr><td>mouse_run_9</td><td>seupd2324-mouse_French_FrenchLight_Standard_top125_Llama3-70b_Pygaggle-Luyu-20-w06</td></tr><tr><td>mouse_run_10</td><td>seupd2324-mouse_French_FrenchLight_Standard_top125_Mixtral-8x7b_Pygaggle-Luyu-20-w06</td></tr><tr><td>seekx_run_1</td><td>seupd2324-seekx_LetLightFR</td></tr><tr><td>seekx_run_2</td><td>seupd2324-seekx_LetLightStopFR</td></tr><tr><td>seekx_run_3</td><td>seupd2324-seekx_LetLightStopSynFR</td></tr><tr><td>seekx_run_4</td><td>seupd2324-seekx_StanMinEN</td></tr><tr><td>seekx_run_5</td><td>seupd2324-seekx_StanMinSynEN</td></tr><tr><td>snu_run_1</td><td>SNU_LDI_listt5</td></tr><tr><td>snu_run_2</td><td>SNU_LDI_monot5</td></tr><tr><td>wonder_run_1</td><td>WONDER_BASELINE</td></tr><tr><td>wonder_run_2</td><td>WONDER_ENGLISH</td></tr><tr><td>wonder_run_3</td><td>WONDER_ENGLISH_FRENCH</td></tr><tr><td>wonder_run_4</td><td>WONDER_FRENCH</td></tr><tr><td>wonder_run_5</td><td>WONDER_TWOPHASE</td></tr><tr><td>xplore_run_1</td><td>XPLORE_French-BM25-FrenchLight-Stop</td></tr><tr><td>xplore_run_2</td><td>XPLORE_French-BM25-FrenchLight-Stop-SynonymMapper</td></tr><tr><td>xplore_run_3</td><td>XPLORE_French-BM25Default-FrenchLight-Stop</td></tr><tr><td>xplore_run_4</td><td>XPLORE_French-LMDirichlet-FrenchLight-Stop</td></tr></tbody></table>