<figure><img src="image_1.png" /></figure>

HAL
open science

# Information Filtering Evaluation: Overview of CLEF 2009 INFILE Track

Romaric Besançon, Stéphane Chaudiron, Djamel Mostefa, Ismaïl Timimi,

Khalid Choukri, Laïb Meriama

► To cite this version:

Romaric Besançon, Stéphane Chaudiron, Djamel Mostefa, Ismaïl Timimi, Khalid Choukri, et al. Information Filtering Evaluation: Overview of CLEF 2009 INFILE Track. Cross Language Evaluation Forum 2009, ISTI-CNR, Pisa, Italy, Sep 2009, Corfu, Greece. pp.342-353. hal-04542264

HAL Id: hal-04542264

https://hal.science/hal-04542264

Submitted on 11 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Overview of CLEF 2009 INFILE track

Romaric Besançon*, Stéphane Chaudiron**, Djamel Mostefa+, Ismaël Timimi**, Khalid Choukri+, Meriama Laïb*

*CEA LIST

18, route du panorama BP 6 92265
Fontenay aux Roses France

**Université de Lille 3 - GERiico

Domaine univ. du Pont de Bois 55-57,
BP 60149 - 59653 Villeneuve d'Ascq cedex France

+ELDA

rue Brillat Savarin
75013 Paris France

romaric.besancon@cea.fr, meriama.laib@cea.fr, stephane.chaudiron@univ-lille3.fr, mostefa@elda.org, ismail.timimi@univ-lille3.fr, choukri@elda.org

## Abstract

The INFILE@CLEF 2009 track is the second run of this track on the evaluation of cross-language adaptive filtering systems. It uses the same corpus as the 2008 track, composed of 300,000 newswires from Agence France Presse (AFP) in three languages: Arabic, English and French, and a set of 50 topics in general and specific domain (scientific and technological information). We proposed this year two tasks : a batch filtering task and an interactive task to test adaptive methods. Results for the two tasks are presented in this paper.

## Categories and Subject Descriptors

H.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4 Systems and Software

## General Terms

Measurement, Performance, Experimentation, Algorithms

## Keywords

Information Filtering, Competitive Intelligence

## 1 Introduction

The purpose of the INFILE (INformation FILtering Evaluation) track is to evaluate cross-language adaptive filtering systems, i.e. the ability of automated systems to successfully separate relevant and non-relevant documents in an incoming stream of textual information with respect to a given profile, the document and profile being possibly written in different languages.

The INFILE track has first been run as a pilot track in CLEF 2008 campaign [Besançon et al, 2008]. Due to some delays in the organization, the participation in the 2008 was weak (only one participant submitted results), so we decided to propose to rerun the campaign in 2009, using the same document collection and topics.

The INFILE project is funded by the French National Research Agency and co-organized by the CEA LIST, ELDA and the University of Lille3-GERiico.

Information filtering in the INFILE track is considered in the context of competitive intelligence: in this context, the evaluation protocol of the campaign has been designed with a particular attention to the context of use of filtering systems by real professional users. Even if the campaign is mainly a technological oriented evaluation process, we adapted the protocol and the metrics, as close as
# Overview of CLEF 2009 INFILE track

Romaric Besançon*, Stéphane Chaudiron**, Djamel Mostefa+, Ismaël Timimi**, Khalid Choukri+, Meriama Laïb*

*CEA LIST

18, route du panorama BP 6 92265

Fontenay aux Roses France

**Université de Lille 3 - GERiiCO

Domaine univ. du Pont de Bois 55-57,

BP 60149 - 59653 Villeneuve d'Ascq cedex France

+ELDA

rue Brillat Savarin

75013 Paris France

romaric.besancon@cea.fr, meriama.laib@cea.fr, stephane.chaudiron@univ-lille3.fr, mostefa@elda.org, ismail.timimi@univ-lille3.fr, choukri@elda.org

## Abstract

The INFILE@CLEF 2009 track is the second run of this track on the evaluation of cross-language adaptive filtering systems. It uses the same corpus as the 2008 track, composed of 300,000 newswires from Agence France Presse (AFP) in three languages: Arabic, English and French, and a set of 50 topics in general and specific domain (scientific and technological information). We proposed this year two tasks : a batch filtering task and an interactive task to test adaptive methods. Results for the two tasks are presented in this paper.

## Categories and Subject Descriptors

H.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3 Information Search and Retrieval; H.3.4 Systems and Software

## General Terms

Measurement, Performance, Experimentation, Algorithms

## Keywords

Information Filtering, Competitive Intelligence

## 1 Introduction

The purpose of the INFILE (INformation FILtering Evaluation) track is to evaluate cross-language adaptive filtering systems, i.e. the ability of automated systems to successfully separate relevant and non-relevant documents in an incoming stream of textual information with respect to a given profile, the document and profile being possibly written in different languages.

The INFILE track has first been run as a pilot track in CLEF 2008 campaign [Besançon et al, 2008]. Due to some delays in the organization, the participation in the 2008 was weak (only one participant submitted results), so we decided to propose to rerun the campaign in 2009, using the same document collection and topics.

The INFILE project is funded by the French National Research Agency and co-organized by the CEA LIST, ELDA and the University of Lille3-GERiiCO.

Information filtering in the INFILE track is considered in the context of competitive intelligence: in this context, the evaluation protocol of the campaign has been designed with a particular attention to the context of use of filtering systems by real professional users. Even if the campaign is mainly a technological oriented evaluation process, we adapted the protocol and the metrics, as close as
possible, to how a normal user would proceed, including through some interaction and adaptation of
his system.

The INFILE campaign can mainly be seen as a cross-lingual pursuit of the TREC 2002 Adaptive
Filtering task [Robertson and Soboroff, 2002] (adaptive filtering track has been run from 2000 to
2002), with a particular interest in the correspondence of the protocol with the ground truth of
competitive intelligence (CI) professionals. In this goal, we asked CI professionals to write the
topics according to their experience in the domain.

Other related campaigns are the Topic Detection and Tracking (TDT) campaigns from 1998 to 2004
[Fiscus and Wheatley, 2004], but in the TDT campaigns, focus was mainly on topics defined as
"events", with a fine granularity level, and often temporally restricted, whereas in INFILE (similar
to TREC 2002), topics are of long-term interest and supposed to be stable, which can induce
different techniques, even if some studies show that some models can be efficiently trained to have
good performance on both tasks [Yang et al., 2005].

## 2 Description of the tasks

In addition to the adaptive filtering task already proposed in 2008 [Besançon et al, 2008], we
introduced in 2009 the possibility to test batch filtering systems.

For both tasks, the document collection consists in a set of newswire articles provided by the
Agence France Presse (AFP) and covering recent years, the topic set is composed of two different
kinds of profiles, one concerning general news and events, and a second one on scientific and
technological subjects.

The filtering process may be crosslingual: English, French and Arabic are available for the
documents and topics, and participants may be evaluated on monolingual runs, bilingual runs, or
multilingual runs (with several target languages).

The purpose of the information filtering process is to associate documents in an incoming stream to
zero, one or several topics: Filtering systems must provide a Boolean decision for each document
with respect to each topic.

For the batch filtering task, participants are provided with the whole document collection and must
return the list of relevant documents for each topic (since the filtering process supposes a binary
decision for each document, the document list does no need to be ranked).

For the adaptive filtering task, the evaluation is performed using an automatic interactive process,
with a simulated user feedback: systems are allowed for each document considered relevant to a
topic to ask for a feedback on this decision (i.e. ask if the document was indeed relevant for the
topic or not), and can modify their behaviour according to the answer. The feedback is allowed only
on kept document, there is no relevance feedback possible on discarded documents. In order to
simulate the limited patience of the user, a limited number of feedbacks is allowed: this number has
been fixed in 2009 to 200 feedbacks (it was 50 in 2008; but most participants considered this
insufficient). The adaptive filtering task uses an interactive client-server protocol, that is described
in more details in [Besançon et al., 2008].

The batch filtering task has been run from April 2nd (document collections and topics made available to the participants) to June 1st (run submission), and the adaptive filtering task has been run from June 3rd to July 10th.
possible, to how a normal user would proceed, including through some interaction and adaptation of
his system.

The INFILE campaign can mainly be seen as a cross-lingual pursuit of the TREC 2002 Adaptive
Filtering task [Robertson and Soboroff, 2002] (adaptive filtering track has been run from 2000 to
2002), with a particular interest in the correspondence of the protocol with the ground truth of
competitive intelligence (CI) professionals. In this goal, we asked CI professionals to write the
topics according to their experience in the domain.

Other related campaigns are the Topic Detection and Tracking (TDT) campaigns from 1998 to 2004
[Fiscus and Wheatley, 2004], but in the TDT campaigns, focus was mainly on topics defined as
"events", with a fine granularity level, and often temporally restricted, whereas in INFILE (similar
to TREC 2002), topics are of long-term interest and supposed to be stable, which can induce
different techniques, even if some studies show that some models can be efficiently trained to have
good performance on both tasks [Yang et al., 2005].

## 2 Description of the tasks

In addition to the adaptive filtering task already proposed in 2008 [Besançon et al, 2008], we
introduced in 2009 the possibility to test batch filtering systems.

For both tasks, the document collection consists in a set of newswire articles provided by the
Agence France Presse (AFP) and covering recent years, the topic set is composed of two different
kinds of profiles, one concerning general news and events, and a second one on scientific and
technological subjects.

The filtering process may be crosslingual: English, French and Arabic are available for the
documents and topics, and participants may be evaluated on monolingual runs, bilingual runs, or
multilingual runs (with several target languages).

The purpose of the information filtering process is to associate documents in an incoming stream to
zero, one or several topics: Filtering systems must provide a Boolean decision for each document
with respect to each topic.

For the batch filtering task, participants are provided with the whole document collection and must
return the list of relevant documents for each topic (since the filtering process supposes a binary
decision for each document, the document list does no need to be ranked).

For the adaptive filtering task, the evaluation is performed using an automatic interactive process,
with a simulated user feedback: systems are allowed for each document considered relevant to a
topic to ask for a feedback on this decision (i.e. ask if the document was indeed relevant for the
topic or not), and can modify their behaviour according to the answer. The feedback is allowed only
on kept document, there is no relevance feedback possible on discarded documents. In order to
simulate the limited patience of the user, a limited number of feedbacks is allowed: this number has
been fixed in 2009 to 200 feedbacks (it was 50 in 2008; but most participants considered this
insufficient). The adaptive filtering task uses an interactive client-server protocol, that is described
in more details in [Besançon et al., 2008].

The batch filtering task has been run from April 2^nd^ (document collections and topics made available to the participants) to June 1^st^ (run submission), and the adaptive filtering task has been run from June 3^rd^ to July 10^th^.
# 3 Test collections

## 3.1 The topics

A set of 50 profiles has been prepared, covering two different categories: the first group (30 topics) deals with general news and events concerning national and international affairs, sports, politics etc and the second one (20 topics) deals with scientific and technological subjects. The scientific topics were developed by CI professionals from INIST<fnref n="1" />, ARIST Nord Pas de Calais<fnref n="2" />, Digiport<fnref n="3" /> and OTO Research<fnref n="4" />. The topics were developed in both English and French. The Arabic version has been translated from English and French by native speakers.

Topics are defined with the following elements: a unique identifier, a title (6 words max.), describing the topic in a few words, a description (20 words max.), corresponding to a sentence-long description, a narrative (60 words max.), corresponding to the description of what should be considered a relevant document and possibly what should not, keywords (up to 5) and an example of relevant text (120 words max.), taken from a document that is not in the collection (typically from the web).

Each record of the structure in the different languages correspond to translations, except for the samples which need to be extracted from real documents. An example of topic in the three languages is presented in Fig. 1.

<figure><img src="image_1.png" /><figcaption>Fig. 1 An example of topic for the INFILE track, in the three languages</figcaption></figure>

<footnote n="1">1 the French Institute for Scientific and Technical Information Center, http://international.inist.fr/</footnote>

<footnote n="2">2 Agence Régionale d'Information Stratégique et Technologique, http://www.aristnpdc.org/</footnote>

<footnote n="3">3 http://www.digiport.org</footnote>

<footnote n="4">4 http://www.otoresearch.fr/</footnote>
# 3 Test collections

## 3.1 The topics

A set of 50 profiles has been prepared, covering two different categories: the first group (30 topics) deals with general news and events concerning national and international affairs, sports, politics etc and the second one (20 topics) deals with scientific and technological subjects. The scientific topics were developed by CI professionals from INIST<fnref n="1" />, ARIST Nord Pas de Calais<fnref n="2" />, Digiport<fnref n="3" /> and OTO Research<fnref n="4" />. The topics were developed in both English and French. The Arabic version has been translated from English and French by native speakers.

Topics are defined with the following elements: a unique identifier, a title (6 words max.), describing the topic in a few words, a description (20 words max.), corresponding to a sentence-long description, a narrative (60 words max.), corresponding to the description of what should be considered a relevant document and possibly what should not, keywords (up to 5) and an example of relevant text (120 words max.), taken from a document that is not in the collection (typically from the web).

Each record of the structure in the different languages correspond to translations, except for the samples which need to be extracted from real documents. An example of topic in the three languages is presented in Fig. 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1 An example of topic for the INFILE track, in the three languages</figcaption></figure>

<footnote n="1">1 the French Institute for Scientific and Technical Information Center, http://international.inist.fr/</footnote>

<footnote n="2">2 Agence Régionale d'Information Stratégique et Technologique, http://www.aristnpdc.org/</footnote>

<footnote n="3">3 http://www.digiport.org</footnote>

<footnote n="4">4 http://www.otoresearch.fr/</footnote>
## 3.2 The document collection

The INFILE corpus is provided by the Agence France Presse (AFP) for research purpose. We used newswire articles in 3 languages: Arabic, English and French<fnref n="5" /> and a 3 years period (2004-2006) which represents a collection of about one and half million newswires for around 10 GB, from which 100,000 documents of each language have been selected to be used for the INFILE filtering test. News articles are encoded in XML format and follow the News Markup Language (NewsML) specifications<fnref n="6" />. An example of document in English is given in Fig. 2. All fields are available to the systems and can be used in the filtering process (including keywords, categorization...).

<figure><img src="image_2.png" /><figcaption>Fig. 2 Exemple of document in the INFILE collection</figcaption></figure>

<footnote n="5">5 Newswires in different languages are not translations from a language to another (it is not an aligned corpus): the same information is generally rewritten to match the interest of the audience in the corresponding country.</footnote>

<footnote n="6">6 NewsML is an XML standard designed to provide a media-independent, structural framework for multi-media news. NewsML was developed by the International Press Telecommunications Council. see http://www.newsml.org/</footnote>
## 3.2 The document collection

The INFILE corpus is provided by the Agence France Presse (AFP) for research purpose. We used newswire articles in 3 languages: Arabic, English and French<fnref n="5" /> and a 3 years period (2004-2006) which represents a collection of about one and half million newswires for around 10 GB, from which 100,000 documents of each language have been selected to be used for the INFILE filtering test. News articles are encoded in XML format and follow the News Markup Language (NewsML) specifications<fnref n="6" />. An example of document in English is given in Fig. 2. All fields are available to the systems and can be used in the filtering process (including keywords, categorization...).

<figure><img src="image_3.png" /><figcaption>Fig. 2 Exemple of document in the INFILE collection</figcaption></figure>

<footnote n="5">5 Newswires in different languages are not translations from a language to another (it is not an aligned corpus): the same information is generally rewritten to match the interest of the audience in the corresponding country.</footnote>

<footnote n="6">6 NewsML is an XML standard designed to provide a media-independent, structural framework for multi-media news. NewsML was developed by the International Press Telecommunications Council. see http://www.newsml.org/</footnote>
Since we need to provide a real-time simulated feedback to the participants, we need to have the identification of relevant documents prior to the campaign, as in [Soboroff and Robertson, 2002]. The method used to build the collection of documents with the knowledge of the relevant documents is presented in details in [Besançon et al., 2008]. A summary of this method is given here.

We used a set of 4 search engines (Lucene<sup>7</sup>, Indri<sup>8</sup>, Zettair<sup>9</sup> and the search engine developed at CEA-LIST) to index the complete collection of 1.4 million documents. Each search engine has been queried using different fields of the topics, which provides us with a pool of runs. We first selected the first 10 retrieved documents of each run, and these documents were assessed manually. We then iterate using a *Mixture of Experts* model, computing a score for each run according to the current assessment and using this score to weight the choice of the next documents to assess. The final document collection is then built by taking all documents that are relevant to at least one topic (core relevant corpus), all documents that have been assessed and judged not relevant (difficult corpus: documents are not relevant, but share something in common with at least one topic, since they have been retrieved by at least one search engine), and a set of documents taken randomly in the rest of the collection (filler corpus, with documents that have not been retrieved by any search engines for any topic, which should limit the number of relevant documents in the corpus that have not been assessed).

Statistics on the number of assessed documents and relevant documents is presented in Table 1. The repartition of relevant documents across topics is presented in Fig3.

<table><caption>**Table 1** Statistics on the number of assessed documents and the number of relevant documents, in each language</caption><thead><tr><th></th><th>eng</th><th>fre</th><th>ara</th></tr></thead><tbody><tr><td>number of documents assessed</td><td>7312</td><td>7886</td><td>5124</td></tr><tr><td>number of relevant documents</td><td>1597</td><td>2421</td><td>1195</td></tr><tr><td>avg number of relevant docs / topic</td><td>31,94</td><td>48,42</td><td>23,9</td></tr><tr><td>std deviation on number of relevant docs / topic</td><td>28,45</td><td>47,82</td><td>23,08</td></tr><tr><td>[min,max] number of relevant docs / topics</td><td>[0,107]</td><td>[0,202]</td><td>[0,101]</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>**Fig. 3** Number of relevant documents for each topic, in each language</figcaption></figure>

## 4 Metrics

The results returned by the participants are binary decisions on the association of a document with a profile. The results, for a given profile, can then be summarized in a contingency table of the form:

<footnote n="7">7 http://lucene.apache.org</footnote>

<footnote n="8">8 http://www.lemurproject.org/indri</footnote>

<footnote n="9">9 http://www.seg.rmit.edu.au/zettair</footnote>
Since we need to provide a real-time simulated feedback to the participants, we need to have the identification of relevant documents prior to the campaign, as in [Soboroff and Robertson, 2002]. The method used to build the collection of documents with the knowledge of the relevant documents is presented in details in [Besançon et al., 2008]. A summary of this method is given here.

We used a set of 4 search engines (Lucene<sup>7</sup>, Indri<sup>8</sup>, Zettair<sup>9</sup> and the search engine developed at CEA-LIST) to index the complete collection of 1.4 million documents. Each search engine has been queried using different fields of the topics, which provides us with a pool of runs. We first selected the first 10 retrieved documents of each run, and these documents were assessed manually. We then iterate using a *Mixture of Experts* model, computing a score for each run according to the current assessment and using this score to weight the choice of the next documents to assess. The final document collection is then built by taking all documents that are relevant to at least one topic (core relevant corpus), all documents that have been assessed and judged not relevant (difficult corpus: documents are not relevant, but share something in common with at least one topic, since they have been retrieved by at least one search engine), and a set of documents taken randomly in the rest of the collection (filler corpus, with documents that have not been retrieved by any search engines for any topic, which should limit the number of relevant documents in the corpus that have not been assessed).

Statistics on the number of assessed documents and relevant documents is presented in Table 1. The repartition of relevant documents across topics is presented in Fig3.

<table><caption>**Table 1** Statistics on the number of assessed documents and the number of relevant documents, in each language</caption><thead><tr><th></th><th>eng</th><th>fre</th><th>ara</th></tr></thead><tbody><tr><td>number of documents assessed</td><td>7312</td><td>7886</td><td>5124</td></tr><tr><td>number of relevant documents</td><td>1597</td><td>2421</td><td>1195</td></tr><tr><td>avg number of relevant docs / topic</td><td>31,94</td><td>48,42</td><td>23,9</td></tr><tr><td>std deviation on number of relevant docs / topic</td><td>28,45</td><td>47,82</td><td>23,08</td></tr><tr><td>[min,max] number of relevant docs / topics</td><td>[0,107]</td><td>[0,202]</td><td>[0,101]</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 3** Number of relevant documents for each topic, in each language</figcaption></figure>

## 4 Metrics

The results returned by the participants are binary decisions on the association of a document with a profile. The results, for a given profile, can then be summarized in a contingency table of the form:

<footnote n="7">7 http://lucene.apache.org</footnote>

<footnote n="8">8 http://www.lemurproject.org/indri</footnote>

<footnote n="9">9 http://www.seg.rmit.edu.au/zettair</footnote>
<table><thead><tr><th></th><th>Relevant</th><th>Not Relevant</th></tr></thead><tbody><tr><th>Retrieved</th><td>a</td><td>b</td></tr><tr><th>Not Retrieved</th><td>c</td><td>d</td></tr></tbody></table>

On these data, a set of standard evaluation measures is computed:

* Precision, defined as $P=\frac{a}{a+b}$

* Recall, defined as $R=\frac{a}{a+c}$

* F-measure, which is a standard combination of precision and recall [Van Rijsbergen, 1979]

$$F=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}}$$

depending on a parameter $\alpha$, and defined as

We used the standard value $\alpha=0.5$, which gives the same importance to precision and recall (F-measure is then the harmonic mean of the two values).

Following the TREC Filtering tracks [Hull and Roberston, 1999, Robertson and Soboroff, 2002] and the TDT 2004 Adaptive tracking task [Fiscus and Wheatley, 2004], we also consider the linear utility, defined as

$$u=w_1 \times a - w_2 \times b$$

where $w_1$ is the importance given to a relevant document retrieved and $w_2$ is the cost of a non relevant document retrieved.

Linear utility is bounded positively (to 1 for a perfect filtering), but unbounded negatively (negative values depend on the number of relevant documents for a profile). Hence, the average value on all profiles would give too much importance to the few profiles on which a systems would perform

$$u_n=\frac{\max(\frac{u}{u_{max}},u_{min})-u_{min}}{1-u_{min}}$$

poorly. To be able to average the value, the measure is scaled as follows:

where $u_{max}$ is the maximum value of the utility and $u_{min}$ a parameter considered to be the minimum utility value under which a user would not even consider the following documents for the profile. In the INFILE campaign, we used the values $w_1=1$, $w_2=0.5$, $u_{min}=-0.5$ (same as in TREC 2002).

We considered last year the detection cost measure (from the Topic Detection and Tracking campaigns [NIST, 1998]), but we do not present this score in this paper (we found that detection cost values were often low and not really discriminant between participants).

To compute average scores, the values are first computed for each profile and then averaged. In order to measure the adaptivity of the systems in the adaptive filtering track, the measures are also computed at different times in the process, each 10,000 documents, and an evolution curve of the different values across time is presented.

Additionally, we use the two following measures, introduced last year in INFILE: the first one is an originality measure, defined as a comparative measure corresponding to the number of relevant documents the system uniquely retrieves (among participants). It gives more importance to systems
<table><thead><tr><th></th><th>Relevant</th><th>Not Relevant</th></tr></thead><tbody><tr><th>Retrieved</th><td>a</td><td>b</td></tr><tr><th>Not Retrieved</th><td>c</td><td>d</td></tr></tbody></table>

On these data, a set of standard evaluation measures is computed:

* Precision, defined as $P=\frac{a}{a+b}$

* Recall, defined as $R=\frac{a}{a+c}$

* F-measure, which is a standard combination of precision and recall [Van Rijsbergen, 1979]

$$F=\frac{1}{\alpha \frac{1}{P}+(1-\alpha)\frac{1}{R}}$$

depending on a parameter $\alpha$, and defined as

We used the standard value $\alpha=0.5$, which gives the same importance to precision and recall (F-measure is then the harmonic mean of the two values).

Following the TREC Filtering tracks [Hull and Roberston, 1999, Robertson and Soboroff, 2002] and the TDT 2004 Adaptive tracking task [Fiscus and Wheatley, 2004], we also consider the linear utility, defined as

$$u=w_1 \times a - w_2 \times b$$

where $w_1$ is the importance given to a relevant document retrieved and $w_2$ is the cost of a non relevant document retrieved.

Linear utility is bounded positively (to 1 for a perfect filtering), but unbounded negatively (negative values depend on the number of relevant documents for a profile). Hence, the average value on all profiles would give too much importance to the few profiles on which a systems would perform

$$u_n=\frac{\max(\frac{u}{u_{max}},u_{min})-u_{min}}{1-u_{min}}$$

poorly. To be able to average the value, the measure is scaled as follows:

where $u_{max}$ is the maximum value of the utility and $u_{min}$ a parameter considered to be the minimum utility value under which a user would not even consider the following documents for the profile. In the INFILE campaign, we used the values $w_1=1$, $w_2=0.5$, $u_{min}=-0.5$ (same as in TREC 2002).

We considered last year the detection cost measure (from the Topic Detection and Tracking campaigns [NIST, 1998]), but we do not present this score in this paper (we found that detection cost values were often low and not really discriminant between participants).

To compute average scores, the values are first computed for each profile and then averaged. In order to measure the adaptivity of the systems in the adaptive filtering track, the measures are also computed at different times in the process, each 10,000 documents, and an evolution curve of the different values across time is presented.

Additionally, we use the two following measures, introduced last year in INFILE: the first one is an originality measure, defined as a comparative measure corresponding to the number of relevant documents the system uniquely retrieves (among participants). It gives more importance to systems
that use innovative and promising technologies that retrieve "difficult" documents.

The second one is an anticipation measure, designed to give more interest to systems that can find the first document in a given profile. This measure is motivated in CI by the interest of being at the cutting edge of a domain, and not missing the first information to be reactive. It is measured by the inverse rank of the first relevant document detected (in the list of the documents), averaged on all profiles. The measure is similar to the mean reciprocal rank (MRR) used for instance in Question Answering Evaluation [Voorhees, 1999], but is not computed on the ranked list of retrieved documents but on the chronological list of the relevant documents.

## 5 Overview of the results

On the 9 participants registered for both tasks, 5 submitted results: 3 participants submitted results for the batch filtering task (a total of 9 runs), 2 for the interactive filtering task (3 runs). Participants were different for the two tasks. Table 1 present the participant list.

<table><caption>**Table 1** Participant list</caption><thead><tr><th>team name</th><th>institute</th><th>country</th></tr></thead><tbody><tr><td>IMAG</td><td>Institut Informatique et Mathématiques Appliquées de Grenoble</td><td>France</td></tr><tr><td>SINAI</td><td>University of Jaen</td><td>Spain</td></tr><tr><td>UAIC</td><td>Universitatea Alexandru Ioan Cuza of IASI</td><td>Romania</td></tr><tr><td>HossurTech</td><td>société CADEGE</td><td>France</td></tr><tr><td>UOWD</td><td>University of Wollongong (Comp.Sci & Engineering)</td><td>Dubai</td></tr></tbody></table>

Concerning the languages, 6 runs out of 9 are monolingual English for the batch filtering task, 3 are multilingual from English to French/English. For the interactive task, one run is monolingual English, one is monolingual French, and one is bilingual French to English. Table 2 summarizes the total number of runs for each language pair. No participant submitted runs with Arabic as source or target language.

<table><caption>**Table 2** Repartition of runs according to the source and target languages</caption><thead><tr><th colspan="4">nb runs</th></tr><tr><th></th><th>english</th><th>French</th><th>Arabic</th></tr></thead><tbody><tr><th>English</th><td>10</td><td>3</td><td>0</td></tr><tr><th>French</th><td>1</td><td>1</td><td>0</td></tr><tr><th>Arabic</th><td>0</td><td>0</td><td>0</td></tr></tbody></table>

The runs and their characteristics are presented in Table 3.

<table><caption>**Table 3** The runs, by team and by run name, and their characteristics</caption><thead><tr><th>team</th><th>run</th><th>task</th><th>source</th><th>target</th><th>topic fields</th><th>document fields</th></tr></thead><tbody><tr><td>IMAG</td><td>IMAG_1</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>batch</td><td>eng</td><td>eng</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>Headline, DataContent, FileName</td></tr><tr><td>SINAI</td><td>topics_1</td><td>batch</td><td>eng</td><td>eng</td><td></td><td></td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>batch</td><td>eng</td><td>eng</td><td></td><td></td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>adaptive</td><td>fre</td><td>eng</td><td>all</td><td></td></tr><tr><td>HossurTech</td><td>hossur-tech-004</td><td>adaptive</td><td>fre</td><td>fre</td><td>all</td><td></td></tr><tr><td>UOWD</td><td>base</td><td>adaptive</td><td>eng</td><td>eng</td><td>title,desc</td><td>DataContent</td></tr></tbody></table>
that use innovative and promising technologies that retrieve "difficult" documents.

The second one is an anticipation measure, designed to give more interest to systems that can find the first document in a given profile. This measure is motivated in CI by the interest of being at the cutting edge of a domain, and not missing the first information to be reactive. It is measured by the inverse rank of the first relevant document detected (in the list of the documents), averaged on all profiles. The measure is similar to the mean reciprocal rank (MRR) used for instance in Question Answering Evaluation [Voorhees, 1999], but is not computed on the ranked list of retrieved documents but on the chronological list of the relevant documents.

## 5 Overview of the results

On the 9 participants registered for both tasks, 5 submitted results: 3 participants submitted results for the batch filtering task (a total of 9 runs), 2 for the interactive filtering task (3 runs). Participants were different for the two tasks. Table 1 present the participant list.

<table><caption>**Table 1** Participant list</caption><thead><tr><td>team name</td><td>institute</td><td>country</td></tr></thead><tbody><tr><td>IMAG</td><td>Institut Informatique et Mathématiques Appliquées de Grenoble</td><td>France</td></tr><tr><td>SINAI</td><td>University of Jaen</td><td>Spain</td></tr><tr><td>UAIC</td><td>Universitatea Alexandru Ioan Cuza of IASI</td><td>Romania</td></tr><tr><td>HossurTech</td><td>société CADEGE</td><td>France</td></tr><tr><td>UOWD</td><td>University of Wollongong (Comp.Sci & Engineering)</td><td>Dubai</td></tr></tbody></table>

Concerning the languages, 6 runs out of 9 are monolingual English for the batch filtering task, 3 are multilingual from English to French/English. For the interactive task, one run is monolingual English, one is monolingual French, and one is bilingual French to English. Table 2 summarizes the total number of runs for each language pair. No participant submitted runs with Arabic as source or target language.

<table><caption>**Table 2** Repartition of runs according to the source and target languages</caption><thead><tr><th colspan="4">nb runs</th></tr><tr><th></th><th>english</th><th>French</th><th>Arabic</th></tr></thead><tbody><tr><th>English</th><td>10</td><td>3</td><td>0</td></tr><tr><th>French</th><td>1</td><td>1</td><td>0</td></tr><tr><th>Arabic</th><td>0</td><td>0</td><td>0</td></tr></tbody></table>

The runs and their characteristics are presented in Table 3.

<table><caption>**Table 3** The runs, by team and by run name, and their characteristics</caption><thead><tr><th>team</th><th>run</th><th>task</th><th>source</th><th>target</th><th>topic fields</th><th>document fields</th></tr></thead><tbody><tr><td>IMAG</td><td>IMAG_1</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>batch</td><td>eng</td><td>eng</td><td>all</td><td>all</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>batch</td><td>eng</td><td>eng</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>DateID, NewsItemID, Slugline, Headline, DataContent, Country, City, FileName</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>batch</td><td>eng</td><td>eng-fre</td><td>num, title, desc, narr, keywords, sample</td><td>Headline, DataContent, FileName</td></tr><tr><td>SINAI</td><td>topics_1</td><td>batch</td><td>eng</td><td>eng</td><td></td><td></td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>batch</td><td>eng</td><td>eng</td><td></td><td></td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>adaptive</td><td>fre</td><td>eng</td><td>all</td><td></td></tr><tr><td>HossurTech</td><td>hossur-tech-004</td><td>adaptive</td><td>fre</td><td>fre</td><td>all</td><td></td></tr><tr><td>UOWD</td><td>base</td><td>adaptive</td><td>eng</td><td>eng</td><td>title,desc</td><td>DataContent</td></tr></tbody></table>
Evaluation scores for the runs in the batch filtering task are presented in Table 4, gathered by the target language (multilingual runs appear in several groups, in order to present the individual scores on each target language). Best result is obtained on monolingual English, but for the only participant that tried multilingual runs, the results obtained for the different target languages (English and French) are comparable.

### monolingual english

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>IMAG</td><td>IMAG_1</td><td>1597</td><td>413</td><td>0,26</td><td>0,30</td><td>0,21</td><td>0,21</td><td>0,43</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>1597</td><td>1267</td><td>0,09</td><td>0,66</td><td>0,13</td><td>0,05</td><td>0,73</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>1597</td><td>1331</td><td>0,06</td><td>0,69</td><td>0,09</td><td>0,03</td><td>0,75</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>1597</td><td>1331</td><td>0,06</td><td>0,69</td><td>0,09</td><td>0,03</td><td>0,75</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>1597</td><td>1507</td><td>0,06</td><td>0,82</td><td>0,09</td><td>0,03</td><td>0,86</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>1597</td><td>109</td><td>0,13</td><td>0,09</td><td>0,07</td><td>0,16</td><td>0,22</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>1597</td><td>66</td><td>0,16</td><td>0,06</td><td>0,07</td><td>0,22</td><td>0,14</td></tr><tr><td>SINAI</td><td>topics_1</td><td>1597</td><td>940</td><td>0,02</td><td>0,50</td><td>0,04</td><td>0,00</td><td>0,57</td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>1597</td><td>196</td><td>0,01</td><td>0,08</td><td>0,01</td><td>0,13</td><td>0,10</td></tr></tbody></table>

### crosslingual english → french

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_4</td><td>2421</td><td>1120</td><td>0,09</td><td>0,44</td><td>0,12</td><td>0,05</td><td>0,58</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>2421</td><td>1905</td><td>0,06</td><td>0,75</td><td>0,10</td><td>0,03</td><td>0,83</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>2421</td><td>1614</td><td>0,06</td><td>0,67</td><td>0,09</td><td>0,02</td><td>0,76</td></tr></tbody></table>

### multilingual english → english/french

<table><caption>**Table 4** Scores for batch filtering runs, sorted by F-score</caption><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_4</td><td>4018</td><td>2387</td><td>0,07</td><td>0,56</td><td>0,11</td><td>0,02</td><td>0,72</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>4018</td><td>3412</td><td>0,05</td><td>0,81</td><td>0,08</td><td>0,02</td><td>0,85</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>4018</td><td>2945</td><td>0,05</td><td>0,70</td><td>0,07</td><td>0,02</td><td>0,80</td></tr></tbody></table>

Scores for the runs in the adaptive filtering task are presented in Table 5. The scores are worse than the scores obtained on the batch filtering results, but the language pairs and the participants are not the same. We also note that both batch and adaptive results for the INFILE 2009 campaign are worse than the results obtained for the adaptive task in the INFILE 2008 edition.

### monolingual english

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UOWD</td><td>base</td><td>1597</td><td>20</td><td>0,00</td><td>0,01</td><td>0,01</td><td>0,03</td><td>0,05</td></tr></tbody></table>

### monolingual french

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-004</td><td>2421</td><td>790</td><td>0,05</td><td>0,31</td><td>0,06</td><td>0,05</td><td>0,53</td></tr></tbody></table>

### crosslingual french → english

<table><caption>**Table 5** Scores for adaptive filtering runs</caption><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-001</td><td>1597</td><td>819</td><td>0,10</td><td>0,45</td><td>0,10</td><td>0,07</td><td>0,59</td></tr></tbody></table>

Results for originality measure are presented in Table 6. The upper part of the table present
Evaluation scores for the runs in the batch filtering task are presented in Table 4, gathered by the target language (multilingual runs appear in several groups, in order to present the individual scores on each target language). Best result is obtained on monolingual English, but for the only participant that tried multilingual runs, the results obtained for the different target languages (English and French) are comparable.

### monolingual english

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>IMAG</td><td>IMAG_1</td><td>1597</td><td>413</td><td>0,26</td><td>0,30</td><td>0,21</td><td>0,21</td><td>0,43</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>1597</td><td>1267</td><td>0,09</td><td>0,66</td><td>0,13</td><td>0,05</td><td>0,73</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>1597</td><td>1331</td><td>0,06</td><td>0,69</td><td>0,09</td><td>0,03</td><td>0,75</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>1597</td><td>1331</td><td>0,06</td><td>0,69</td><td>0,09</td><td>0,03</td><td>0,75</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>1597</td><td>1507</td><td>0,06</td><td>0,82</td><td>0,09</td><td>0,03</td><td>0,86</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>1597</td><td>109</td><td>0,13</td><td>0,09</td><td>0,07</td><td>0,16</td><td>0,22</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>1597</td><td>66</td><td>0,16</td><td>0,06</td><td>0,07</td><td>0,22</td><td>0,14</td></tr><tr><td>SINAI</td><td>topics_1</td><td>1597</td><td>940</td><td>0,02</td><td>0,50</td><td>0,04</td><td>0,00</td><td>0,57</td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>1597</td><td>196</td><td>0,01</td><td>0,08</td><td>0,01</td><td>0,13</td><td>0,10</td></tr></tbody></table>

### crosslingual english → french

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_4</td><td>2421</td><td>1120</td><td>0,09</td><td>0,44</td><td>0,12</td><td>0,05</td><td>0,58</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>2421</td><td>1905</td><td>0,06</td><td>0,75</td><td>0,10</td><td>0,03</td><td>0,83</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>2421</td><td>1614</td><td>0,06</td><td>0,67</td><td>0,09</td><td>0,02</td><td>0,76</td></tr></tbody></table>

### multilingual english → english/french

<table><caption>**Table 4** Scores for batch filtering runs, sorted by F-score</caption><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_4</td><td>4018</td><td>2387</td><td>0,07</td><td>0,56</td><td>0,11</td><td>0,02</td><td>0,72</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>4018</td><td>3412</td><td>0,05</td><td>0,81</td><td>0,08</td><td>0,02</td><td>0,85</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>4018</td><td>2945</td><td>0,05</td><td>0,70</td><td>0,07</td><td>0,02</td><td>0,80</td></tr></tbody></table>

Scores for the runs in the adaptive filtering task are presented in Table 5. The scores are worse than the scores obtained on the batch filtering results, but the language pairs and the participants are not the same. We also note that both batch and adaptive results for the INFILE 2009 campaign are worse than the results obtained for the adaptive task in the INFILE 2008 edition.

### monolingual english

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>UOWD</td><td>base</td><td>1597</td><td>20</td><td>0,00</td><td>0,01</td><td>0,01</td><td>0,03</td><td>0,05</td></tr></tbody></table>

### monolingual french

<table><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-004</td><td>2421</td><td>790</td><td>0,05</td><td>0,31</td><td>0,06</td><td>0,05</td><td>0,53</td></tr></tbody></table>

### crosslingual french → english

<table><caption>**Table 5** Scores for adaptive filtering runs</caption><thead><tr><th>team</th><th>run</th><th>num_rel</th><th>num_rel_ret</th><th>precision</th><th>recall</th><th>F-score</th><th>Utility</th><th>anticipation</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-001</td><td>1597</td><td>819</td><td>0,10</td><td>0,45</td><td>0,10</td><td>0,07</td><td>0,59</td></tr></tbody></table>

Results for originality measure are presented in Table 6. The upper part of the table present
originality scores for every run that has the same target language (i.e. the number of relevant documents that this particular run uniquely retrieves). Since this global comparison may not be fair for participants who submitted several runs, which are presumably variants of the same technique and will share most of the relevant retrieved documents, we present in the lower part of the table the originality scores using only one run for each participant (we chose the run with the best recall score). We see here that participant with lower F-scores can have a better originality score. However, due to the small number of participants, the relevance of the originality score is arguable in this context, since it seems to be strongly linked to the difference of the recall score.

originality on all runs

<table><thead><tr><th colspan="3">target lang=eng</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_3</td><td>39</td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>18</td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>15</td></tr><tr><td>SINAI</td><td>topics_1</td><td>9</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>4</td></tr><tr><td>IMAG</td><td>IMAG_1</td><td>1</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>0</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>0</td></tr><tr><td>UOWD</td><td>base</td><td>0</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>0</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>0</td></tr></tbody></table>

<table><thead><tr><th colspan="3">target lang=fre</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-004</td><td>177</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>82</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>0</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>0</td></tr></tbody></table>

originality on best run

<table><thead><tr><th colspan="3">target lang=eng</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_3</td><td>267</td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>20</td></tr><tr><td>SINAI</td><td>topics_1</td><td>9</td></tr><tr><td>IMAG</td><td>IMAG_1</td><td>4</td></tr><tr><td>UOWD</td><td>base</td><td>0</td></tr></tbody></table>

<table><caption>**Table 6** Originality scores</caption><thead><tr><th colspan="3">target lang=fre</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>hossur-tech-004</td><td>1292</td></tr><tr><td>HossurTech</td><td>uaic_3</td><td>177</td></tr></tbody></table>

## 6 Conclusion

The INFILE campaign has been organized for the second time this year in CLEF, to evaluate adaptive filtering systems in a cross-language environment. The document and topic collection were the same as the 2008 edition of the INFILE@CLEF track. Two tasks have been proposed: a batch filtering task and an adaptive filtering task, that used an original setup to simulate the incoming of newswires documents, and the interaction of a user through a simulated feedback. We had this year more participants than last year and more results to analyze. However, the innovative crosslingual aspect of the task has still not really been explored, since most runs were monolingual English and no participant used the Arabic topics or documents. The lack of participation for the adaptive task is also disappointing since it does not provide enough data to compare batch techniques to adaptive techniques and does not allow to conclude on the interest of the use of the used feedback on the documents.

## References
originality scores for every run that has the same target language (i.e. the number of relevant documents that this particular run uniquely retrieves). Since this global comparison may not be fair for participants who submitted several runs, which are presumably variants of the same technique and will share most of the relevant retrieved documents, we present in the lower part of the table the originality scores using only one run for each participant (we chose the run with the best recall score). We see here that participant with lower F-scores can have a better originality score.

However, due to the small number of participants, the relevance of the originality score is arguable in this context, since it seems to be strongly linked to the difference of the recall score.

originality on all runs

<table><thead><tr><th colspan="3">target lang=eng</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_3</td><td>39</td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>18</td></tr><tr><td>SINAI</td><td>googlenews_2</td><td>15</td></tr><tr><td>SINAI</td><td>topics_1</td><td>9</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>4</td></tr><tr><td>IMAG</td><td>IMAG_1</td><td>1</td></tr><tr><td>UAIC</td><td>uaic_1</td><td>0</td></tr><tr><td>IMAG</td><td>IMAG_3</td><td>0</td></tr><tr><td>UOWD</td><td>base</td><td>0</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>0</td></tr><tr><td>IMAG</td><td>IMAG_2</td><td>0</td></tr></tbody></table>

<table><thead><tr><th colspan="3">target lang=fre</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>HossurTech</td><td>hossur-tech-004</td><td>177</td></tr><tr><td>UAIC</td><td>uaic_3</td><td>82</td></tr><tr><td>UAIC</td><td>uaic_2</td><td>0</td></tr><tr><td>UAIC</td><td>uaic_4</td><td>0</td></tr></tbody></table>

originality on best run

<table><thead><tr><th colspan="3">target lang=eng</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>uaic_3</td><td>267</td></tr><tr><td>HossurTech</td><td>hossur-tech-001</td><td>20</td></tr><tr><td>SINAI</td><td>topics_1</td><td>9</td></tr><tr><td>IMAG</td><td>IMAG_1</td><td>4</td></tr><tr><td>UOWD</td><td>base</td><td>0</td></tr></tbody></table>

<table><caption>**Table 6** Originality scores</caption><thead><tr><th colspan="3">target lang=fre</th></tr><tr><th>team</th><th>run</th><th>originality</th></tr></thead><tbody><tr><td>UAIC</td><td>hossur-tech-004</td><td>1292</td></tr><tr><td>HossurTech</td><td>uaic_3</td><td>177</td></tr></tbody></table>

## 6 Conclusion

The INFILE campaign has been organized for the second time this year in CLEF, to evaluate adaptive filtering systems in a cross-language environment. The document and topic collection were the same as the 2008 edition of the INFILE@CLEF track. Two tasks have been proposed: a batch filtering task and an adaptive filtering task, that used an original setup to simulate the incoming of newswires documents, and the interaction of a user through a simulated feedback. We had this year more participants than last year and more results to analyze. However, the innovative crosslingual aspect of the task has still not really been explored, since most runs were monolingual English and no participant used the Arabic topics or documents. The lack of participation for the adaptive task is also disappointing since it does not provide enough data to compare batch techniques to adaptive techniques and does not allow to conclude on the interest of the use of the used feedback on the documents.

## References
[Besancon et al, 2008] Besancon R., Chaudiron S., Mostefa D., Hamon O., Timimi I. and Choukri K. (2008) Overview of CLEF 2008 INFILE Pilot Track, Overview of CLEF 2008 INFILE Pilot Track.

[Fiscus and Wheatley, 2004] Fiscus, J. and Wheatley, B. (2004). Overview of the tdt 2004 evaluation and results. In TDT’02. NIST.

[Hull and Roberston, 1999] Hull, D. and Roberston, S. (1999). The trec-8 filtering track final report. In Proceedings of the Eighth Text REtrieval Conference (TREC-8). NIST.

[NIST, 1998] NIST (1998). The topic detection and tracking phase 2 (tdt2) evaluation plan. http://www.nist.gov/speech/tests/tdt/1998/doc/tdt2.eval.plan.98.v3.7.pdf.

[Robertson and Soboroff, 2002] Robertson, S. and Soboroff, I. (2002). The trec 2002 filtering track report. In Proceedings of The Eleventh Text Retrieval Conference (TREC 2002). NIST.

[Soboroff and Robertson, 2002] Soboroff, I. and Robertson, S. (2002). Building a filtering test collection for trec 2002. In Proceedings of The Eleventh Text Retrieval Conference (TREC 2002). NIST.

[Van Rijsbergen, 1979] Van Rijsbergen, C. (1979). Information Retrieval. Butterworths, London.

[Voorhees, 1999] Voorhees, E. (1999). The trec-8 question answering track report. In Proceedings of the Eighth Text REtrieval Conference (TREC-8). NIST.

[Yang et al., 2005] Yang, Y., Yoo, S., Zhang, J., and Kisiel, B. (2005). Robustness of adaptive filtering methods in a cross-benchmark evaluation. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 98–105, Salvador, Brazil.
[Besancon et al, 2008] Besancon R., Chaudiron S., Mostefa D., Hamon O., Timimi I. and Choukri K. (2008) Overview of CLEF 2008 INFILE Pilot Track, Overview of CLEF 2008 INFILE Pilot Track.

[Fiscus and Wheatley, 2004] Fiscus, J. and Wheatley, B. (2004). Overview of the tdt 2004 evaluation and results. In TDT’02. NIST.

[Hull and Roberston, 1999] Hull, D. and Roberston, S. (1999). The trec-8 filtering track final report. In Proceedings of the Eighth Text Retrieval Conference (TREC-8). NIST.

[NIST, 1998] NIST (1998). The topic detection and tracking phase 2 (tdt2) evaluation plan. http://www.nist.gov/speech/tests/tdt/1998/doc/tdt2.eval.plan.98.v3.7.pdf.

[Robertson and Soboroff, 2002] Robertson, S. and Soboroff, I. (2002). The trec 2002 filtering track report. In Proceedings of The Eleventh Text Retrieval Conference (TREC 2002). NIST.

[Soboroff and Robertson, 2002] Soboroff, I. and Robertson, S. (2002). Building a filtering test collection for trec 2002. In Proceedings of The Eleventh Text Retrieval Conference (TREC 2002). NIST.

[Van Rijsbergen, 1979] Van Rijsbergen, C. (1979). Information Retrieval. Butterworths, London.

[Voorhees, 1999] Voorhees, E. (1999). The trec-8 question answering track report. In Proceedings of the Eighth Text Retrieval Conference (TREC-8). NIST.

[Yang et al., 2005] Yang, Y., Yoo, S., Zhang, J., and Kisiel, B. (2005). Robustness of adaptive filtering methods in a cross-benchmark evaluation. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 98–105, Salvador, Brazil.