# The Quaero Evaluation Campaign on Term Extraction

Thibault Mondary, Adeline Nazarenko, Haifa Zargayouna, Sabine Barreaux

## ► To cite this version:

Thibault Mondary, Adeline Nazarenko, Haifa Zargayouna, Sabine Barreaux. The Quaero Evaluation Campaign on Term Extraction. The 8th international conference on Language Resources and Evaluation (LREC), May 2012, Istanbul, Turkey. pp.663-669. hal-00699356

HAL Id: hal-00699356

https://hal.science/hal-00699356

Submitted on 20 May 2012

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The Quæro Evaluation Initiative on Term Extraction

Thibault Mondary¹, Adeline Nazarenko¹, Haïfa Zargayouna¹, Sabine Barreaux²

¹ LIPN - UMR 7030 CNRS / Paris 13 University, France

² INIST-CNRS, Vandoeuvre-lès-Nancy, France

## Abstract

The Quæro program has organized a set of evaluations for terminology extraction systems in 2010 and 2011. Three objectives were targeted in this initiative: the first one was to evaluate the behavior and scalability of term extractors regarding the size of corpora, the second goal was to assess progress between different versions of the same systems, the last one was to measure the influence of corpus type. The protocol used during this initiative was a comparative analysis of 32 runs against a gold standard. Scores were computed using metrics that take into account gradual relevance. Systems produced by Quæro partners and publicly available systems were evaluated on pharmacology corpora composed of European Patents or abstracts of scientific articles, all in English. The gold standard was an unstructured version of the pharmacology thesaurus used by INIST-CNRS for indexing purposes. Most systems scaled with large corpora, contrasted differences were observed between different versions of the same systems and with better results on scientific articles than on patents. During the ongoing adjudication phase domain experts are enriching the thesaurus with terms found by several systems.

**Keywords:** Term extraction, Evaluation, Quæro, Pharmacology, Gradual relevance, Scalability

## 1. Introduction

Computational terminology is a twenty years old discipline that aims at building automatically or semi- automatically terminological resources from corpora. A lot of terminological tools have been developed, but despite the progress made, it remains difficult to get a clear idea of the maturity of this research field and to compare approaches. Some efforts have been made to set up an evaluation protocol adapted to the specificity of terminological tasks. The TEMREC task of the first NTCIR initiative (Kando et al., 1999) was the first initiative dedicated to the evaluation of term extraction, but it was not reconducted due to its lack of popularity. CoRRecT (Enguehard, 2003) has proposed an interesting dataset and protocol to evaluate term recognition in corpora (a task close to controlled indexing). CE-SART (Mustafa El Hadi et al., 2006) was the most comprehensive challenge: a gold standard list of terms and an acquisition corpus were chosen for a specific domain (medicine), the systems had to extract terms from the acquisition corpus and their results were compared against the gold standard using various relevance criteria.

Quæro is a program promoting research and industrial innovation on technologies related to the analysis and classification of multimedia and multilingual documents¹. Partners technologies are yearly evaluated in internal or external initiatives. In this context, an internal Terminology Extraction evaluation was organized by LIPN in 2010 and reconducted in 2011.

This paper presents the protocol and results obtained during these two evaluation initiatives.

## 2. The Task

Term extractors output lists of relevant terms from a domain specific corpus. The terms may be mono or polylexical units, i.e. made of one or several words. The goal of the evaluation was to measure the quality of the resulting lists of terms, through a comparison with a gold standard. We really focused on the specific task of term extraction without taking into account term ranking and variant clustering. This experiment has been done on the pharmacology domain: we proposed a set of corpora, every participant had to extract a list of relevant terms and submit it for evaluation. A participant could submit several runs and each run was evaluated against the gold standard.

Three objectives were targeted in this initiative. The first one was to evaluate the behavior and scalability of systems regarding the size of corpora (the ability of extractors to deal with large corpora and the stability of scores regarding the size of the acquisition corpora). The second goal was to assess the progress between different versions of the same systems. The last one was to measure the influence of corpus type on the performance of term extractors.

## 3. Corpora and Gold Standard

Two different types of pharmacology corpora were used. A first set of European patents was supplied by Jouve, an industrial Quæro partner. From it, three self-including corpora of growing size (called C1, C2 and

¹http://www.quaero.org
Peptide Extraction From Ionic Conjugates:

[0042] A 50 mg sample of an ionic molecular conjugate was mixed into 20 mls of methylene chloride. The mixture was sequentially extracted with 50 ml, 20 ml and 20 ml portions of 2N acetic acid. The acetic acid extracts were combined and analyzed for peptide content by high performance liquid chromatography(HPLC). Peptide analysis by HPLC is as follows. HPLC analysis were performed using a Waters model M-45 solvent delivery pump and an EM Science MACS 700 detector at wavelength 220 nm and 1.0 AUFS. Peptides were run using a Lichrospher (EM separations) C18, 100A, 5um, 25cm x 4.6 mm column and 30% acetonitrile/0.1% TFA as an isocratic eluent buffer.

[0043] Following are details (Table VI) of...

<table><caption>TABLE VI IN-VITRO ASSAY DATA</caption><thead><tr><th rowspan="2">DAY OF ASSAY</th><th colspan="3">PERCENT OF TOTAL PEPTIDE RELEASED</th></tr><tr><th>Example #8</th><th>Example #9</th><th>Example #10</th></tr></thead><tbody><tr><td>1</td><td>5.5%</td><td>12.5%</td><td>11%</td></tr><tr><td>7</td><td>26.9%</td><td>21.3%</td><td>53%</td></tr><tr><td>14</td><td>55.2%</td><td>47.3%</td><td>55%</td></tr></tbody></table>

...

<caption>Figure 1: Excerpt of a patent</caption>

Voluntary exercise improves stress coping and lowers anxiety. Because of the role of GABA in these processes, we investigated changes in the central GABAergic system in rats with free access to a running wheel for 4 weeks. The control animals had no access to a running wheel. Using in-situ hybridisation histochemistry, we studied changes in gene expression of various GABA::(A) receptor subunits as well as the GABA-synthesising enzyme glutamic acid decarboxylase-67 (GAD67) in the forebrain. There were region-specific decreases in alpha 2, beta 3 and gamma 2 subunit mRNA expression and region-specific increases in beta 1 subunit expression. The alpha 5 and delta subunits, in the forebrain specifically associated with extrasynaptic GABA::(A) receptors in the hippocampus, showed differential increases in expression levels.

...

<caption>Figure 2: Excerpt of a typical abstract from CA</caption>

C3 in the following) were built, in order to test the systems' scalability. C1 contains 500,000 words, C2 1.5M words and C3 2.5M words. An excerpt of a patent is presented on Figure 1.

In order to measure the systems' behavior against another type of corpora of the same domain, an additional corpus (called CA) composed of abstracts of scientific articles in pharmacology was extracted from the PASCAL<fnref n="2" /> database and provided by INIST-CNRS. CA weights 1.5M words (same size as C2), a typical abstract is presented on Figure 2.

The gold standard was an unstructured version of the pharmacology thesaurus<fnref n="3" /> used by INIST-CNRS for indexing purposes. It consists of 76,466 English terms, mainly about general pharmacology, diseases and drugs. An excerpt is presented on Figure 3.

5-HT3 Serotonin receptor

5-HT4 Serotonin receptor

5S-RNA

5s rrna

...

Bacillus subtilis ribonuclease

Bacterial lipopolysaccharide receptors

Connective tissue activating factor

...

Recombinant microorganism

Recombinant protein

Recombinant virus

<caption>Figure 3: Excerpt of the gold standard</caption>

## 4. Metrics

Scoring was performed using terminological precision, recall and F-Measure, introduced in (Nazarenko et al., 2009; Zargayouna and Nazarenko, 2010) and presented on Equations 1–3. The terminological precision and recall metrics take into account a gradual relevance. The systems' outputs are tuned to find their maximal correspondence with the gold standard, which means that the outputs are adapted to the terminological type and granularity of the gold standard.

$$ TPrecision = \frac{\sum_{i \in T(O)} rel_{GS}(i)}{|T(O)|} \quad (1) $$

$$ TRecall = \frac{\sum_{i \in T(O)} rel_{GS}(i)}{|GS|} \quad (2) $$

$$ TFMeasure = \frac{2 \times TPrecision \times TRecall}{TPrecision + TRecall} \quad (3) $$

$|T(O)|$ is the size of tuned term extractor output $T(O)$, $|GS|$ the size of the gold standard. $rel_{GS}(i)$ is the relevance of a term $i$ with respect to the gold standard. It is based on a terminological distance $d_t$ and on a threshold $\tau$ on this distance:

$$ rel_{GS}(i) = \begin{cases} 1 - \delta(i) & \text{if } \delta(i) < \tau \\ 0 & \text{otherwise} \end{cases} \quad (4) $$

where $\delta(i) = min_{e \in GS}(d_t(i, e))$. $d_t$ is computed as the mean of a string and a complex term distances that is based on a normalized edit distance and takes into account word permutation.

LIPN developed a scoring tool called Termometer<fnref n="4" /> which is available under GPLv3.

<footnote n="2">²The multidisciplinary bibliographical database produced by INIST-CNRS. PASCAL page on the official INIST site — http://inist.fr/spip.php?article11.</footnote>

<footnote n="3">³Available on TermSciences, the multidisciplinary terminological portal developed by INIST-CNRS — http://www.termsciences.fr</footnote>

<footnote n="4">⁴http://sourceforge.net/projects/termometerxd</footnote>
<table><caption>Table 1: Challengers, with Quæro partners highlighted</caption><thead><tr><th></th><th>Runs on</th><th>Comes from</th></tr></thead><tbody><tr><td>Acabit 4.3</td><td>C1, C2, C3, CA</td><td>LINA</td></tr><tr><td>Fastr 2.04</td><td>C1</td><td>LIMSI</td></tr><tr><td>Jv1</td><td>C3</td><td>Jouve</td></tr><tr><td>YaTeA 0.5</td><td>C1, C2, C3, CA</td><td>LIPN</td></tr><tr><td>MIG-YaTeA</td><td>C1, C2, C3, CA</td><td>INRA/MIG</td></tr><tr><td>TermExtractor</td><td>C1, C2, C3</td><td>LCL</td></tr><tr><td>TermoStatWeb 2</td><td>C1</td><td>OLST</td></tr><tr><td>TermoStatWebV3-multi</td><td>C1, C2, C3, CA</td><td>OLST</td></tr><tr><td>TermoStatWebV3-nomulti</td><td>C1, C2, C3, CA</td><td>OLST</td></tr><tr><td>TS3-multi</td><td>C1, C2, C3</td><td>INRIA/TEXMEX</td></tr><tr><td>TS3-nomulti</td><td>C1, C2, C3</td><td>INRIA/TEXMEX</td></tr></tbody></table>

## 5. Participants and Systems

Three Quæro partners participated in the evaluation: INRA/MIG (academic partner), INRIA/TEXMEX (academic partner) and Jouve (industrial partner). Six publicly available term extractors, not developed within Quæro, were also tested: Acabit (Daille, 2003), TermExtractor (Sclano and Vellardi, 2007), TermoStatWeb, TermoStatWebV3 (Drouin, 2003; Drouin, 2006) and YaTeA (Aubin and Hamon, 2006). During the second evaluation initiative, one system became unavailable (TermExtractor) and another underwent a major revision (TermoStatWeb). The participating systems are presented on Table 1 and detailed below.

**Acabit** is a terminology extraction tool which takes as input a part-of-speech tagged corpus and proposes as output a ranked list of multi-word terms. Acabit is based on syntactical patterns and statistical filtering. It works on English and French, the tested version was 4.3⁵.

**Fastr** is a multi-lingual tool for automatic indexing. In free indexing mode, it acts as a term extractor and extracts terms with their variations from a corpus. Fastr works on French and English, the tested version was 2.04⁶.

**Jv1** is developed by Jouve for patent classification. It produces nominal or verbal phrases based on shallow parsing and statistical metrics. This system works on English, German and French.

**YaTeA** aims at extracting noun phrases that look like terms from a corpus. It is based on simple syntactic patterns and endogenous disambiguation. Exogenous disambiguation using external resources is also possible, but it was not used here. YaTeA works on English and French, the tested version was 0.5⁷.

**MIG-YaTeA** is an enriched version of YaTeA developed by INRA/MIG. It uses two post-processing filters for cleaning and merging results. The first filter gets rid of extraction errors, incomplete terms, spelling errors and overgeneralized terms. The second filter merges terms sharing the same lemma. For this initiative, only one representative per cluster was output.

**TermExtractor** is an online term extractor designed to build ontologies. It is based on two entropy measures. The first one is used to select the terms which are consensually referred throughout the corpus documents, the second one is used to select only the terms which are relevant to the domain of interest. TermExtractor<fnref n="8" /> works on English.

**TermoStatWeb** is an online term extractor based on TermoStat (Drouin, 2003), which basic principle is to compare the distribution of words between a specialized document (the corpus to be processed) and a large corpus of “general language”, using different statistical measures, e.g. log likelihood, specificity or chi2. It works on English, Spanish, French and Italian. This version has been superseded by TermoStatWebV3 at the end of 2010.

<footnote n="5">⁵http://www.bdaille.fr/index.</footnote>

<footnote>php?option=com_content&task=blogcategory&id=5&Itemid=5</footnote>

<footnote n="6">⁶http://perso.limsi.fr/Individu/jacquemi/FASTR</footnote>

<footnote n="7">⁷http://search.cpan.org/~thhamon/Lingua-YaTeA-0.5</footnote>

<footnote n="8">⁸http://lcl2.di.uniroma1.it/termextractor, available until early 2011</footnote>
**TermoStatWebV3** is the new version of TermoStatWeb<fnref n="9" />. Different runs were produced, some with only simple terms (*TermoStat-WebV3-nomulti*) and other with both simple and multi-terms (*TermoStatWebV3-multi*).

**TS3** is an experimental version of TermoStatWebV3 enriched by INRIA/TexMex with post-processing and filtering. Different runs were produced, some with only simple terms (*TS3-nomulti*) and other with both simple and multi-terms (*TS3-multi*).

## 6. Evaluation Results

Terminological scores have been computed for a total of 32 runs. Table 2 presents the number of returned terms. Huge differences can be observed. Although the corpus of abstracts (CA) contains 1.5MWords as C2, term extractors produce much more term candidates from CA than from C2.

Overall results are presented on Table 3, with best scores highlighted in bold font for each corpus. Two families of systems can be observed from the last two tables: those who favor precision by returning a small amount of correct terms (*Jv1*, TermExtractor, TermoStatWebV3-multi and TermoStatWebV3-nomulti) and those who favor recall (*Acabit*, *Fastr*, *MIG-YaTeA*, *TS3-multi*, *TS3-nomulti* and *YaTeA*). TermoStatWeb cannot be clearly categorized. This difference between systems can be explained by their design choices.

### 6.1. Scalability

Almost all systems scale with large corpora, i.e. provide results without crashing. However, one system has not been tested on C1 and C2 because it was designed to work with very large corpora.

For a first group of systems, precision decreases as the corpus size increases (it is an expected behavior, as more terms are able to be found, more terms might be erroneous). For the systems of a second group (*TermoStatWebV3* and *TS3*), however, the results (amount of terms, TPrecision, TRecall) do not increase monotonically with the corpus size. This is probably due to the statistical underlying filtering strategy.

### 6.2. Systems evolution

Evolution can be observed between the different versions of the same systems. As described before, *TS3* and *TermoStatWebV3* both derived from

<table><caption>Table 3: Evaluation raw results</caption><thead><tr><th></th><th>TPrecision</th><th>TRecall</th><th>TFMeasure</th></tr></thead><tbody><tr><td><strong>Acabit</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>60.31%</td><td>8.65%</td><td>15.13%</td></tr><tr><td>C2</td><td>56.26%</td><td>14.24%</td><td>22.72%</td></tr><tr><td>C3</td><td>53.62%</td><td>16.45%</td><td>25.18%</td></tr><tr><td>CA</td><td>54.06%</td><td>19.74%</td><td>28.92%</td></tr><tr><td><strong>Fastr</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>54.20%</td><td>10.12%</td><td>17.06%</td></tr><tr><td><strong>Jv1</strong></td><td></td><td></td><td></td></tr><tr><td>C3</td><td>74.05%</td><td>1.86%</td><td>3.62%</td></tr><tr><td><strong>YaTeA</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>31.22%</td><td>10.09%</td><td>15.25%</td></tr><tr><td>C2</td><td>23.99%</td><td>16.96%</td><td>19.87%</td></tr><tr><td>C3</td><td>19.24%</td><td>19.80%</td><td>19.52%</td></tr><tr><td>CA</td><td>28.76%</td><td>23.78%</td><td>26.03%</td></tr><tr><td><strong>MIG-YaTeA</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>57.30%</td><td>10.64%</td><td>17.94%</td></tr><tr><td>C2</td><td>52.09%</td><td>18.48%</td><td>27.28%</td></tr><tr><td>C3</td><td>48.85%</td><td><strong>22.13%</strong></td><td><strong>30.46%</strong></td></tr><tr><td>CA</td><td>55.08%</td><td><strong>26.57%</strong></td><td><strong>35.85%</strong></td></tr><tr><td><strong>TermExtractor</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>78.81%</td><td>0.40%</td><td>0.79%</td></tr><tr><td>C2</td><td>76.16%</td><td>0.73%</td><td>1.44%</td></tr><tr><td>C3</td><td>75.82%</td><td>0.83%</td><td>1.64%</td></tr><tr><td><strong>TermoStatWeb</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>67.05%</td><td>5.48%</td><td>10.14%</td></tr><tr><td><strong>TermoStatWebV3-multi</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>76.93%</td><td>4.29%</td><td>8.13%</td></tr><tr><td>C2</td><td>76.32%</td><td>6.90%</td><td>12.66%</td></tr><tr><td>C3</td><td>77.98%</td><td>5.94%</td><td>11.04%</td></tr><tr><td>CA</td><td>83.46%</td><td>10.15%</td><td>18.10%</td></tr><tr><td><strong>TermoStatWebV3-nomulti</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td><strong>85.78%</strong></td><td>2.13%</td><td>4.15%</td></tr><tr><td>C2</td><td><strong>84.65%</strong></td><td>3.18%</td><td>6.14%</td></tr><tr><td>C3</td><td><strong>85.28%</strong></td><td>2.95%</td><td>5.69%</td></tr><tr><td>CA</td><td><strong>89.55%</strong></td><td>5.21%</td><td>9.84%</td></tr><tr><td><strong>TS3-multi</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>70.63%</td><td><strong>12.25%</strong></td><td><strong>20.88%</strong></td></tr><tr><td>C2</td><td>53.66%</td><td><strong>19.43%</strong></td><td><strong>28.53%</strong></td></tr><tr><td>C3</td><td>66.41%</td><td>18.89%</td><td>29.41%</td></tr><tr><td><strong>TS3-nomulti</strong></td><td></td><td></td><td></td></tr><tr><td>C1</td><td>82.84%</td><td>5.44%</td><td>10.21%</td></tr><tr><td>C2</td><td>77.32%</td><td>7.16%</td><td>13.10%</td></tr><tr><td>C3</td><td>82.46%</td><td>7.66%</td><td>14.01%</td></tr></tbody></table>

*TermoStatWeb*, *MIG-YaTeA* is an evolution of *YaTeA*.

Figure 4 presents the evolution of TPrecision, TRecall and TFMeasure between *TermoStatWeb* and *TS3-multi*. We observe improvements, mainly for TRecall and TFMeasure. Figure 5

<footnote n="9">⁹http://olst.ling.umontreal.ca/~drouin/termostat_web</footnote>
<table><caption>Table 2: Number of returned terms</caption><thead><tr><th></th><th>C1 (500k words)</th><th>C2 (1.5M words)</th><th>C3 (2.5M words)</th><th>CA (1.5M words)</th></tr></thead><tbody><tr><td>Acabit</td><td>25,519</td><td>65,200</td><td>89,883</td><td>123,468</td></tr><tr><td>Fastr</td><td>36,932</td><td>failed</td><td>failed</td><td>failed</td></tr><tr><td>Jv1</td><td>-</td><td>-</td><td>2,129</td><td>-</td></tr><tr><td>YaTeA</td><td>47,499</td><td>125,245</td><td>184,729</td><td>206,733</td></tr><tr><td>MIG-YaTeA</td><td>25,634</td><td>64,823</td><td>91,907</td><td>119,137</td></tr><tr><td>TermExtractor</td><td>412</td><td>782</td><td>924</td><td>-</td></tr><tr><td>TermoStatWeb</td><td>9,490</td><td>failed</td><td>failed</td><td>-</td></tr><tr><td>TermoStatWebV3-multi</td><td>5,544</td><td>10,146</td><td>8,026</td><td>13,699</td></tr><tr><td>TermoStatWebV3-nomulti</td><td>2,166</td><td>3,417</td><td>3,081</td><td>5,056</td></tr><tr><td>TS3-multi</td><td>25,789</td><td>70,009</td><td>59,467</td><td>-</td></tr><tr><td>TS3-nomulti</td><td>6,899</td><td>10,056</td><td>10,563</td><td>-</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Figure 4: Increase of TPrecision, TRecall and TFMeasure from TermoStatWeb to TS3-multi</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 5: Increase of TPrecision, decrease of TRecall and TFMeasure from TermoStatWeb to TermoStatWebV3-multi</figcaption></figure>

exhibits the evolution between TermoStatWeb and TermoStatWebV3-multi: in this case, improvements are more significant in terms of TPrecision.

Progression between TS3-multi and TermoStat-WebV3-multi is presented on Figure 6. Significant improvements can be observed in terms of TPrecision, mostly for C2. However, TRecall notably decreases because TermoStatWebV3 filters terms much more aggressively, focusing on precision.

The most significant improvement is obtained between YaTeA and MIG-YaTeA, as presented on Figure 7. TPrecision, TRecall and TFMeasure increase as the corpus grows.

## 6.3. Sensitivity to the corpus type

Regarding the sensitivity to the corpus type, results show that CA almost always provides better scores than C2 (although they both weight 1.5M words), both in term of TPrecision and in term of TRecall. The contrast between C2 and CA can be seen on Figure 8. These improvements can be explained because CA, composed of 7,030 abstracts where C2 contains only 106 patents, can potentially cover a larger field of knowledge.

<figure><img src="image_4.png" /><figcaption>Figure 6: Increase of TPrecision, decrease of TRecall and TFMeasure from TS3-multi to TermoStatWebV3-multi</figcaption></figure>

## 7. Conclusion

This paper presents the results of two consecutive initiatives on term extraction evaluation. The comparison of the results obtained during the two years was used to measure systems evolution. The protocol was easily reproduced for the second initiative. It provides inter-
<figure><img src="image_5.png" /><figcaption>Figure 7: Increase of TPrecision, TRecall and TFMeasure from YaTeA to MIG-YaTeA</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 8: Impact of the corpus type: Evolution of TPrecision, TRecall and TFMeasure from C2 to CA</figcaption></figure>

esting keys to better characterize term extractors in their design choices, their scalability regarding corpus size and their robustness with respect to corpus type.

As the evaluation organizer, LIPN set up an adjudication phase after 2011 initiative, in order to get more reliable precision results. We actually noticed that relevant terms are missing in the gold standard, which potentially penalizes the systems. Tested term extractors have provided a total of more than one hundred thousand of distinct terms. Some of them are found by at least two systems and are not already present in the gold standard.

Some experts from INIST-CNRS are asked to check whether these terms are relevant or not. To ease this process, LIPN has developed a web based validation interface, which allows experts to view terms in context (the sentences where they appear). A screenshot of this interface is presented on Figure 9. For each term, experts have to decide if it is GOOD, BAD or ?, a generic category for problematic terms. A free comment zone is available to indicate the correct form when needed.

These evaluations were also helpful for developers and for the gold standard provider. INIST-CNRS plans to exploit the results of the ongoing adjudication work to extend their thesaurus.

## 8. Acknowledgement

This work has been partially founded by OSEO under the Quæro program.

## 9. References

Sophie Aubin and Thierry Hamon. 2006. Improving term extraction with terminological resources. In Tapio Salakoski, Filip Ginter, Sampo Pyysalo, and Tapio Pahikkala, editors, *Advances in Natural Language Processing 5th International Conference on NLP, FinTAL 2006*, pages 380–387, Turku, Finland. Springer.

Béatrice Daille. 2003. Conceptual structuring through term variations. In F. Bond, A. Korhonen, D. MacCarthy, and A. Villacicencio, editors, *Proceedings of ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment*, pages 9–16.

Patrick Drouin. 2003. Term extraction using non-technical corpora as a point of leverage. *Terminology*, 9(1):99–115.

Patrick Drouin. 2006. Termhood experiments: quantifying the relevance of candidate terms. *Modern Approaches to Terminological Theories and Applications*, 36:375–391.

Chantal Enguehard. 2003. Correct : Démarche coopérative pour l’évaluation de systèmes de reconnaissance de termes. In *Actes de la conférence Traitement Automatique des Langues Naturelles (TALN) 2003*, pages 339–345, Batz-sur-Mer, France, Juin.

N. Kando, K. Kuriyama, T. Nozue, K. Eguchi, H. Kato, S. Hidaka, and J. Adachi. 1999. The NTCIR workshop: the first evaluation workshop on japanese text retrieval and cross-lingual information retrieval. In *Proceedings of the 4th International Workshop on Information Retrieval with Asian Languages (1RAL’99)*.

Widad Mustafa El Hadi, Ismail Timimi, Marianne Dabbadie, Khalid Choukri, Olivier Hamon, and YunChuang Chiao. 2006. Terminological resources acquisition tools: Toward a user-oriented evaluation model. In *Proc. of LREC’06*, pages 945–948, Genova, Italy, May.

Adeline Nazarenko, Haïfa Zargayouna, Olivier Hamon, and Jonathan Van Puymbrouck. 2009. Évaluation des outils terminologiques : enjeux, difficultés et propositions. *Traitement Automatique des Langues (TAL)*, 50(1 varia):257–281.
<figure><img src="image_7.png" /><figcaption>Figure 9: Screenshot of the validation interface</figcaption></figure>

F. Sclano and P. Velardi. 2007. Termextractor: a web application to learn the shared terminology of emergent web communities. *Enterprise Interoperability II*, pages 287–290.

Haïfa Zargayouna and Adeline Nazarenko. 2010. Evaluation of Textual Knowledge Acquisition Tools: a Challenging Task. In *Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010)*, pages 435–440, Valletta, Malte.