Language Resources as by-Product of Evaluation: The MULTITAG Example

In this paper, we show how the paradigm of evaluation can function as language resource producer for high quality and low cost validated language resources. First the paradigm of evaluation is presented, the main points of its history are recalled, from the first deployment that took place in the USA during the DARPA/NIST evaluation campaigns, up to latest efforts in Europe (SENSEVAL2/ROMANSEVAL2, CLEF, CLASS etc.). Then the principle behind the method used to produce high-quality validated language at low cost from the byproducts of an evaluation campaign is exposed. It was inspired by the experiments (Recognizer Output Voting Error Recognition) performed during speech recognition evaluation campaigns in the USA and consists of combining the outputs of the participating systems with a simple voting strategy to obtain higher performance results. Here we make a link with the existing strategies for system combination studied in machine learning. As an illustration we describe how the MULTITAG project funded by CNRS has built from the by-products of the GRACE evaluation campaign (French Part-Of-Speech tagging system evaluation campaign) a corpus of around 1 million words, annotated with a fine grained tagset derived from the EAGLES and MULTEXT projects. A brief presentation of the state of the art in Part-Of-Speech (POS) tagging and of the problem posed by its evaluation is given at the beginning, then the corpus itself is presented along with the procedure used to produce and validate it. In particular, the cost reduction brought by using this method instead of more classical methods is presented and its generalization to other control task is discussed in the conclusion. 1. The paradigm of Evaluation Comparative evaluation in language engineering has been used as a basic paradigm in the USA DARPA program on human language technology since 1984. Activities similar in kind, have been pursued in Europe, both at national and at European level, but on a smaller scale and over a limited time (Mariani and Paroubek, 1999). The latest efforts concerning evaluation in Europe are CLEF (Cross Language Text Retrieval System Evaluation in collaboration with NIST and TREC conference), SENSEVAL2/ROMANSEVAL-2 (Kilgarriff, 1998) and CLASS (evaluation across FP5 project clusters). Comparative evaluation is a paradigm in which a set of participants compare the results of their systems using the same or similar control tasks (Bernsen et al., 1999) and related data with metrics that are agreed upon. More precisely, Comparative evaluation consists in (1) choosing or creating a control task, (2) in gathering system or component developers and integrators who are interested in testing their systems against those of others, (3) in organizing an evaluation campaign which necessarily involves distributing linguistic data for training and testing the systems, and (4) in defining the protocol and the metrics which will be used in the results assessment. A control task is the function that the participating systems have to perform during an evaluation together with the conditions under which this function must be performed (e.g. for parser evaluation, a control task can be the bracketing of the constituents). Every deployment of the paradigm of evaluation in the field of Language Engineering entails the production of linguistic data: the organizers build the reference and test data sets and the participants apply their systems on test data to produce evaluation data. When a quantitative black-box (Sparck-Jones and Galliers, 1995) evaluation methodology is applied the data produced is generally abundant and could easily be re-used as training material if the cost of filtering out the errors was not so high. 2. Combining to Improve In machine learning, it is well known that ensemble methods or committees of learning machines can often improve the performance of a system in comparison to a single learning machine. A very promising algorithm based on this principle now under investigation is Ada boost (Schwenk, 1999). In the same field, people have applied for a long time “winner take all” strategies to combine, inside the same system, the output of several basic processing units (Simpson, 1990). In the course of its evaluation program on speech recognition (S., 1998), NIST developed the ROVER (Recognizer Output Voting Voting Error Reduction) (Fiscus, 1997), to produce a composite Automatic Speech Recognition system output from the output of several ASR systems. Such composite system has an error rate inferior to the one of any of its components. In the ROVER, the output of several ASR systems is first combined into a single transition network using a modified version of the dynamic programming alignment technique used by NIST to score ASR systems1. This network is then explored and a simple voting strategy (highest number of votes) is used to select the best scoring word at each decision point. In (Fiscus, 1997), NIST reports a incremental 5.6% Word Error Rate reduction (12.5% relative) using voting by frequency of occurrence and maximum confidence (the output of the ASR systems was annotated with confidence measures (Chase, 1997)). Concerning, POS tagging, the principle of combination has been used in the past by (Marquez and Padro, 1998) who combines two taggers to annotate a corpus and by (Tufis, 1999) who uses several versions of same tagger but trained on different data. The SCLITE tool is freely available from http://www.itl. nist.gov/iaui/894.01/software.htm 3. The GRACE POS tagging evaluation campaign and its data. GRACE(Adda et al., 1999) was the first large scale evaluation campaign for Part-Of-Speech tagging for the French language. It was part of the French program CCIIL (Cognition, Intelligent Communication and Language Engineering), jointlypromoted by the Engineering Sciences and Human Sciences departments of the CNRS. The call for tenders was published in November 1995 and the first year has been devoted to bootstrapping the program by defining and installing the different organization committees. From the participants point of view, GRACE was made of 3 phases: training, dry-run and test. The first one was used by the participant to calibrate their systems on untagged data, the two others were complete runs of the evaluation protocol were the participants had to tag a large amount of text and to provide a mapping between their tagset and the reference tagset which had been derived with their collaboration from the EAGLES format (Leech and Wilson, 1995). The training corpus was distributed globally to all the participants in January 1996, while the dry run corpus was distributed individually to each participant in an encrypted form during the fall of 1996. The results were discussed during a workshop restricted to the participants, a satellite event to the Journees Scientifiques et Techniques du Reseau FRANCIL, in April 1997 (Adda. et al., 1997). The test corpus was distributed in the same manner as for the dry run, at the end of December 1997. The preliminary results of the tests were discussed with the participants in a workshop in May 1998. The final results were disclosed on the WEB2 during fall of 1998 as soon as they had been validated by the organizers (cross validation with two different processing chains based on different algorithms and developed at two different sites) and the participants. At the beginning there were 18 participants from 5 different countries (CA, USA, D, CH, FR), from both public research and industry, and 3 evaluators EPFL, INaLF-CNRS and Limsi-CNRS.The 2 corpus providers were Limsi and INaLF. Out of the 21 initial participants, 17 only took part in the dry run and only 13 completed the tests. The size of the training corpus was around 10 million words of untagged texts, evenly distributed between literary works and newspaper articles. For the dry run, the participants tagged a corpus of roughly 450,000 words with a similar genre distributionand the performance measure was computed over 20,000 words to which a reference description had been manually assigned. For the tests, the participants had to mark a corpus of 650,000 words and the measure was taken over 40,000 words. GRACE used the quantitative black box metrics: Decision and Precision, which were derived especially for GRACE from the metrics used in Information Retrieval (Precision and Recall). Precision measures the ability of a POS tagger to assign a correct tag to a given word form, and Decision measures the capacity of a POS tagger to restrict for a given word form the number of candidate tags with respect to a given tagset. One of the lessons to draw from the GRACE experience, is that ideally, results should be cross-validated with two different processing chains, based on different algorithms http://www.limsi. fr/TLP/grace (when this is possible) and developed at two different sites in order to ensure their accuracy and quality. The evaluation toolkit of GRACE has been packaged as a demonstration by the ELSE project and is freely available3. GRACE proved to be a success; its results are: a better knowledge of the existing systems in each domain and of their state of development; precise evaluation metrics defined in collaboration with the participants; an evaluation toolkit freely available, a new product on the market (one participant decided to add a tagger to his catalogue as a result of his participation); the creation of a community of actors interested in evaluation; and last of all, the initial data to build the new linguistic resource described here.