Methods and Practical Issues in Evaluating Alignment Techniques

This paper describes the work achieved in the first half of a 4-year cooperative research project (ARCADE), financed by AUPELF-UREF. The project is devoted to the evaluation of parallel text alignment techniques. In its first period ARCADE ran a competit ion between six systems on a sentence-to-sentence alignment task which yielded two main types of results. First, a large reference bilingual corpus comprising of texts of different genres was created, each presenting various degrees of difficulty with respect to the aligmnent task. Second, significant methodological progress was made both on the evaluation protocols and metrics, and the algorithms used by the different systems. For the second phase, which is now underway, ARCADE has been opened to a larger number of teams who will tackle the problem of word-level alignment. 1 I n t r o d u c t i o n In the last few years, there has been a growing interest in parallel text alignment techniques. These techniques a t tempt to map various textual units to their translation and have proven useful for a wide range of applications and tools. A simple example of such a tool is probably the TrunsSearch bilingual concordancing system (Isabelle et al., 1993), which allows a user to query a large archive of existing translations in order to find ready-made solutions to specific translation problems. Such a tool has proved extremely useful not only for translators, but also fbr bilingual lexicographers (Langlois, 1996) and terminologists (Dagan and Church, 1994). More sophisticated applications based on alignment technology have also been the object of recent work, such as the automatic building of bilingual lexical resources (Melamed, 1996; Klavans azld Tzoukermann, 1995), the automatic verification of translations (Macklovitch, 1995), the automatic dictation of translations (Brousseau et al., 1995) and even interactive machine translation (Foster et al., 1997). Enthusiasm for this relatively new field was sparked early on by the apparent demonstration that very simple techniques could yield almost perfect results. For instance, to produce sentence alignments, Brown et al. (1991) and Gale and Church (1991) both proposed methods that completely ignored the lexical content of the texts and both reported accuracy levels exceeding 98%. Unfortunately performance tends to deteriorate significantly when aligners are applied to corpora which are widely different fi'om the training corpus, and/or where the alignments are not straightforward. For instance graphics, tables, "floating" notes and missing segments, which are very common in real texts, all result in a dramatic loss of efficiency. The truth is that, while text alignment is mostly an easy problem, especially when considered at the sentence level, there are situations where even humans have a hard time making the right decision. In fact, it could be argued that, ultimately, text alignment is no easier tha~l the more general problem of natural laalguage understanding. In addition, most research efforts were directed towards the easiest problem, that of sentence-to-sentence alignment (Brown et al., 1991; Gale and Church, 1991; Debili, 1992; Kay and RSscheisen, 1993; Simard et al., 1992; Simard and Plamondon, 1996). Alignment at the word and term level, which is extremely useful for applications such as lexical resource extraction, is still a largely unexplored research area(Melamed, 1997). In order to live up to the expectations of the