An Experiment in Hybrid Dictionary and Statistical Sentence Alignment

The task of aligning sentences in parallel corpora of two languages has been well studied using pure statistical or linguistic models. We developed a linguistic method based on lexical snatching with a bilingual dictionary and two statistical methods based on sentence length ratios and sentence offset probabilities. This paper seeks to further our knowledge of the alignment task by comparing the performance of the alignment models when used separately and together, i.e. as a hybrid system. Our results show that for our English-Japanese corpus of newspaper articles, the hybrid system using lexical matching and sentence length ratios outperforms the pure methods. 1 I n t r o d u c t i o n There have been many approaches proposed to solve the problem of aligning corresponding sentences in parallel corpora. With a few notable exceptions however, much of this work has focussed on either corpora containing European language pairs or clean-parallel corpora where there is little reformatting. In our work we have focussed on developing a method for robust matching of English-Japanese sentences, based primarily on lexical snatching. The method combines statistical information from byte length ratios. We show in this paper that this hybrid model is more effective than its constituent parts used separately. The task of sentence alignment is a critical first step in many automatic applications involving the analysis of bilingual texts such as extraction of bilingual vocabulary, extraction of translation templates, word sense disambiguation, word and phrase alignment, and extraction of parameters for statistical translation models. Many software products which aid human translators now contain sentence alignment tools as an aid to speeding up editing and terminology searching. Various methods have been developed for sentence alignment which we can categorise as either lexical such as (Chen, 1993), based on a large-scale bilingual lexicon; statistical such as (Brown et al., 1991) (Church, 1993)(Gale and Church, 1993)(Kay and Rhshcheisen, 1993), based on distributional regularities of words or byte-lengdl ratios and possibly inducing a bilingual lexicon as a by-product, or hybrid such as (Vtsuro et al., 1994) (Wu, 1994), based on some combination of the other two. Neither of the pure approaches is entirely satisfactory for the following reasons: • Text volume limits the usefulness of statistical approaches. We would often like to be able to align small amounts of text, or texts from various domains which do not share the same statistical properties. • Bilingual dictionary coverage limitations mean that we will often encounter problems establishing a correspondence in non-general domains. • Dictionary-based approaches are founded on an assumption of lexictd correspondence between language pairs. We cannot always rely on this for non-cognate language pairs, such as English and Japanese. • Texts are often heavily reformatted in translation, so we cannot assume that the corpus will be clean, i.e. contain many one-to-one sentence mappings. In this case statistical methods which rely on structure correspondence such as byte-length ratios may not perform well. These factors suggest that some hybrid method may give us the best combination of coverage and accuracy when we have a variety of text domains, text sizes and language pairs. In this paper we seek to fill a gap in our under. ~tanding and to show how the various components of the hybrid method influence the quality of sentem:e alignment for Japanese and English newspaper articles. 2 B i l i n g u a l S e n t e n c e A l i g n m e n t The task of sentence alignment is to match corresponding sentences in a text from one language to sentences in a translation of that text in another language. Of particular interest to us is the application to Asian language pairs. Previous studies such as (Fung and V<a, 1994) have commented