Tatsuya Ishisaka† Kazuhide Yamamoto† † Nagaoka University of Technology 1603-1 Kamitomiokamachi, Nagaoka, Niigata 940-2188, Japan {ishisaka,ykaz}@nlp.nagaokaut.ac.jp
Abstract
To address the shortage of Japanese-English parallel corpora, we developed a parallel cor- pus by collecting open source software man- uals from the Web.  The constructed cor- pus contains approximately 500 thousand sen- tence pairs that were aligned automatically by an existing method. We also conducted statis- tical machine translation (SMT) experiments with the corpus and confirmed that the corpus is useful for SMT.
Introduction
1
Multilingual parallel corpora are required to sup- port many tasks in natural language processing. For example, statistical machine translation (SMT) re- quires a parallel corpus for training, and cross- lingual processing such as information retrieval and information extraction also use parallel corpora. There is no doubt on the importance of parallel cor- pora for any language pair. Specially, Japanese-English parallel corpora are Although some parallel corpora very scarce. (Utiyama and Isahara, 2007) are available, the do- mains and sizes of these corpora are limited. In general, European countries use multiple lan- guages officially.  Based on this multilingual envi- ronment, Koehn (2005) has built a corpus by col- lecting parallel texts in eleven languages from the proceedings of the European Parliament, which are published on the Web. However, some countries such as Japan have no such language situation, that leads us difficulties for Masao Utiyama†† Eiichiro Sumita†† †† MASTAR Project National Institute of Information and Communications Tehnology 3-5, Hikaridai, Seika, Soraku, Kyoto 619-0289, Japan {mutiyama,eiichiro.sumita} @nict.go.jp creating parallel corpora.  Hence, more efforts are needed to collect them effectively. Available Japanese-English parallel corpora are scarce. However, there are a lot of translated texts on the Web. Specially, open source manuals are trans- lated into Japanese from English by volunteer trans- lators. We collected such English and Japanese texts. Then, the sentences in collected texts were automat- ically aligned, resulting in a parallel corpus made from open source software manuals.  Manuals of open source software has been used for making a parallel corpus named OPUS (Tiedemann and Ny- gaard 2004), which was made from OpenOffice.org documentation 1 , KDE manuals including KDE mes- sages 2 , and PHP manuals 3 . However, the Japanese- English part of OPUS is not large.  In contrast, we collected about 500 thousand sentence pairs. In ad- dition, our work involved extensive human efforts to ensure the quality of our parallel corpus. The original and translated texts often proscribe copy, distribute, display, and make derivative works. Our target texts are open source software manuals. Such open source software manuals are often pub- lished under open licenses under which we can mod- ify and distribute them. The translation quality of open source software manuals are considered to be relatively high, be- cause they are translated by many translators who belong to the projects and drafts of the translations are corrected by other project members.  Therefore, we can trust the quality of software manuals. In the following we present how we collect, clean, and align software manuals.  We also illustrate per- formance of SMT experiments using the corpus.
2
Target license
We will publish a parallel corpus constructed from open source software manuals.  This action is con- sidered as a redistribution with modifications.  We therefore target licenses that allow redistribution and modifications. Here are four example licenses that allow redistri- bution and modification. MIT License 4 The MIT License is very open. It is necessary only to include the copyright notice and the permission notice. FreeBSD Documentation License 5 The FreeBSD Documentation License is similar to the MIT License. It is as follows: Redistribution and use in source (SGML Doc- Book) and 'compiled' forms (SGML, HTML, PDF, PostScript, RTF and so forth) with or without modification, are permitted provided that the following conditions are met: (1) Redistributions of source code (SGML DocBook) must retain the their copy- right notice, this list of conditions and the their disclaimer as the first lines of this file unmodified. (2) Redistributions in compiled form (trans- formed to other DTDs, converted to PDF, PostScript, RTF and other formats) must reproduce the their copyright no- tice, this list of conditions and the their disclaimer in the documentation and/or other materials provided with the distri- bution. Creative Commons licenses 6 Creative Com- mons licenses include several license types. We in- troduce the "Attribution-Share Alike 3.0 Unported" model in which we can copy, distribute, transmit, and adapt the work under the following conditions; Attribution You must attribute the work in the manner specified by the author or li- censor (but not in any way that suggests that they endorse you or your use of the work). Share Alike  If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, sim- ilar or a compatible license.
Common Development and Distribution Li-
cense 7 Common Development and Distribution License (CDDL) is a long and detailed license.  Thus, we describe CDDL briefly using the FAQ of NetBeans 8 . Modify NetBeans source code and redistribute it for free (or for sale) as long as I follow the terms of the CDDL, including the following provisions: ● We must bundle the CDDL with any source code version we distribute, ● We must make my changes to the Net- Beans source code (but not new source files I create) available to the NetBeans community under the CDDL (so the community can benefit from my changes or improvements), ● We cannot modify the rights granted un- der the CDDL License, ● We can add external files to NetBeans, compile these and redistribute them for free or for sale and we do not need to make such external files or changes to them available in source code form or bi- nary form to the NetBeans project. If my value-add is worth the price, we can sell it.
3
Characteristics of manuals
The software manuals are difficult to handle with. Japanese open source software manuals usually contain both Japanese and English texts.  Conse- quently, some parts of manuals are not needed for making parallel corpora. For example, when manu- als explain commands, the commands are not trans- lated into Japanese.  Further, open source software manuals often include program source codes, which are not translated. Software is periodically updated, primarily by adding new features and functionality.  Thus, new sentences are typically added to the corresponding manuals.  As a result, the latest original document version may be newer than the translated document version.  Therefore, we have to collect original and translated documents with matching versions.  This needs human efforts. In many cases, translated open source software manuals are HTML files with HTML tags that con- form to the original document’s format, but this need not be so.  Therefore, we have to modify translated documents to match the original format.  In addi- tion, the formats of open source manuals differ from project to project. Consequently, we have to write a script to extract text portions for each project.
4
Constructing the corpus
Constructing the Japanese-English corpus takes three steps; (1) searching for open source software manuals on the Web (2) cleaning up documents, and (3) aligning sentences. We describe these three steps in the following sub- sections.
4.1
Searching for open source software
manuals
We used Web search engines manually to search for open source software manuals.  We searched for Japanese Web pages containing phrases such as翻訳プロジェクト(translation project).  Then, we manually checked if those pages contained soft- ware manuals.  If they had, we downloaded manu- als.  We also searched for the corresponding manu- als in English.  When downloading documents, we checked and matched the versions of both English and Japanese documents. Table 1 show list of collected manuals and their URLs.  In the table, JF represents "Linux Japanese FAQ Project."  JF translates documents related with Linux. JM means “JM project.” JM translates Linux manual pages.  RFC represents "Request for Com- ments."  Note that RFCs are not manuals, but their contents are similar to manuals and their use and importance are widespread. Others are open source software manuals.
Cleaning up documents
4.2
Software manuals contain HTML tags. We normal- ize documents by deleting HTML tags with a Perl script using pattern matching. This script is tailored to each software manual. Sentences in software manuals are often broken by newlines.  It is difficult to judge whether a new- line character represents a sentence end or not.  For example, headings are usually separated by new- lines without periods.  We delete newline charac- ters in a paragraph, which is defined by a text region separated by empty lines, if that paragraph contains punctuation.  Otherwise, newlines are not deleted be- cause they are regarded as sentence ends.
4.3  Aligning sentences
We use Utiyama and Isahara's alignment method, because their method has been successfully used in aligning noisy Japanese-English parallel texts (Utiyama and Isahara, 2007).  Below is a concise description of their algorithm. We begin by obtaining the maximum similarity sentence alignments. Let J and E be a Japanese text file and an English text file, respectively.  We cal- culate the maximum similarity sentence alignments (J 1 ,E 1 ), (J 2 ,E 2 ), . . . , (J m ,E m ), using a dynamic programming matching method (Gale and Church, 1993), where ( J i , E i ) is a Japanese and English sen- tence alignment pair in J and E. We allow 1-to- n, n-to-1 (0 ≤ n ≤ 5), or 2-to-2 alignments when aligning sentences.  The similarity between J i and E is calculated based on word overlap (i.e., num- ber of word pairs from J i and E i that are transla- tions of each other based on a bilingual dictionary with 450,000+ entries).  The similarity between a Japanese document, J, and an English document, E, (noted AVSIM(J,E)) is calculated using:
A high AVSIM( J,E ) value occurs when the sen- tence alignments in J and E take on high similarity values. We also calculate the ratio of the number of
sentences between J and E (noted R( J,E )) using:
where | J | is the number of sentences in J , and | E | is the number of sentences in E . A high R(J,E) value occurs when |J| ～ |E|. Consequently, R(J, E) can be used to measure the proportion of potentially corresponding sentences. Using AVSIM( J,E ) and R( J,E ), we defined the similarity between J and E (noted AR(J,E)) as
Finally, we define the score of alignment J i and E i as
A high Score( J i , E i ) value occurs in the follow- ing case:  (1) sentences J i and E i are similar, (2) documents .J and E are similar, and (3) the number of sentences | J | and | E | are similar.  Score( J i , E i ) combines both sentence and document similarities to discriminate between correct and incorrect align- ments.
Results of sentence alignment
4.4
We examined the results of the sentence alignment and concluded that 1-to-1, 1-to-2, or 2-to-1 sentence alignments are clean. Thus, we extracted only these sentence alignments to make our parallel corpus. Although we included all 1-to-1, 1-to-2, or 2-to-1 alignments, it is possible to extract only highly pre- cise sentence alignments if we use the score defined in Equation 4, as verified in (Utiyama and Isahara, 2007). Table 2 shows the number of aligned sentences. Overall, there are a total of just under 500 thou- sand sentences. Among these, over 90% of sentence alignments are 1-to-1. Table 3 shows examples of aligned sentences. As the examples show, some Japanese sentences in- clude English words, such as “PUT” or “root”.  We also see that both long and short sentences are in- cluded. We found that over 80% of sentence alignments were precisely aligned.  We think that further im- provements are possible, since we have failed to clean up some noisy sentences.  Our simple pattern match rules did not work well for removing some sentences such as notes by translators.  We expect that the alignment accuracy would improve if we re- move such noisy sentences.
5
MT Experiments
MT experiments were conducted to verify the use- fulness of our constructed corpus for SMT. We used the Moses system (Koehn et al., 2007).  We used GIZA++ (Och and Ney ,2003) for word alignment and SRILM (Stolcke,2002) for language modeling. In our experiments, we used 5-gram language mod- els.  Minimum error rate training (MERT) was per-
formed to tune the decoder's parameters on the basis of the bilingual evaluation understudy (BLEU) score (Papinei et al., 2002). The evaluation was done us- ing a single reference.  Tuning was performed us- ing the standard technique developed by Och (Och, 2003). The test and development data were extracted from the aligned JF sentences. Each of test and de- velopment data consists of 500 sentences. In the following experiments, we simulated a sit- uation where an SMT system was applied to help volunteer translators translate English JF documents into Japanese. We want to use all parallel sentences efficiently to help translators.  This is a problem of domain adaptation.  All of paralell sentences were translated from English to Japanese.  Therefore we did MT experiments from English. In the first experiment, we used all parallel sen- tences (excluding development and test sentences) as our training data, which contained approximately 500 thousand parallel sentences, as shown in Ta- ble 2. The BLEU score obtained was 37.38. In the second experiment, we used only JF par- allel sentences (approximately 100 thousand sen- tences). The BLEU score obtained was 40.02. In the third experiment, we linearly interpolated language models of the first and second experiments. We changed the weight of JF's language model from 0.1, 0.3, 0.5, 0.7, and 0.9.  The BLEU scores were 38.40, 39.30, 38.92, 40.07, and 42.53.  The BLEU score was highest when the weight is 0.9. The trans- lation model used was that in the first experiment. In the fourth experiment, we log-linearly interpo- lated translation models of the first and second ex- periments.  The weights were set with MERT. The BLEU score was 41.26. The language model used was that in the first experiment. In the final experiment, we used the language model with a weight of 0.9 in the third experiment and the translation model in the fourth experiment. The BLEU score was 44.36, which was the highest in these experiments.
6
Discussion
The BLEU scores obtained were relatively high for a Japanese-English corpus.  For example, Utiyama and Isahara (2007) reported a maximum BLEU score of 25.89 for patent document SMT. However, we have to be careful about our ex- perimental results.  First, our test sentences were extracted from those having the highest alignment scores, which might not be representative samples. This was because sentences with low alignment scores could be wrong alignments, which were not suitable for measuring SMT performance.  We plan to sample test sentences from the whole corpus and clean them for the purpose of evaluation in our fu- ture work.  Second, our Japanese word segmenter segmented ASCII words into characters.  (Japanese words were segmented properly.)  For example, "word" was segmented into "w o r d" in the cor- pus used in the above experiments.  Consequently, the BLEU scores obtained were optimistic, though the occurrences of ASCII words were much smaller than those of Japanese words. Because the BLEU scores were rather optimistic, we manually examined test sentences.  We found that short sentences were generally translated well and longer sentences were not translated well. Overall, we concluded that our parallel corpus is useful for English-Japanese SMT developments. We also hope that this corpus will be useful for support- ing human translations of these manuals.
7
Conclusion
We have reported a project on developing a Japanese-English parallel corpus made from soft- ware manuals.  It has approximately 500 thousand sentence pairs. The corpus will be available at http://www2.nict.go.jp/x/x161/members/mutiyama/ manual/index.html