Masao Utiyama and Hitoshi Isahara
National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun,Kyoto, 619-0289 Japan E-mail: {mutiyama,isahara}@nict.go.jp
Abstract
We describe a Japanese-English patent parallel corpus created from the Japanese and US patent data provided for the NTCIR-6 patent retrieval task. The corpus contains about 2 million sentence pairs that were aligned automatically. This is the largest Japanese-English parallel corpus, which will be available to the public after the 7th NTCIR workshop meeting.  We estimated that about 97% of the sentence pairs were correct alignments and about 90% of the alignments were adequate translations whose English sentences reflected almost perfectly the contents of the corresponding Japanese sentences.
1.
Introduction
The rapid and steady progress in corpus-based machine translation (MT) (Nagao, 1981; Brown et al., 1993) has been supported by large parallel corpora, such as the Arabic-English and Chinese-English parallel corpora dis- tributed by the Linguistic Data Consortium (Ma and Cieri, 2006) and the Europarl corpus (Koehn, 2005) consisting of 11 European languages. However, large parallel corpora do not exist for many language pairs. Much work has been undertaken to overcome this lack of parallel corpora.  For example, Resnik and Smith (2003) have proposed mining the web to collect parallel corpora Munteanu and Marcu for low-density language pairs. (2005) have extracted parallel sentences from large Chi- nese, Arabic, and English non-parallel newspaper cor- pora. Utiyama and Isahara (2003) have extracted Japanese- English parallel sentences from a noisy-parallel corpus. We have recently aligned Japanese and English sentences in Japanese and US patent data provided for the NTCIR-6 patent retrieval task (Fujii et al., 2007).  We used Utiyama and Isahara's method to extract clean sentence alignments. The number of extracted sentence alignments was about 2 million.  These sentence pairs and all alignment data that were produced during the alignment procedure are planed to be used in the NTCIR-7 patent MT task. 1  This is the largest Japanese-English parallel corpus, which will be available to the public after the 7th NTCIR workshop meeting. In Section 2., we describe the resources used to develop the patent parallel corpus. In Sections 3., 4., and 5., we describe the alignment procedure, the basic statistics of the patent parallel corpus, and the MT experiments conducted on the patent corpus.
2.
Resources
Our patent parallel corpus was constructed from the patent data provided for the NTCIR-6 patent retrieval task.  The patent data consists of 1993-2002
● USPTO patent data published in 1993-2000.
The Japanese patent data consists of about 3.5 million doc- uments, and the English data consists of about 1 million documents. We identified 84,677 USPTO patents that originated from Japanese patents by using the priority information de- scribed in the USPTO patents. 2  We examined these 84,677 Japanese and English patent pairs and found that the "De- tailed Description of the Preferred Embodiments" part ( em- bodiment part for short) and the "Background of the Inven- tion" part (background part for short) of each application tend to be literal translations of each other. We, thus, de- cided to use these parts to construct our patent parallel cor- pus. We used simple pattern matching programs to extract the embodiment and background parts from the whole docu- ment pairs and obtained 77,014 embodiment part pairs and 72,589 background part pairs. We then applied the align- ment procedure described in Section 3.  to these 149,603 pairs. We call these embodiment and background parts doc- uments.
3.
Alignment procedure
Score of sentence alignment
3.1.
We used Utiyama and Isahara's method (Utiyama and Isa- hara, 2003) to score sentence alignments. We first aligned sentences 3 in each document by using a standard DP match- ing method (Gale and Church, 1993; Utsuro et al., 1994). We allowed 1-to-n, n-to-1 (0 < n < 5), or 2-to-2 align- ments when aligning the sentences.  A concise descrip- tion of the algorithm used is described elsewhere (Utsuro et al., 1994). 4  Here, we only discuss the similarities between Japanese and English sentences used to calculate scores of sentence alignments. Let J i and E i be the word tokens of the Japanese and En- glish sentences for i -th alignment. The similarity between J i and E is:5
|J i | = no. of Japanese word tokens in i -th alignment | E i | = no. of English word tokens in i-th alignment δ( j, e ) = 1if j and e can be a translation pair otherwise 0 deg(j) = ∑ c ∈ e i δ(j,e) deg(e) = ∑ j ∈ j i δ(j,e) J i and E i were obtained as follows: We used ChaSen 6 to morphologically analyze the Japanese sentences and extract content words, which consisted of J i .  We used a max- imum entropy tagger 7 to POS-tag the English sentences and extract content words.  We also used WordNet's li- brary 8 to obtain lemmas of the words, which consisted of Ei. To calculate δ( j, e ), we looked up an English- Japanese dictionary that was created by combining en- tries from the EDR Japanese-English bilingual dictionary, the EDR English-Japanese bilingual dictionary, the EDR Japanese-English bilingual dictionary of technical terms, and the EDR English-Japanese bilingual dictionary of tech- nical terms. 9  The combined dictionary had over 450,000 entries. After obtaining the maximum similarity sentence align- ments using DP matching, we calculated the similarity be- tween a Japanese document, J , and an English document, E , (AVSIM( J, E )), as defined by (Utiyama and Isahara, 2003), using:
where ( J 1 , E 1 ), ( J 2 , E 2 ), . . . ( J m , E m )  are  the  sentence alignments obtained using DP matching. A high AVSIM( J, E ) value occurs when the sentence alignments in J and E take high similarity values. Thus, AVSIM( J, E ) measures the similarity between J and E . We also calculated the ratio of the number of sentences be- tween J and E ( R ( J, E )) using:
where | J | is the number of sentences in J , and | E | is the number of sentences in E .  A high R ( J, E ) value occurs when | J | ～ | E | . Consequently, R ( J, E ) can be used to measure the literalness of translation between J and E in terms of the ratio of the number of sentences. Finally, we defined the score of alignment J i and E i as
A high Score( J i , E i ) value occurs when ●  sentences J i and E i are similar ●  documents J and E are similar ● numbers of sentences | J | and | E | are similar Score( J i , E i ) combines both sentence and document simi- larities to discriminate between correct and incorrect align- ments.
3.2. Noise reduction in sentence alignments
We used the following procedure to reduce noise in the sentence alignments obtained by using the previously de- scribed aligning method on the 149,603 document pairs. The number of sentence alignments obtained was about 7 million. From these alignments, we extracted only one-to- one sentence alignments because this type of alignment is the most important category for sentence alignment.  As a result, about 4.2 million one-to-one sentence alignments were extracted. We sorted these alignments in decreasing order of scores and removed alignments whose Japanese sentences did not end with periods to reduce alignment pairs considered as noise. We also removed all but one of the identical alignments. Two individual alignments were determined to be identical if they had the same Japanese and English sentences. Consequently, the number of align- ments obtained was about 3.9 million. We examined 20 sentence alignments ranked between 1,999,981 and 2,000,000 from the 3.9 million alignments to determine if they were accurate enough to be included in a parallel corpus.  We found that 17 of the 20 alignments were almost literal translations of each other and 2 of the 20 alignments had more than 50% overlap in their contents. We also examined 20 sentence alignments ranked between 2,499,981 and 2,500,000 and found that 13 of the 20 align- ments were almost literal translations of each other and 6 of the 20 alignments had more than 50% overlap. Based on these observations, we decided to extract the top 2 million one-to-one sentence alignments. Finally, we removed some sentence pairs from this top 2 million alignments that were too long (more than 100 words in either sentence) or too imbalanced ( length of longer sentence  > 5). The number of sentence alignments thus obtained was 1,988,732.  We call these 1,988,732 sentence alignments the ALL data set (ALL for short) in this paper. We also asked a translation agency to check the validity of 1000 sentence alignments randomly extracted from ALL. The translation agency conducted a two-step procedure for verification. In the first step, they marked a sentence align- ment as:
Table 2: Number of sentence alignments ● A if the Japanese and English sentences matched as a whole. ● B if these sentences had more than 50% overlap in
to check if the alignment was correct. The number of align- ments marked as A was 973, B was 24 and C was 3. In the second step, they marked an alignment as: ● A if the English sentence reflected almost perfectly the
●  B if about 80% of the contents were shared. ● C if less than 80% of the contents were shared.
to check if the alignment was an adequate translation pair. The number of alignments marked as A was 899, B was 72, C was 26, and X was 3.  Based on these evaluations, we concluded that the sentence alignments in ALL are useful for training and testing MT systems. In the next section, we describe the basic statistics of this patent parallel corpus.  In Section 5., we describe the MT experiments conducted on ALL.
4.
Statistics of the patent parallel corpus
4.1.
Comparison of ALL and source data sets
We compared the statistics of ALL with those of the source patents and sentences from which ALL was extracted to see how ALL represented the sources. To achieve this, we used the primary international patent classification (IPC) code assigned to each USPTO patent. The IPC consists of eight sections, ranging from A to H. We only used sections G (Physics), H (Electricity) and B (Per- forming operations; Transporting). We categorized patents as O (Other) if they were not covered by these three sec- tions.
Table 3: Number of patents
As described in Section 2., 84,677 patent pairs were ex- tracted from the original patent data.  These patents were classified into G, H, B or O, as listed in the "Source" col- umn of Table 1. We counted the number of patents included in each section of ALL. We regarded a patent to be included in ALL when some sentence pairs in that patent were in- cluded in ALL. The number of such patents are listed in the “ALL” column of Table 1. Table 1 shows that about 60% ( 51059 × 100) of the source patent pairs were included in ALL. It also shows that the distributions of patents with respect to the IPC code were similar between ALL and Source. Table 2 lists the numbers of one-to-one sentence alignments in ALL and Source, where Source means the about 4.2 million one-to-one sentence alignments described in Sec- tion 3.2.  The results in this table show that about 47.6 % ( 1988732 × 100) sentence alignments were included in ALL. The results also show that the distribution of the sentence alignments are similar between ALL and Source. Based on these observations, we concluded that ALL rep- resented Source well. In the following, we use "G,"H,"B",and "O" to denote the data in ALL whose IPC were G, H, B, and O, respectively.
Basic statistics
4.2.
We measured the basic statistics of G, H, B, O, and ALL. We first randomly divided patents from each of G, H, B and O into training (TRAIN), development (DEV), devel- opment test (DEVTEST), and test (TEST) data sets.  One unit of the sampling was a single patent.  That is, G, H, B and O consisted of 19340, 16145, 7287, and 8287 patents (See Table 1), and the patents from each group were di- vided into TRAIN, DEV, DEVTEST, and TEST. We as- signed 91% of the patents to TRAIN, and 3% of the patents to DEV, DEVTEST, and TEST. We merged the TRAIN, DEV, DEVTEST and TEST of G, H, B, and O to create those of ALL. Table 3 lists the number of patents in these data sets and Table 4 lists the number of sentence align- ments. We then counted the number of types (distinct words) and
Table 5: Number of types (English)
tokens (running words) in these datasets. Tables 5 and 6 list the number of types for English and Japanese sen- tences.  Tables 7 and 8 list the number of tokens.  To count these numbers, we used ChaSen to segment Japanese sen- tences into tokens and used a simple tokenizer 10 to tok- enize English sentences.  All English words were lower- cased.  In these tables, the figures in columns "TRAIN", “DEV”, “DEVTEST” and “TEST” are the number of types and tokens in these datasets and the figures in the "Whole" columns are the number of types and tokens in each G, H, B, O, and ALL.
Statistics pertaining to MT
4.3.
We measured some statistics pertaining to MT. We first measured the distribution of sentence length (in words) in ALL. The mode of the length (number of words) was 23 for the English sentences and was 27 for the Japanese sen- tences. Figure 1 shows the percentage of sentences for En- glish (en) and Japanese (ja) with respect to their lengths. This figure shows that the distributions of sentence length were relatively flat and that there were many long sentences
Table 7: Number of tokens in million (English)
in ALL. This suggests that patents contain many long sen- tences that are generally difficult to translate. We then measured the coverage of vocabulary. Tables 9 and 10 list the coverage for the types and tokens in each TEST section using the vocabulary in the corresponding TRAIN section for the English and Japanese datasets. These tables show that the percentages of types in TEST covered by the vocabulary in TRAIN were relatively low for both English and Japanese.  However, the coverage of tokens was quite high. This suggests that patents are not so difficult to trans- late in terms of token coverage.
MT experiments
5.
MT system
5.1.
We used the baseline system for the shared task of the 2006 NAACL/HLT workshop on statistical machine trans- lation (Koehn and Monz, 2006) to conduct MT experi- ments on our patent corpus. The baseline system consisted of the Pharaoh decoder (Koehn, 2004), SRILM (Stolcke, 2002), GIZA++ (Och and Ney, 2003), mkcls (Och, 1999), Carmel, 11 and a phrase model training code. We followed the instruction of the shared task baseline sys- tem to train our MT systems. 12  We used the phrase model training code of the baseline system to extract phrases from TRAIN. We used the trigram language models made from TRAIN. To tune our MT systems, we did minimum er- ror rate training 13 (Och, 2003) on 1000 randomly extracted sentences from DEV using BLEU (Papineni et al., 2002) as the objective function. Our evaluation metric was %BLEU scores. 14  We tokenized and lowercased the TRAIN, DEV, DEVTEST, and TEST data sets as described in Section 4.2. We conducted two MT experiments as described in the fol- lowing sections.
Table 10: Percentages of words in test sentences covered by training vocabulary (Japanese)
Comparing reordering limits
5.2.
For the first experiment, we translated 1000 randomly sam- pled sentences in each DEVTEST data set to compare dif- ferent reordering limits. 15  We compared a reordering limit of 4 with no limitation.  The results of Tables 11 and 12 show that the %BLEU scores for no limitation consistently outperformed those for "limit=4".  These results coincide with those of Koehn et al. (2005) who reported that larger reordering limits provide better performance for Japanese- English translations. Based on this experiment, we used no reordering limit in the next experiment.
Cross section MT experiments
5.3.
For the second experiment, we conducted cross section MT experiments. The results are shown in Tables 13 and 14. For example, as listed in Table 13, when we used section G as TRAIN and used section H as TEST, we got a %BLEU
Each MT system was trained on each of the G, H, B, O, and ALL TRAIN data sets, tuned for both reordering limits using each DEV data set, and applied to 1000 randomly sampled sentences extracted from each DEVTEST data set to calcu- late the %BLEU scores listed in the table.  The source language was English and the target lan- guage was Japanese. Table 11: Comparing reordering limits (English-Japanese)
Table 12: Comparing reordering limits (Japanese-English) score of 23.51 for English-Japanese translations, whose rel- ative %BLEU score was 0.87 (=23.51/26.88) of the largest %BLEU score obtained when using ALL as TRAIN. In this case, we used all sentences in TRAIN of G to extract phrases and make a trigram language model. We used 1000 randomly sampled sentences in DEV of section G to tune our MT system.  We used all sentences in TEST of section H to calculate %BLEU scores (See Table 4 for the number of sentences in each section of TRAIN and TEST). The results in these tables show that MT systems performed the best when the training and test sections were the same. These results suggest that patents in the same section are similar to each other while patents in different sections are dissimilar. Consequently, we need domain adaptation when we apply our MT system trained in a section to another section.  However, as shown in the ALL rows, when we used all available training sentences, we obtained the high- est %BLEU scores for all but one case. This suggests that if we have enough data to cover all sections we can achieve good performance for all sections. Table 15 lists 15 example translations obtained from the Japanese-English MT system trained and tested on TRAIN and TEST of ALL. Reference translations were marked us- ing an R and MT outputs were marked using an M. The vertical bars ( | ) represent the phrase boundaries given by the Pharaoh decoder.  These examples were sampled as follows: We first randomly sampled 1000 sentences from TEST of ALL. The correctness and adequacy of the align- ment of these sentences were determined by a translation agency, as described in Section 3.2. We then selected 899 A alignments whose English translation reflected almost perfectly the contents of the corresponding Japanese sen- tences.  Next, we selected short sentences containing less than 21 words (including periods) because the MT outputs of long sentences are generally difficult to interpret. In the end, we had 212 translations. We sorted these 212 transla- tions in decreasing order of average n-gram precision16 and selected five sentences from the top, middle, and bottom of these sorted sentences. 17 Table 15 shows that top examples (1 to 5) were very good translations. These MT translations consisted of long phrases that contributed to the fluency and adequacy of translations. We think that the reason for this good transla- tion is partly due to the fact that patent documents generally Table 9: Percentages of words in test sentences covered by training vocabulary (English)
Table 13: %BLEU scores (relative %BLEU scores) for cross section MT experiments (English-Japanese)
contain many repeated expressions. For example, example 2R is often used in patent documents. We also noticed that "lcd61" in example 5M was a very specific expression and was unlikely to be repeated in different patent documents, even though it was successfully reused in our MT system to produce 5M. We found a document that contained "lcd61" in TRAIN and found that it was written by the same com- pany who wrote a patent in TEST that contained example 5R, even though these two patents were different.  These examples show that even long and/or specific expressions are reused in patent documents. We think that this charac- teristic of patents contributed to the good translations. The middle and bottom examples (6 to 15) were generally not good translations.  These examples adequately trans- lated individual phrases. However, they failed to adequately reorder phrases.  This suggests that we need more accu- rate models for reordering.  Thus, our patent corpus will be a good corpus for comparing various reordering models (Koehn et al., 2005; Nagata et al., 2006; Xiong et al., 2006).
Discussion
6.
We have described the characteristic of our patent paral- lel corpus and showed that it could be a good corpus for promoting MT research. In this section, we describe three issues about ALL that we found during investigating it as described in Sections 3., 4., and 5. We want to resolve these issues when we extend it for the NTCIR-7 patent MT task. Issue 1. Our noise reduction procedure described in Sec- tion 3.2. reduced the number of sentences from about 4.2 million to about 3.9 million.  This reduction could be too aggressive.  We want to investigate the effect of noise re- duction on the MT performance in our future work. Issue 2.  The English tokenizer we used can not handle some expressions properly. For example, it can not handle character entity references. That is, it tokenizes &amp; for example. Although the sentences containing character entity references are about 0.3% in ALL, we want to im- prove our tokenizer to remedy tokenization errors. Issue 3. We randomly split patents into TRAIN, DEV, DE- VTEST, and TEST as described in Section 4.2. This split resulted in a lot of repetitions of long and/or specific expres- sions as described in the previous section.  Consequently, the %BLUE scores obtained in our experiments could be optimistic. We want to try another split in our future work. For example, we can use the patents in 1993 to 1997 as TRAIN, and use those in 1998, 1999, and 2000 as DEV, DEVTEST, and TEST, respectively.
7.
Conclusion
Large-scale parallel corpora are indispensable language re- sources for MT. However, there are only a few publicly available large-scale parallel corpora. We developed a Japanese-English patent parallel corpus created from the Japanese and US patent data provided for the NTCIR-6 patent retrieval task.  We used Utiyama and Isahara's method and extracted about 2 million clean sen- tence alignments. This is the largest Japanese-English par- allel corpus, whose size is comparable to other large-scale parallel corpora. This corpus and its extension are planed to be used in the NTCIR-7 patent MT task. We hope that the patent corpus described in this paper will promote MT research in general and the Japanese-English patent MT research in particular.
8.