# WCC-JC: A Web-Crawled Corpus for Japanese-Chinese Neural Machine Translation
## 4 Experiment and Evaluation
### Evaluation
#### 4.2.3 Machine Translation Performance and Manual Evaluation  
In Tables 4-7, A was the test data of ASPEC-JC; O was the test data of OpenSubtitles; W was the test data of WCC-JC and N was the test data of 185 sentences extracted from the text of NHK Radio's \" ( ( ( ( ( ( ( ( ( ( ((((((((((((                                        2 2 2 2 2 2 2 22222222222222222222222222222222222222222222222  
From Tables 4-7, we could conclude that different corpora had the highest BLEU values on their own test data, except for the test data O under Japanese-Chinese (J\(\rightarrow\)C)'s character level with transformer, where our WCC-JC was more effective than OpenSubtitles' own translation. This also showed the generalizability of WCC-JC. Additionally, WCC-JC achieved better results than ASPEC-JC and OpenSubtitles on both test data W and N in all cases. Moreover, by comparing the results of different tables, we could basically conclude that the transformer model of character level was more effective in both J\(\rightarrow\)C and Chinese\(\rightarrow\)Japanese (C\(\rightarrow\)J) directions. There are many possible reasons that affect low BLEU values. The most likely cause is that one Japanese sentence had been translated into many different Chinese sentences, also known as a one-to-many situation. Regarding the problem that our BLEU values in Tables 4-7 are not very high, we investigated the duplicate Japanese sentences, which are one-to-many Japanese-Chinese sentences. The details are shown in Table 8, which shows the top 10 duplicate Japanese sentences. As we can see, these sentences were all very common daily-use utterances in spoken language, and they were translated differently when used in different scenarios, that is, one-to-many. Table 9 shows the results of one-to-many translations (only ten kinds are shown here), and the corresponding English translations show that these were colloquial short sentences. Although the characters were different, the meanings were actually very similar, that is, only because the spoken language in different scenarios will be a little different, even if when translated into English it has basically a similar meaning. However, when evaluating the translation results, we cannot include all the translation references. This was also a problem that was difficult to avoid with the evaluation metric of machine translation. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Data\textbackslash Method**} & \multicolumn{6}{c}{**Character Level**} \\  & \multicolumn{3}{c}{**LSTM**} & \multicolumn{3}{c}{**Transformer**} \\ \hline Test data\(\rightarrow\) & A & O & W & N & A & O & W & N \\ \hline ASPEC-JC & **38.8** & 1.2 & 2.8 & 2.8 & **44.8** & 1.7 & 4.2 & 5.1 \\ OpenSubtitles & 0.2 & **5.5** & 4.0 & 5.6 & 0.1 & **4.0** & 2.3 & 4.1 \\ WCC-JC & 3.2 & 3.4 & **13.8** & **8.5** & 3.5 & 3.9 & **17.1** & **9.0** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Chinese\(\rightarrow\)Japanese translation experiment results in character level (BLEU values). \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Data\textbackslash Method**} & \multicolumn{6}{c}{**Subword Level**} \\  & \multicolumn{3}{c}{**LSTM**} & \multicolumn{3}{c}{**Transformer**} \\ \hline Test data\(\rightarrow\) & A & O & W & N & A & O & W & N \\ \hline ASPEC-JC & **39.8** & 1.1 & 2.9 & 3.3 & **44.3** & 1.4 & 4.0 & 5.3 \\ OpenSubtitles & 0.2 & **4.2** & 2.6 & 3.6 & 0.2 & **4.6** & 2.6 & 4.0 \\ WCC-JC & 2.3 & 2.7 & **13.0** & **6.4** & 2.8 & 2.3 & **15.3** & **7.7** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Chinese\(\rightarrow\)Japanese translation experiment results in subword level (BLEU values).