言語処理学会 第21回年次大会 発表論文集 (2015年3月)

Error categories in English to Japanese translations
Chiho Toyoshima† , Kikuko Tanabe† , Anthony Hartley♮ and Kyo Kageura‡†1Kobe College, ♮ Tokyo University of Foreign Studies, ‡ University of Tokyo ɹIntroduction

within the scheme? - the coverage question

Assessing and assuring translation quality is one of
the main concerns for translation services, translation teaching institutions and the MT industry. An
inherent problem concerning translation quality assessment is that it inevitably involves human judgments and thus can be subjective.1 As White (2003)
notes, “[T]here are many ways to translate the samething, and reasonable translators will disagree about
which is best.” Several error classification schemes
have been proposed, such as that of the American Translators Association, SAE J 2450 and MeLLANGE error types (Secară (2005) gives a compact
introduction to these schemes), aiming to guide human assessments.
We are currently developing an online system
for translation training (Minna no Hon’yaku forTranslation Training: MNH-TT), in which modified and simplified MeLLANGE error types are used(Babych, et. al. 2012; Utiyama, et. al. 2014). MeLLANGE error types were developed as an languageindependent scheme and were proved applicableto translations among English, German, Spanish,French and Italian. They have not, however, beenvalidated for translations between languages whose
structures are radically different. Also, the applicability of these error types to translations made byless advanced learners (such as undergraduate students) has not been examined fully.
We carried out an experiment to check whether
the MeLLANGE scheme is useful for diagnosing
translation errors in English-to-Japanese translations
made by students with different learning levels.
Here we present part of the results, focusing mainlyon the following two questions:• Are there any errors identified by translation teachers that resist plausible classification
1

While automated metrics for MT quality evaluation are often presented as objective, many, including BLEU, rely on comparison with one or more human reference translations whose
quality is simply assumed but not independently validated.

• Does the distribution of errors across thescheme suggest that it is reasonably discriminant? - the granularity question

2Setup of translation experiment

In our experiment, 19 students (11 graduates and 8undergraduates)2 translated 12 English texts which
deal with current affairs. The texts were all takenfrom the same independent online broadcasting site,
Democracy Now!3 , to keep the level of the texts approximately the same. Some texts were translated bymore than one student. The undergraduates are identified as A to H, and graduates as I to S. The translations were done between July and September 2014.
Students were asked to translate within the MNH orMNH-TT environment, which provide basically thesame working conditions including lookup of dictionaries (Abekawa & Kageura 2009).

3Assigning error types

The translations of these texts were checked and error types were assigned to each error instance by thefirst and the second author of this paper, using theadapted version of the MeLLANGE scheme (Secară2005) shown in Table 1.
The assignment of error types was carried out asfollows:
1. The first author assigned error types to each error instance, while at the same time recording
why the instance met the criteria for classification as such.
2. The first and the second authors checked together the assignment of error types, while at2
We took some background information about students, including their TOEIC scores, so that we can analyze the resultof translation errors in terms of the students’ competencies. We
will report this aspect in the future.
3
http://www.democracynow.org/― 1076 ―Copyright(C) 2015 The Association for Natural Language Processing.
All Rights Reserved.

Table 1. Modified MeLLANGE error types
X1
X2X3X4X5X6X7X8X9X10X11X12X13X14
X15
X16

Group
content (C)lexis (L)grammar (G)text (T)(ST) When she met Nelson Mandela after his releasefrom prison, ...
(TT) ൴ঁ͸ωϧιϯɾϚϯσϥ͕ग़ॴͨ͠‫ޙ‬ग़ձͬTypeomission
addition
distortion
sl-intrusion
tl-intrusion
indecision
incorrect-term
inappropriate-collocation
syntaxpreposition/particle
inflection
spelling
punctuation
tl-inappropriate-register
awkward-stylecohesion

ͨࡍʹɺ
ɾ
ɾ
ɾ

the same time reviewing the validity of the criteria introduced from the first step.
Note that we did not concern ourselves here withinter-assigner agreement, as at this stage what matters is to validate the coverage and granularity of theerror scheme and to establish guidelines for assigning error types to error instances. A total of 709 errors were identified in all the translations. Amongthem, 3 error tokens were categorized as undecidable. The other 706 errors were classified under oneof the error types. Below are the three undecidable
errors.
Case 1:(ST) When Americans truly know about how muchpain and suffering the US air strikes have caused ...
they will reject this devastating targeted killing program.
(TT) ถࠃ͸ࠃ͕ۭͨ͠ऻʹΑͬͯͲͷ͘Β͍ͷۤ௧Λ΋ͨΒ͔ͨ͠ͱ͍͏͜ͱΛ͔ͬ͠Γ෼͔ͬͨΒɺ൵ࢂͳඪతࡴ֐‫ܭ‬ըΛऴΘΒͤΔͩΖ͏ɻ

The literal translation of “Americans” is “ΞϝϦΧਓ.” However, it is not easy to judge whether thisshould be classified as an error within this particulartranslation. If it is, then where is it classified?
Case 2:(ST) The reality is, racial discrimination and segregation go hand in hand.
(TT) ‫࣮ݱ‬͸ɼਓछࠩผͱਓछִ཭͕දཪҰମͰ͋Δ
͜ͱͩɻThis TT uses “da/dearu” while the remainder of thetranslation text basically adopts “desu/masu” style.
It is clear that this error falls under “text” related errors, but we need to decide which error type should
be used.
Case 3:The Japanese “ग़ॴ” has a negative connotation, andthus may not be good to use for the case of Nelson
Mandela, as he was unjustly and wrongly imprisoned. For now we cannot decide sensibly which error type should be assigned to this case (though itmay well be a case of content distortion - X3).
These case are left pending. Separately, it
should be noted that some of the error type assignments can only be done with reference to “projectdependent” resources such as glossaries specificallymade for that particular translation tasks.
All in all, we can give a positive answer to thefirst question we raised in the Introduction and conclude that the modified MeLLANGE error scheme
is adequate for capturing all errors in English-toJapanese translations, although in-depth understanding of the error types may well be required to use theerror scheme consistently. The fact that error types
could be successfully assigned to error instances inthe translations made by both graduate and undergraduate students shows that it can be used in avariety of situations in translation and translation
training. As such, our experiment partly validates
the original aim of the MeLLANGE error scheme,
which aims “to create a language independent product which will fit to as many translation institutions
as possible” (Secară 2005).

4

Distribution of error types

Here we observe the quantitative nature of errorgroups and error types, in order to answer the secondquestion raised in the Introduction. While the differences in errors made by different groups is not ourmain concern here, we sometimes refer to the difference between translations made by graduate andundergraduate students4 .
4.1

Error groups

Table 2 shows the number of errors as seen from thefour error groups of content (C), lexis (L), grammar(G) and text (T). “UD” stands for errors whose types
were undecidable. While errors related to content(C) account for almost half of all errors (49 per cent),other error groups are also observed, with lexis4
There is no guarantee that each of these groups constitutes
a coherent group from the point of view of competence in translation or language skills, but from the educational point of view
they do, by definition.

― 1077 ―Copyright(C) 2015 The Association for Natural Language Processing.
All Rights Reserved.

Table 2. Number of errors (per 1,000 words) byerror groups
C
45.45
31.9
52.08
41.42
14.3
53.61
14.36
33.25
3.17
43.8
38.19
32.83
19.52
11.85
14.25
31.33
6.78
27.2
12.72
27.79
49.46
15.13
0.54

L
26.17
23.31
20.83
4.73
11.7
15.15
2.61
25.86
0
26.76
14.32
15.15
22.3
11.85
48.43
7.23
1.36
10.36
5.78
15.49
27.57
11.49
0.74

G
20.66
4.91
14.32
7.1
3.9
8.16
13.05
14.78
0
9.73
9.55
7.58
7.43
2.37
2.85
16.87
0
10.36
6.94
8.45
15.04
5.49
0.65

T
15.11
3.68
0
1.18
1.3
4.66
2.61
3.69
6.35
14.6
0
5.05
4.65
0
8.55
4.82
0
2.59
2.31
4.27
7.6
4.29
1UD
00
1.3
0
0
0
0
0
0
0
0
0
0.93
0
00
1.36
00
0.19
0.33
0.44
2.32

0

5

10

15

20

25

30

35

A
B
C
D
EF
G
H
I
JK
L
M
NO
P
Q
RS
Mean
(%)
Sdev
Cval

C

L

G

TUD

Figure 1. Mean errors by error groups for undergraduate (black) and graduate (gray) students
related errors accounting for 28 per cent, grammarrelated errors 15 per cent, and text-related errors 7
per cent. The coefficient of variation (Cval in Table 3) indicates that the number of text-related errors varies widely among translators while contentrelated errors vary much less (assuming that we cantake the range of random variation as constant).
Figure 1 shows the number of errors by errorgroups for translations made by graduate and undergraduate students. Although the difference between
graduate and undergraduate students is not our mainconcern in this paper, it is interesting to note that theaverage numbers of content-related and grammarrelated errors are substantially higher for undergraduate students than for graduate students.
4.2

Error types

Error types, rather than the error groups we just saw,
constitute the most important level, as it is thesewhich are assigned to individual error instances. Table 3 shows the normalised quantity of the sixteen
error types and undecidable errors. Figure 2 shows
the mean number of errors for all, undergraduate andgraduate translators.
Slightly more than a quarter of all errors areclassified as content distortion (X3), which mostprobably reflects the misunderstanding of source
texts. This is followed by incorrect usage of terms(X7: 16 per cent), inappropriate collocations (X8:13 per cent) and intrusion of source language in content (X4: 11 per cent). They together constitute two
thirds of all errors. Content error by target language
intrusion (X5) was not observed at all, while contentindecision (X6) was observed only in a very smallnumber.
All in all, while a relatively small number oferror types account for a large portion of actual errors, other less-frequently observed error types wereclearly identified. We can thus reasonably conclude
that the modified MeLLANGE error types can sufficiently capture errors for English-to-Japanese translations.
Figure 2 shows some interesting points. Whilethe 3.99 overall mean error for undergraduates ismuch higher than the 2.79 for graduates (the overall mean is 3.29), graduate translators tend to makecertain types of errors more frequently than do undergraduates, i.e. content addition (X2), collocation(X8), inappropriate register of TL text (X14), andawkward style (X15). This may reflect the fact that,in general, graduate students tend to have less difficulty in understanding SL texts, while tending tointroduce inappropriate expressions in the processof producing TL texts. Although full investigationof the factors leading to these quantitative tendencies remains one of our future tasks, the fact that thequantitative observation of error types enables us toraise this kind of questions indicates that the error
types are very useful for diagnosing translations.

5Conclusions and outlook

This paper reports the result of our experimentin assigning modified MeLLANGE error types toJapanese translations of English texts. In relation
to the main questions we raised in the Introduction,
we found that errors can be assigned to error types
almost exhaustively. Moreover, although we can observe a few error types which account for a large portion of errors, the scheme appears useful for discriminating between a range of translation errors in En-― 1078 ―Copyright(C) 2015 The Association for Natural Language Processing.
All Rights Reserved.

Table 3. Number of errors (per 1,000 words) by error types
X2
0
1.23
1.3
1.18
1.3
1.17
0
0
0
00
5.05
0
2.37
5.7
2.41
1.36
1.3
0
1.28
2.27
1.61

X3
24.79
20.86
22.14
27.22
9.1
22.14
6.53
18.47
3.17
21.9
28.64
27.78
11.15
2.37
5.7
26.51
4.07
11.66
6.94
15.85
28.08
9.21

X2X3X4 X5 X6X7X8 X9 X10 X11 X12 X13 X14
11.02
0
0 20.66 5.51 1.38 1.38 2.75 6.89 8.260
9.82
00 4.91 18.4
0 2.450
0 2.45023.44
00 18.23
2.6 5.21 1.3 2.6 1.3 3.91
0
7.1
00 3.55 1.18
0 1.180 1.18 4.73
0
3.9
006.5
5.2
0
0
0 2.6 1.3
09.32
00 12.82 2.33 2.33
0 1.17 1.17 3.5 2.33
2.61
00 2.61
0 1.31 3.92 1.31
0 6.530
7.39
00 17.24 8.62 1.23 4.93
0 7.39 1.23 1.23
00000000000
19.46
00 12.17 14.6 2.43
00 2.43 4.87 7.3
4.77
0
0 4.77 9.55
00
0 2.39 7.16
0000 10.1 5.05 2.53
0 2.530 2.53 2.53
4.65
0
0 9.29 13.01 1.86 1.86 0.93 0.93 1.86 1.86
2.37
0
0 4.74 7.11
00
0 2.3700
0
0 2.85 25.64 22.79
00
0
0 2.8500
0
0
0 7.23 2.41
0 2.41 4.82 7.23 2.41
00
0 1.36
0
0
0
0
000
10.36
0
0 6.48 3.89
0 2.590 1.3 6.48 2.59
5.78
0
0 3.47 2.31 3.47
0 1.16
0 2.31 1.6
6.42
0 0.15 8.66 6.81 1.27 1.03 0.78 1.83 3.54 1.15
11.37
0 0.27 15.34 12.06 2.25 1.82 1.38 3.24 6.27 2.04
6.36
0 0.64 7.17 6.26 1.46 1.46 1.03 2.21 2.56 1.78

X15 X16 UD
0 15.1500 3.68000 1.3
0 1.18001.3
01.17 1.17
00 2.6101.23 1.23
00 6.3502.43 4.8700000 2.53
00 2.79 0.93
0002.855.7
00 2.4100000
0 1.3
0 1.1600.4 2.74 0.19
0.71 4.85 0.34
0.85 3.48 0.43

0510

15

X1A
9.64
B
4.91
C
5.21
D
5.92
E0F
20.97
G
5.22
H
7.39
I0J
2.43
K
4.77
L0M
3.72
N
4.74
O0P
2.41
Q
1.36
R
3.89
S0
Mean 4.35
(%)
7.71
Sdev
4.76

X1X4X5X6X7X8

X9X10X11X12X13X14X15X16

UD

Figure 2. Mean errors by error types for all (black), undergraduates (dark gray) and graduates (light gray)glish to Japanese translation. We are currently producing a guideline which uses examples to explain
how to assign error types, to be published as one partof the MNH-TT user guide. At the same time, weare experimenting with the applicability of the error
types to other language pairs such as Japanese andGerman.

References

Acknowledgement

Secară, A. (2005) “Translation evaluation: A stateof the art survey,” Proc. eCoLoRe/MeLLANGE
Workshop. pp. 39–44.

This study is partly supported by JSPS Grant-inAid (A) 25240051 “Archiving and Using Translation Knowledge to Construct Collaborative Translation Training Aid System”.

Abekara, T. and Kageura, K. (2007) “A translation aid system with a stratified lookup interface,”
Proc. 45th ACL Poster and Demo. pp. 5–8.
Babych, B. et. al. (2012) “Scaffolding, capturing,preserving interactions in collaborative translation,” LTTC International Conference.

Utiyama, M. et. al. (2009) “Minna no Hon’yaku:
A website for hosting, archiving, and promotingtranslations,” 31th Translating and the Computer.
White, J. (2003) “How to evaluate machine translation,” in H. Somers (ed.) Computers and Translation, Amsterdam: Benjamins.

― 1079 ―Copyright(C) 2015 The Association for Natural Language Processing.
All Rights Reserved.