Consistent Classiﬁcation of Translation Revisions:
A Case Study of English-Japanese Student Translations*Atsushi Fujita (NICT)
Kikuko Tanabe (Kobe College)
Chiho Toyoshima (Kansai Gaidai Univ.)
Mayuka Yamamoto (Honyaku Center Inc.)
Kyo Kageura (U. of Tokyo)
Anthony Hartley (Rikkyo U.)
1. Quality Assessment of Student Translations

Step 1. Iden:fy erroneous or deﬁcient text spans andsuggest a revision proposal for reasoning
Step 2. Classify each issue into an abstract issue typeMark Potok is a senior fellow at the Southern Poverty Law Center.
マーク・ポッドック氏は南部貧困法律センターの上級研究員だ。
ポトク
Should not be shown tostudents [Klaudy, 96]

Summary of this work
Quality assessment of human-produced transla:onsl Development of issue typology & decision treel For assessing English-Japanese student transla:onsl Using the OntoNotes methodl High consistency of issue classiﬁca:onl Installa:on into a transla:on exercise course in a university
lincorrect transla:on of term
Diﬃcult to consistently
classify e.g., [Lommel+, 14]

2. Development of an Issue Classiﬁcation Scheme

Our decision tree

Given an issue

Q2a: Is all content in the SD translatedin proper quan22es in a proper way?

High consistency to guarantee beier learning performancel Wide applicability for transla:on studies from various aspects
l Radically diﬀerent language pairs, e.g.,. English & Japanesel Diﬀerent levels of learners: undergrad. vs. graduate students
lQ2b: Is the error related toa term in the given glossary?
X7 incorrect-term
X1 omission
X2 addi:on
X3 distor:onLevel 2

Seman:c errors

Step 1. Annotate issues in the given 3 SD/TD pairs
Step 2. Terminate the itera:on if the agreement ra:o >= 90%
Step 3. Discuss the factor of consistent decision making
Step 4. Update the issue typology & decision tree
3. Notably High Inter-assessor Agreement

Q3a: Is it a gramma2cal issue?
Q3b: predeﬁned speciﬁc type of error?
X8 colloca:on
X10 preposi:on/par:cle
X11 inﬂec:on
X12 misspelling
X13 punctua:onLevel 3TD linguis:c issues

Using our carefully designed scheme
Given faithful & careful annota:on

X9 other-grammar/syntax

Q4a: Does it hurt cohesiveness of the TD?
X16 incohesive
Q4b: Does it hurt ﬂuency?

ValidaCon 1: classiﬁca:on of 575 issues in 17 TDs
A, B: anonymous paid workers (professional translators)l C, D: iden:ﬁed assessors (two of the authors)

Q4c: Is it too literal?
X4b too-literal
X15 clumsy

lMany carelessmistakes

AB

Level 4

TD felicity issues

Level 5 Q5a: Is it unsuitable for the intended register?

67.7%/κ=0.613

TD register issues

67.1%/κ=0.592
63.3%/κ=0.554
57.9%/κ=0.490
86.6%/κ=0.831
Agr. ra:o [%] / κ value

DDisagreements
were easily solvedValidaCon 2: assessment of 10 TDs from scratch

X14 inappropriate-registerQ6a: Is it anyways problema;c?
Other issue

61.4%/κ=0.523
CNoQ1a: Is it an unjus2ﬁed copy of the SD element?
X4a untranslated
Level 1 Q1b: Do mul2ple op2ons remain in the TD?
X6 indecision
IncompletenessOur objec:veItera:ve reﬁnement of the MNT-TT typology [Babych+, 12]
using the OntoNotes method [Hovy+, 06]

Yes

Not an issue

4. Application to a Translation Course at a University

As a means of feedback for assignments
More X3
Useful for ﬁnding clusters& monitoring their learning paierns

24

10

5

D: same as in valida:on 1 à 561 issuesl E: undergraduate learner translator à 406 issues
Diﬀerent levels
l 340 iden:cal text spansof sensi:vity
with similar revision proposals
lMore X3

2311192622

More X7

1022
1718

12
22

23

917
26
525
8D

E

6

12

14

3

8

More X7 & X4b

Much higher than involving
anonymous paid workers

13111627

920

76725

Understandable byan actual learner

27

4

19

85.0%/κ=0.794

1424413

For 1st SD

For 2nd SD

1813
15

115

20

21

16

Less X3

21

Less X3