3"step'Parallel'Corpus'Cleaning'
using'Monolingual'Crowd'Workers
Toshiaki'Nakazawa,'Sadao'Kurohashi'(Kyoto'University)'
Hayato'Kobayashi,'Hiroki'Ishikawa'
Manabu'Sassano'(Yahoo'Japan'CorporaHon)
20/05/2015@PACLING2015

Parallel'Corpora• EssenHal'resources'for'almost'all'MT'systems'• The'quality'and'quanHty'greatly'aﬀect'the'translaHon'quality'• Can'be'automaHcally'constructed'from'exisHng'resources'– Europarl patent'families,'Wikipedia…'• Need'to'manually'construct'it'for'domains'which'do'not'have'enough'exisHng'resources

Quality'of'Parallel'Corpus• TranslaHon'ﬂaws'are'inevitable'even'thought'
the'professionals'translate'– Homer&nods'('in'Japanese)'• The'number'of'ﬂaws'might'be'reduced'by'reviewing'the'whole'corpus,'but'impossible'– The'size'of'the'parallel'corpus'is'usually'very'big'– Very'costly'if'we'ask'professionals'to'modify'This'Work• Detect'and'edit'the'translaHon'ﬂaws'in'the'exisHng'manually"translated'parallel'corpus'in'eﬀecHve'and'cheap'way'• Use'crowdsourcing'in'3"steps'
1. Fluency'Judgement'
2. Edit'of'Unnatural'Sentences'
3. VeriﬁcaHon'of'Edits'• The'workers'can'be'monolingual'Outline• MoHvaHon'• Brief'introducHon'of'collaboraHve'research'
between'Yahoo'Japan'and'Kyoto'University'• 3"step'parallel'corpus'cleaning'• Experiments'– Parallel'corpus'cleaning'– TranslaHon'• Conclusion'and'Future'Work'

BRIEF&INTRODUCTION&OF&COLLABORATIVE&RESEARCH

CollaboraHve'Research'Between'
Yahoo'Japan'and'Kyoto'University'• Goal:'Improve'the'Chinese"to"Japanese'translaHon'for'E"commerce'site'• Task:'– Develop'a'corpus"based'MT'system'– Construct'a'parallel'corpus'of'EC"site,'especially'
fashion'domain'Fashion"domain'EC"site'Parallel'Corpus[FDEC'Corpus]• 1.2M'sentences'(Zh:'6.3M,'Ja:'8.7M'words)'• Manually'translated'from'fashion'item'pages'of'Chinese'EC"site'(taobao)'into'Japanese'• Most'of'the'sentences' ere'translated'by'
Chinese'naHve'speakers'(through'the'translaHon'company)'– Found'many'translaHon'ﬂaws'in'the'Japanese'translaHons'Mother"tongue'Principle“A&translator&should,&as&far&as&possible,&translate&into&his&own&mother&tongue&or&into&a&language&of&which&he&or&she&has&a&mastery&equal&to&that&of&
his&or&her&mother&tongue.”'
RecommendaHon'on'the'Legal'ProtecHon'of'Translators'and'TranslaHons'and'the'PracHcal'Means'to'improve'the'Status'of'Translators,'UNESCO,'22'Nov.'1976
hkp://portal.unesco.org/en/ev.php"URL_ID=13089&URL_DO=DO_TOPIC&URL_SECTION=201.html

Source'NaHves'vs.'Target'NaHves
Source&Na8ves Target&Na8ves
Background'knowledge'
about'the'input'sentence'HighMedium/Low

Fluency'and'grammaHcal'
correctness'of'the'output'sentence

Medium/Low

High• Pros'and'cons'for'source'and'target'naHve'
speakers'• Target'naHves'for'translaHon'modiﬁcaHon'[Albrecht+,'2009]

Examples'of'TranslaHon'Flaws• InserHon'
Ja:'
1'''''''(order&one&type&at&at'your&own&will)'• Remaining'Chinese'character'
Ja:'	
Ref:'	''

Hanzi Kanji• UnnaturalJa:''''(when&cleaning,&turn&the&power&oﬀ,&please'pull&out&the&plug)Other'TranslaHon'Flaws• Omission'(not'translated)'
Zh:'
Ja:'

'

• MistranslaHon'
Zh:
Ja:'

'(func>ons&of&humidiﬁer)'(func>ons&of&dehumidiﬁer)''Zh:'wood'ear'mushroom
frill
skirt'with'wood'ear'mushroom?

our'framework'cannot'ﬁx'these'kinds'of'ﬂaws

3>STEP&PARALLEL&CORPUS&CLEANING&

3"steps'of'Cleaning
1. Fluency'Judgement'– detects'the'translaHon'ﬂaws'

2. Edit'of'Unnatural'Sentences'– edit'the'translated'sentences'

3. VeriﬁcaHon'of'Edits'– check'if'the'edited'translaHon'is'beker'than'the'original'one''

Step'1:'Fluency'Judgement• Task:'judge'if'the'sentences'are'natural'and'grammaHcally'correct'• Only'showing'the'translated'(target,'Japanese)'sentences'e.g.'

1'Is'this'sentence'natural'and'grammaHcally'correct?
No!!
No!!

Step'2:'Edit'of'Unnatural'Sentences• Task:'edit'the'unnatural'translated'sentences'• only'showing'the'translated'sentences,'or'
show'the'source'sentence'as'well'for'the'reference'e.g.'1Please'modify'this'sentence'to'be'natural'and'grammaHcally'correct1
Nothing'to'modify!

Step'3:'VeriﬁcaHon'of'Edits• Task:'judge'if'the'edited'translaHon'is'beker'
than'the'original'one'• This'step'is'important'to'further'improve'the'quality'of'the'outcome'because'the'edits'are'not'necessarily'correct'
e.g.'

1vs.

1

Is'the'right'one'more'natural'and'grammaHcally'correct'than'the'let'one?
Yes!!
Yes!!

'CORPUS&CLEANING&EXPERIMENTS

Crowdsourcing'Service'in'the'World

hkp://www.crowdinfo.jp/2014/02/02/world"crowdsourcing"service/Crowdsourcing
hkp://crowdsourcing.yahoo.co.jp• Several'styles'of'crowdsourcing'tasks'such'as'
Yes/No'quesHons'and'free'wriHngs'• The'service'is'run'in'Japan;'therefore'most'of'
the'workers'are'Japanese'• Not'able'to'select'the'workers'by'their'
abiliHes'• The'workers'in'our'experiments'do'not'necessarily'understand'Chinese'– perhaps'almost'all'of'them'does'notStep'1:'Fluency'Judgement• 358,085'sentences'from'the'FDEC'corpus'with'length'between'10'and'130'characters'• Only'Japanese'sentences'are'shown'• Asked'5'diﬀerent'workers'for'each'quesHon#&unnatural
5
4
3
2
1
0
#'sents.'
13,056' 35,048' 60,200' 83,150' 93,187' 73,444'
raHo(3.6%) (9.8%) (16.8%) (23.2%) (26.0%) (20.5%)30%!

Step'2:'Edit'of'Unnatural'Sentences• 47,420'sentences'which'were'judged'as'
unnatural'by'4'or'more'workers'in'Step'1'• Original'Chinese'sentence'is'also'shown'• Asked'3'diﬀerent'workers'for'each'quesHon#&edits#'sents.'
raHo

3
2
1
0
3,755' 12,498' 18,289' 12,878'(7.9%) (26.4%) (38.6%) (27.2%)

Step'3:'VeriﬁcaHon'of'Edits• 54,550'edits'which'were'generated'in'Step'2'• Original'Chinese'sentence'is'also'shown'• Asked'5'diﬀerent'workers'for'each'quesHon#&beNer#'sents.'
raHo

5
4
3
25,053' 16,478' 7,706'(45.9%) (30.2%) (14.1%)

23,338'(6.1%)

1
1,462'(2.7%)

0513'(0.9%)

TranslaHon'Experiment• Dataset:'whole'FDEC'Corpus'#&sentences
Train
Dev
Test

Original
1,220,597
11,186
11,200Cleaned1,256,908
11,48911,495– Cleaned:'veriﬁed'to'be'beker'by'the'majority'• Decoder:'KyotoEBMT'[Richardson+,'2014]'• EvaluaHon:'BLEU

Experimental'Results
Train
Dev
Test
BLEU

Original Cleaned Cleaned Cleaned
Original Original Cleaned Cleaned
Original Original Original Cleaned
21.39
21.69
21.34
21.12

• Corpus'cleaning'contributes'to'improve'the'translaHon'quality!'• Cleaning'the'Dev'and'Test'sets'has'bad'eﬀect'
on'translaHon'quality…'

Natural,'but'Incorrect/Unequal• Reviewed'100'edits'which'are'judged'to'be'
more'natural'than'the'original'sentence'by'5'workers''• Found'3'types'of'inequaliHes'
1. deleHon'of'symbols'(8'cases)'
2. omission'(13'cases)'
3. mistranslaHon'(5'cases)''''see'the'proceedings'for'detailed'examples

Experimental'Results
Train
Dev
TestBLEU

Original Cleaned Cleaned Cleaned
Original Original Cleaned Cleaned
Original Original Original Cleaned21.39
21.69
21.34
21.12

• The'inequaliHes'have'bad'eﬀect'on'the'automaHc'evaluaHon'scores'because'they'
suppose'the'content'of'the'input'and'output'
are'strictly'equal''Crowdsourcing'Cost• Cost'for'cleaning'6.8M'words'used'in'the'experiments

FeeTimeProfessional*
Our&Work
40'million'JPY 2.6'million'JPY
1700'days
186'hours*'These'values'are'esHmated'from'hkp://www.editage.com''

Conclusion'and'Future'Work• Proposed'a'framework'of'cleaning'exisHng'
parallel'corpora'eﬃciently'and'cheaply'– 3"step'monolingual'crowdsourcing'– Improved'the'ﬂuency'of'the'sentences'• Future'work'– How'to'reduce'the'inequaliHes' f'the'edits?'– How'to'improve'the'correctness'of'the'translaHon'
by'monolingual'workers?'