Pronoun-Focused MT and Cross-Lingual Pronoun Prediction:
Findings of the 2015 DiscoMT Shared Task on Pronoun Translation
Christian Hardmeier

Preslav Nakov

Uppsala University

Qatar Computing Research Institute

Uppsala University

Dept. of Linguistics and Philology

HBKU

Dept. of Linguistics and Philology

first.last@lingfil.uu.se

pnakov@qf.org.qa

first.last@lingfil.uu.se

Jörg Tiedemann

Yannick Versley

Mauro Cettolo

Uppsala University

University of Heidelberg

Fondazione Bruno Kessler

Dept. of Linguistics and Philology

Institute of Computational Linguistics

Trento, Italy

first.last@lingfil.uu.se

versley@cl.uni-heidelberg.de

cettolo@fbk.eu

Abstract

Organizing a shared task involves clearly defining the problem, then creating suitable datasets
and evaluation methodologies. Having such a setup
makes it possible to explore a variety of approaches
for solving the problem at hand since the participating groups independently come up with various
ways to address it. All of this is highly beneficial
for continued research as it creates a well-definedbenchmark with a low entry barrier, a set of results to compare to, and a collection of properly
evaluated ideas to start from.
We decided to base this shared task on the problem of pronoun translation. Historically, this was
one of the first discourse problems to be consideredin the context of SMT (Le Nagard and Koehn,
2010; Hardmeier and Federico, 2010); yet, it isstill far from being solved. For an overview of theexisting work on pronoun translation, we refer thereader to Hardmeier (2014, Section 2.3.1). The typical case is an anaphoric pronoun – one that refers
to an entity mentioned earlier in the discourse, its
antecedent. Many languages have agreement constraints between pronouns and their antecedents.
In translation, these constraints must be satisfied
in the target language. Note that source language
information is not enough for this task. To see
why, consider the following example for English–
French:1

We describe the design, the evaluation
setup, and the results of the DiscoMT 2015
shared task, which included two subtasks, relevant to both the machine translation (MT) and the discourse communities: (i) pronoun-focused translation, a practical MT task, and (ii) cross-lingual pronoun prediction, a classification task that
requires no specific MT expertise and isinteresting as a machine learning task in
its own right. We focused on the English–French language pair, for which MT
output is generally of high quality, but has
visible issues with pronoun translation due
to differences in the pronoun systems ofthe two languages. Six groups participatedin the pronoun-focused translation task andeight groups in the cross-lingual pronoun
prediction task.

1

Sara Stymne

Introduction

Until just a few years ago, there was little awareness of discourse-level linguistic features in statistical machine translation (SMT) research. Since
then, a number of groups have started working on
discourse-related topics, and today there is a fairly
active community that convened for the first timeat the Workshop on Discourse in Machine Translation (DiscoMT) at the ACL 2013 conference in
Sofia (Bulgaria). This year sees a second DiscoMT
workshop taking place at EMNLP 2015 in Lisbon(Portugal), and we felt that the time was ripe tomake a coordinated effort towards establishing thestate of the art for an important discourse-related
issue in machine translation (MT), the translationof pronouns.

The funeral of the Queen Mother willtake place on Friday. It will be broadcast
live.
Les funérailles de la reine-mère auront
lieu vendredi. Elles seront retransmises
en direct.
1 The example is taken from Hardmeier (2014, 92).

1
Proceedings of the Second Workshop on Discourse in Machine Translation (DiscoMT), pages 1–16,
Lisbon, Portugal, 17 September 2015. c 2015 Association for Computational Linguistics.

Here, the English antecedent, the funeral of the
Queen Mother, requires a singular form for theanaphoric pronoun it. The French translation of
the antecedent, les funérailles de la reine-mère, isfeminine plural, so the corresponding anaphoricpronoun, elles, must be a feminine plural form
too. Note that the translator could have chosen totranslate the word funeral with the French word
enterrement ‘burial’ instead:Experiments on discourse-related aspects of MT
are unlikely to be successful unless a strong MT
baseline is used. Also, evaluation is much easierif there are clear, relevant, measurable contrasts inthe translation task under consideration (Hardmeier,2012). For the DiscoMT shared task, we chose to
study translation from English into French because
this language pair is known from other evaluations
such as WMT or IWSLT to have good baseline performance. Also, there are interesting differences in
the pronoun systems of the two languages. French
pronouns agree with the grammatical gender of
their antecedent in both singular and plural. In English, the singular pronouns he and she agree with
the natural gender of the referent of the antecedent,
and the pronoun it is used with antecedents lacking natural gender; the plural pronoun they is notmarked for gender at all.

L’enterrement de la reine-mère aura lieu
vendredi. Il sera retransmis en direct.
This time, the antecedent noun phrase (NP) is masculine singular and thus requires a masculine singular anaphoric pronoun and singular verb forms.
Therefore, correctly translating anaphoric pronouns
requires knowledge about a pronoun’s antecedentand its translation in the target language.
Early SMT research on pronoun translation focused exclusively on agreement in the target language (Le Nagard and Koehn, 2010; Hardmeierand Federico, 2010). While this is one of the main
issues with pronoun translation, it soon became
clear that there were other factors as well. On the
one hand, the same source language pronoun canhave both anaphoric and non-anaphoric functions,with different constraints. On the other hand, anaphoric reference can be realized through different
types of referring expressions, including personalpronouns, demonstrative pronouns, zero pronouns,
full noun phrases, etc., with different languages exploiting these means in different ways. The precise
mechanisms underlying these processes in various
language pairs are not well understood, but it iseasy to see that pronoun translation is not a trivial
problem, e.g., by noting that the number of pronouns on the source and on the target side of thesame parallel text may differ by up to 40 % (Mitkov
and Barbu, 2003).

2

The text type, or “domain”, considered in theshared task is that of public lectures delivered at
TED conferences. This choice was motivated by
the ready availability of suitable training data in the
WIT3 corpus (Cettolo et al., 2012), together withthe fact that this text type is relatively rich in pronouns compared to other genres such as newswire(Hardmeier et al., 2013b).
In the pronoun-focused translation task, participants were given a collection of English input
documents, which they were asked to translate into
French. As such, the task was identical to other MTshared tasks such as those of the WMT or IWSLT
workshops. However, the evaluation of our sharedtask did not focus on general translation quality, but
specifically on the correctness of the French translations of the English pronouns it and they. Since
measuring pronoun correctness in the context ofan actual translation is a very difficult problem in
itself, the evaluation of this task was carried out
manually for a sample of the test data.

Task Description

The cross-lingual pronoun prediction task was a
gap-filling exercise very similar to the classification
problem considered by Hardmeier et al. (2013b).
Participants were given the English source text ofthe test set along with a full reference translation
created by human translators. In the reference translations, the French translations of the English pronouns it and they were substituted with placeholders. For each of these placeholders, the participants
were asked to predict a correct pronoun from asmall set of nine classes (see Table 1), given thecontext of the reference translation.

The shared task had two subtasks. The first subtask,
pronoun-focused translation, required full translation of texts from one language into another withspecial attention paid to the translation of pronouns.
The second, cross-lingual pronoun prediction, wasa classification task requiring only the generation of
pronouns in the context of an existing translation.
Its purpose was to lower the entrance barrier by
allowing the participants to focus on the actual pronoun translation problem without having to worry
about the complexities of full MT.
2

ce
elle
elles
il
ils
ça
cela
on
OTHER

The French pronoun ce (sometimes
with elided vowel as c’) as in the expression c’est ‘it is’feminine singular subject pronoun
feminine plural subject pronoun
masculine singular subject pronoun
masculine plural subject pronoun
demonstrative pronoun (including the
misspelling ca and the rare elided
form ç’)demonstrative pronoun
indefinite pronoun
some other word, or nothing at all,should be inserted

IWSLT14.train

179k

3.63M

3.88M

1415

IWSLT14.dev2010
IWSLT14.tst2010
IWSLT14.tst2011
IWSLT14.tst2012

887
1664
818
1124

20,1k
32,0k
14,5k
21,5k

20,2k
33,9k
15,6k
23,5k

8
11
8
11

DiscoMT.tst2015

2093

45,4k

48,1k

12

en

tokensfr

talks

Table 2 provides statistics about the in-domain
tokenized bitexts we supplied for training, development and evaluation purposes.
Note that TED talks differ from other text types
with respect to pronoun use. TED speakers frequently use first- and second-person pronouns (singular and plural): first-person pronouns to refer tothemselves and their colleagues or to themselves
and the audience, and second-person pronouns torefer to the audience, to the larger set of viewers, orto people in general. Moreover, they often use thepronoun they without a specific textual antecedent,
in phrases such as “This is what they think”, aswell as deictic and third-person pronouns to referto things in the spatio-temporal context shared by
the speaker and the audience, such as props andslides. In general, pronouns are abundant in TED
talks, and anaphoric references are not always veryclearly defined.

The evaluation for the cross-lingual pronoun prediction task was fully automatic, comparing thepredictions made by the participating systems withthe translations actually found in the reference.

Datasets

As already noted, the corpus data used in the DiscoMT shared task comes from the TED talks. In
the following, the datasets are briefly described.
3.1

segs

Table 2: Statistics about the bilingual linguistic
resources for the shared task.

Table 1: The nine target pronoun classes predictedin the cross-lingual pronoun prediction task.

3

DatasetData Sources

TED is a non-profit organization that “invites theworld’s most fascinating thinkers and doers [...] togive the talk of their lives”. Its website2 makes theaudio and video of TED talks available under theCreative Commons license. All talks are presented and captioned in English, and translated byvolunteers world-wide into many languages. In
addition to the availability of (audio) recordings,
transcriptions and translations, TED talks pose interesting research challenges from the perspectiveof both speech recognition and machine translation. Therefore, both research communities aremaking increased use of them in building benchmarks. TED talks address topics of general interest
and are delivered to a live public audience whose
responses are also audible on the recordings.3 The
talks generally aim to be persuasive and to change
the viewers’ behaviour or beliefs. The genre of the
TED talks is transcribed planned speech.

3.2

Selection Criteria

The training and the development datasets for ourtasks come from the English-French MT task of
the IWSLT 2014 evaluation campaign (Cettolo etal., 2014). The test dataset for our shared task,
named DiscoMT.tst2015, has been compiled fromnew talks added recently to the TED repository thatsatisfy the following requirements:
1. The talks have been transcribed (in English)
and translated into French.
2. They were not included in the training, development, and test datasets of any IWSLT
evaluation campaign, so DiscoMT.tst2015 canbe used as held-out data with respect to those.
3. They contain a sufficient number of tokensof the English pronouns it and they translated
into the French pronouns listed in Table 1.
4. They amount to a total number of words suitable for evaluation purposes (e.g., tens of thousands).

2 http://www.ted.com

3 The following overview of text characteristics is based on
work by Guillou et al. (2014).

3

To meet requirement 3, we selected talks forwhich the combined count of the rarer classes ça,
cela, elle, elles and on was high. The resulting
distribution of pronoun classes, according to theextraction procedure described in Section 5.1, canbe found in Table 8 further below.
We aimed to have at least one pair of talks givenby the same speaker and at least one pair translatedby the same translator. These two features are notrequired by the DiscoMT shared task, but could be
useful for further linguistic analysis, such as theinfluence of speakers and translators on the use of
pronouns. Talks 1756 and 1894 were presented by
the same speaker, and talks 205, 1819 and 1825
were translated by the same translator.
Once the talks satisfying the selection criteria
were found, they were automatically aligned at thesegment level and then manually checked in order
to fix potential errors due to either automatic orhuman processing. Table 3 shows some statistics
and metadata about the TED talks that are part of
the DiscoMT.tst2015 set.
talk id

segs

205
1756
1819
1825
1894
1935
1938
1950
1953
1979
2043
2053

189
186
147
120
237
139
107
243
246
160
175
144

4,188
4,320
2,976
2,754
5,827
3,135
2,565
5,989
4,520
2,836
3,413
2,828

4,109
4,636
3,383
3,078
6,229
3,438
2,802
6,416
4,738
2,702
3,568
3,023

J.J. Abrams
A. Solomon
S. ShahB. Barber
A. Solomon
S. Chandran
P. Evans
E. Snowden
L. Page
M. Laberge
N. Negroponte
H. Knabe

total

2,093

45,351

48,122–entokensfr

The parallel data were taken from OPUS
(Tiedemann, 2012), which provides sentencealigned corpora with annotation. The latter isuseful for finding document boundaries, which
can be important when working with discourseaware translation models. All training data werepre-processed with standard tools from the Moses
toolkit (Koehn et al., 2007), and the final datasets
were lower-cased and normalized (punctuation was
unified, and non-printing characters were removed).
The pre-processing pipeline was made available on
the workshop website in order to ensure compatibility between the submitted systems.
The parallel data were prepared for word alignment using the cleaning script provided by Moses,with 100 tokens as the maximum sentence length.
The indexes of the retained lines were saved tomake it possible to map sentences back to the annotated corpora. The final parallel corpus contained
2.4 million sentence pairs with 63.6 million words
in English and 70.0 million words in French. We
word-aligned the data using fast_align (Dyer et al.,2013) and we symmetrized the word alignments using the grow-diag-final-and heuristics. The phrase
tables were extracted from the word-aligned bitext using Moses with standard settings. We alsofiltered the resulting phrase table using significancetesting (Johnson et al., 2007) with the recommended filter values and parameters. The phrase table
was provided in raw and binary formats to make iteasy to integrate it in other systems.
For the language model, we used all monolingual
datasets and the French parts of the parallel datasetsand trained a 5-gram language model with modified
Kneser-Ney smoothing using KenLM (Heafield etal., 2013). We provided the language model in
ARPA format and in binary format using a trie data
structure with quantization and pointer compression.
The SMT model was tuned on the IWSLT 2010
development data and IWSLT 2011 test data using
200-best lists and MERT (Och, 2003). The resulting baseline system achieved reasonably goodscores on the IWSLT 2010 and 2012 test datasets(Table 4).

speakerTable 3: Statistics about the talks that were included in DiscoMT.tst2015.

4

Pronoun-Focused Translation

4.1

Baseline System

For comparison purposes and to lower the entry
barrier for the participants, we provided a baseline
system based on a phrase-based SMT model. The
baseline system was trained on all parallel andmonolingual datasets provided for the DiscoMT
shared task, namely aligned TED talks from the
WIT3 project (Cettolo et al., 2012), as well as Europarl version 7 (Koehn, 2005), News Commentary
version 9 and the shuffled news data from WMT
2007–2013 (Bojar et al., 2014).

test set

BLEU

IWSLT 2010
IWSLT 2012

33.86
40.06

(BP=0.982)(BP=0.959)

Table 4: Baseline models for English-French machine translation: case-insensitive BLEU scores.
4We experimented with additional datasets andother settings (GIZA++ instead of fast_align, unfiltered phrase tables), but could not improve.
All datasets, models and parameters were madeavailable on the shared task website to make iteasy to get started with new developments and tocompare results with the provided baseline. For
completeness, we also provided a recasing modelthat was trained on the same dataset to render it
straightforward to produce case-sensitive output,
which we required as the final submission.
4.2

ITS 2 (Loáiciga and Wehrli, 2015) was a rulebased machine translation system using syntaxbased transfer. For the shared task, it was extended
with an anaphora resolution component influenced
by Binding Theory (Chomsky, 1981).
For the sixth submission, A 3-108, no systemdescription paper was submitted. Its output seemed
to have been affected by problems at the basic MT
level, yielding very bad translation quality.

4.3

Evaluation Methods

Evaluating machine translations for pronoun correctness automatically is difficult because standard
assumptions fail. In particular, it is incorrect toassume that a pronoun is translated correctly if itmatches the reference translation. If the translationof an anaphoric pronoun is itself a pronoun, it has
to agree with the translation of its antecedent, anda translation deviating from the reference may be
the only correct solution in some cases (Hardmeier,2014, 92). Doing this evaluation correctly would
require a working solution to the cross-lingual pronoun prediction task, the second challenge of ourshared task. Given the current state of the art, wehave little choice but to do manual evaluation.4
Our evaluation methodology is based on the gapfilling annotation procedure introduced by Hardmeier (2014, Section 9.4). We employed two annotators, both of whom were professional translators,
native speakers of Swedish with good command of
French. Tokens were presented to the annotatorsin the form of examples corresponding to a single
occurrence of the English pronouns it or they. For
each example, the sentence containing the pronoun
was shown to the annotator along with its machinetranslation (but not the reference translation) andup to 5 sentences of context in both languages. In
the MT output, any French pronouns aligned tothe pronoun to be annotated were replaced witha placeholder. The annotators were then asked toreplace the placeholder with an item selected froma list of pronouns that was based on the classes of
the cross-lingual pronoun prediction task (Table 1).
Compared to the perhaps more obvious methodology of having the annotators judge examples
as good or bad, treating evaluation as a gap-fillingtask has the advantage of avoiding a bias in favourof solutions generated by the evaluated systems.

Submitted Systems

We received six submissions to the pronounfocused translation task, and there are system descriptions for five of them. Four submissions werephrase-based SMT systems, three of which werebased on the baseline described in Section 4.1. One
was a rule-based MT system using a completely
different approach to machine translation.
The IDIAP (Luong et al., 2015) and the AUTO POST EDI T (Guillou, 2015) submissions werephrase-based, built using the same training and tuning resources and methods as the official baseline.
Both adopted a two-pass approach involving an
automatic post-editing step to correct the pronoun
translations output by the baseline system, and bothof them relied on the Stanford anaphora resolution
software (Lee et al., 2011). They differed in theway the correct pronoun was assigned: the IDIAP
submission used a classifier with features that included properties of the hypothesized antecedent
together with the output of the baseline system,
whereas the AUTO - POST EDI T system followed a
simpler rule-based decision procedure.
The UU - TIEDEMANN system (Tiedemann, 2015)
was another phrase-based SMT system extending
the official baseline. In contrast to the other submissions, it made no attempt to resolve pronominal
anaphora explicitly. Instead, it used the Docent
document-level decoder (Hardmeier et al., 2013a)with a cross-sentence n-gram model over determiners and pronouns to bias the SMT model towards
selecting correct pronouns.
The UU - HARDMEIER system (Hardmeier, 2015)
was yet another phrase-based SMT using Docent,but built on a different baseline configuration. It
included a neural network classifier for pronoun
prediction trained with latent anaphora resolution(Hardmeier et al., 2013b), but using the Stanford
coreference resolution software at test time.

4 While discourse-aware MT evaluation metrics were proposed recently (Guzmán et al., 2014b; Joty et al., 2014;
Guzmán et al., 2014a), they do not specifically focus on pronoun translation.

5

Machine Translation Evaluation

http://stp.lingfil.uu.se/~ch/DiscoMT2015.maneval/index.php

Machine Translation Evaluation (Annotator: Christian)Source:Translation:

This is a program called Boundless Informant .

C' est un programme appelé illimitée informateur .

What is that ?

Qu' est-ce que c' est ?

So , I 've got to give credit to the NSA for using appropriate
names on this .

Donc , je dois donner crédit à la NSA pour noms appropriées à cesujet .

This is one of my favorite NSA cryptonyms .

C' est une de mes préférées NSA cryptonyms .

Boundless Informant is a program that the NSA hid from
Congress .

Bornes informateur est un programme que la NSA a caché duCongrès .

The NSA was previously asked by Congress , was there any
ability that they had to even give a rough ballpark estimate of
the amount of American communications They said no . They
said , we don 't track those stats , and we can 't track those stats.

La NSA avait auparavant demandé par le Congrès , a-t-on capacité
qu' ils devaient même donner une estimation de la quantité deBallpark américain des communications , ils ont dit non . XXX ont
dit , on ne voie ces statistiques , et nous ne pouvons pas suivre ces
statistiques .

Select the correct pronoun:
il

elle

Other

il

ils

elles

Bad translation

elle

ils

ce

on

il/ce

ça/celaDiscussion required

elles

ce

ça/celaonMultiple options possible

Figure 1: The web interface used for annotation.

Previous example: -- / Current example: 534
2/196 examples annotated.

This is particularly relevant when the overall
We believe there are two reasons for this. On thequality of the translations is imperfect and the eval- one hand, the MT output came from a number ofGuidelines
uators might be tempted to accept the existing solu- systems of widely varying quality, while previous
For each example, you are presented with up to 5 sentences of English source text and a corresponding French machine translation. In thetion if it looks
remotely plausible. Moreover, thiswork considered different variants of a single syslast sentence, an English pronoun is marked up in red, and (in most cases) the French translation contains a red placeholder for a pronoun.
You
are
asked
to
selectapronoun
that
fits
in
the
context.
form of annotation creates a dataset of correct pro- tem. Achieving consistent annotation turned out toPleaseinselectpronoun thatof
shouldbe inserted
in theFrench textof the placeholdermore
XXX todifficult
create the mostnoun translations
thethecontext
MToutputthatbeinsteadconsiderably
forfluent
the lower-qualitytranslation possible while preserving the meaning of the English sentence as much as possible.
can be used in Iffuture
and thatcompletions
would arebeavailable,
very selectsystems.
the other
unlikedifferent,work
equally grammatical
the appropriateOn
checkboxes
and clickhand,on "Multiple
optionsthe annotators
possible". The buttons "il/ce" and "ça/cela" are special shortcuts for cases where these two options are possible.
difficult to obtainotherwise.
usedbyHardmeier(2014),
ours
had a linguistic
Select "Other" if the sentence should be completed with a pronoun not included in the list.
Select "Bad translation" if there is no way to create a grammaticalbackgroundand faithful translation
without
making majorbutchangesto theastranslators,
notin MT. This is
In the annotationinterface, the pronouns ça andsurrounding text.
Select "Discussion
required" if
an example
should be thetaken up forprobably
discussion.
an advantage as far as unbiased annotacela were merged
into a single
class
becauseMinor disfluencies (e.g., incorrect verb agreement or obviously missing words) can be ignored. For instance, if the placeholder should
tions
are
concerned,
but it may have increased theannotators foundthemselves
unabletomakeaconbe replaced with the words c'est, just select "ce".
You should always try to select the pronoun that best agrees with the antecedent in the machine translation, even if the antecedent isinitial
time
to
get
used
tosurrounding
the taskwords
andsuchitsas purpose.
sistent and principled
distinction
thetwo the pronoun's agreement with the immediatelytranslated incorrectly,
and even between
if this forces you
to violate
verbs,
adjectives
orparticiples.
So
if
the
antecedentrequiresaplural
form,
but
the
placeholder
occurswithasingularverb,
you should
pronouns, andselectthethegrammarbooks we consulted
We computed inter-annotator agreementin terms
correct plural pronoun and ignore the agreement error.
Ifthe
Frenchtranslation
doesn'tcontainaplaceholder,youshouldcheck
ifapronouncorresponding
to
the
one
markedup
inthe(Grevisse and Goosse, 1993; Boysen, 1996) did notof Krippendorff’s α (Krippendorff, 2004) andEnglish source should be inserted somewhere and indicate which if so.
offer enough guidance
createreliable
Scott’s
π correct(Scott,1955),
the NLTK
toolkitIf the French totranslation
doesn'tcontain a guidelines.
placeholder, but it alreadyincludes the
pronoun(usually anusing
object pronounlike le,la or les), you should annotate the example as if there had been a placeholder instead of the pronoun (i.e., click on "Other" in the caseMoreover, the annotationinterfaceallowedthean(Bird et al., 2009), over 28 examples annotated byof an object pronoun).
notators to select BAD TRANSLATION if the MT
the two annotators. After two rounds of discusDecision help for difficult cases
output was not sufficiently well-formed to be annot- sion and evaluation, we reached an agreement of
Always remember that the goal of the annotation is to create a standard to evaluate a machine translation system by.
ated with a pronoun. However, they were instructed
α = 0.561 and π = 0.574. These agreement figto be tolerant of ill-formed translations and to use
ures are lower than those reported by Hardmeier
the label
BAD TRANSLATION only if it was neces(2014, 149), which we believe is mostly
due to the
1 of 215.07.15 16:59
sary to make more than two modifications to thefactors discussed above. Some of the disagreement
sentence, in addition to filling in the placeholder, also seems to stem from the annotators’ different
to make the output locally grammatical.
propensity to annotate examples with demonstratIn earlier work, Hardmeier (2014) reported an
ive pronouns. This point was addressed in discusannotation speed of about 60 examples per hour. sions with the annotators, but we did not have time
While our annotators approached that figure after
for another round of formal annotator training andcompleted training, the average speed over the en- agreement evaluation. We do not believe this had atire annotation period was about one third lower
major negative effect on the MT evaluation quality
in this work, mostly because it proved to be moresince, in most cases where the annotators disagreed
difficult than anticipated to settle on a consistent
about whether to annotate ça/cela, the alternative
set of guidelines and reach an acceptable level of
personal pronoun would be annotated consistently
inter-annotator agreement.
if a personal pronoun was acceptable.
6In case of insurmountable difficulties, the annotators had the option to mark an example with thelabel DISCUSSION REQUIRED. Such cases wereresolved at the end of the annotation process.
In total, we annotated 210 examples for each of
the six submitted systems as well as for the official
baseline system. The examples were paired acrossall systems, so the same set of English pronouns
was annotated for each system. In addition, thesample was stratified to ensure that all pronoun
types were represented adequately. The stratification was performed by looking at the pronounsaligned to the English pronouns in the referencetranslation and separately selecting a sample ofeach pronoun class (according to Table 1) in proportion to its relative frequency in the complete test
set. When rounding the individual sample sizes
to integer values, we gave slight preference to therarer classes by rounding the sample sizes upwards
for the less frequent and downwards for the more
frequent classes.
After completing the human evaluation, we calculated a set of evaluation scores by counting how
often the output of a particular system matched themanual annotation specific to that system. Thisis straightforward for the annotation labels corresponding to actual pronouns (ce, ça/cela, elle,
elles, il, ils and on). The examples labelled as BAD
TRANSLATION were counted as incorrect. The
label OTHER leads to complications because thislabel lumps together many different cases such as
the use of a pronoun not available as an explicit label, the complete absence of a pronoun translationon the target side, the translation of a pronoun witha full noun phrase or other linguistic construct, etc.
As a result, even if the MT output of an example
annotated as OTHER contains a translation that iscompatible with this annotation, we cannot be surethat it is in fact correct. This must be kept in mind
when interpreting aggregate metrics based on ourannotations.
The evaluation scores based on manual annotations are defined as follows:

Accuracy without OTHER (Acc–O)
This isan accuracy score computed only over thoseexamples that are not labelled OTHER, so it
does not suffer from the problem describedabove. However, the set of examples annotated as OTHER differs between systems, which
could in theory be exploited by a system to increase its score artificially, e.g., by predicting
OTHER for all hard cases. In practice, it is very
unlikely that this happened in this evaluation
since details about the evaluation modalities
were not known to the participants at submission time.

Accuracy with OTHER (Acc+O)
Our primaryevaluation score is accuracy over all 210 examples, i.e., the proportion of examples forwhich the pronouns in the MT output are compatible with those in the manual annotation.
We include items labelled OTHER and countthem as correct if the MT output contains any
realisation compatible with that label.

In addition to the above manual evaluation
scores, we also computed automatic scores(Table 5). This includes the pronoun precision/recall scores as defined by Hardmeier andFederico (2010), as well as four standard MT evaluation metrics: BLEU (Papineni et al., 2002), NIST
(Doddington, 2002), TER (Snover et al., 2006), andMETEOR (Denkowski and Lavie, 2011).

Pronoun-specific Fmax -score
To permit amore fine-grained interpretation of the evaluation results, we also computed individual
precision, recall and F-score values for each of
the pronoun labels available to the annotators(excluding OTHER and BAD TRANSLATION).
Since multiple correct choices are possible foreach example, an example need not (and cannot) match each of the annotated pronouns to
be correct. To account for this, we operatewith a non-standard definition of recall, which
we call Rmax because it can be interpreted asa sort of upper bound on the “intuitive” notion of recall. Rmax for a given type of pronoun counts as matches all correct examples
labelled with a given pronoun type, even if theactual pronoun used is different. To illustrate,
suppose an example is annotated with il andce, and the MT output has ce. This example
would be counted as a hit for the Rmax of bothpronoun types, il and ce. The Fmax score in
Table 6 is the harmonic mean of standard precision and Rmax .
Pron-F The fine-grained precision and recall
scores give rise to another aggregate measure,labelled Pron-F in Table 6, which is an F-scorebased on the micro-averaged precision andrecall values of all pronoun types.

7

Pronoun Evaluation
P
R
F
BASELINE
IDIAP
UU - TIEDEMANNUU - HARDMEIER
AUTO - POST EDI T
ITS 2
A 3-108

0.371
0.346
0.386
0.347
0.329
0.184
0.054

0.361
0.333
0.353
0.333
0.276
0.187
0.045

Standard MT Evaluation Metrics
BLEU
NIST
TER METEOR

0.366
0.340
0.369
0.340
0.300
0.188
0.049

37.18
36.42
36.92
32.58
36.91
20.94
4.06

8.04
7.89
8.02
7.66
7.98
5.96
2.77

46.74
48.18
46.93
49.04
46.94
60.95
88.49

60.05
59.26
59.92
57.50
59.70
47.90
25.59

Table 5: Pronoun-focused translation task: automatic metrics.

BASELINE
IDIAP
UU - TIEDEMANNUU - HARDMEIER
AUTO - POST EDI T
ITS 2
A 3-108

Acc+O

Acc–O Pron-F

ceFmax Scores for Individual Pronouns
ça/cela
elle
elles
il
ils

on

0.676
0.657
0.643
0.581
0.543
0.419
0.081

0.630
0.617
0.590
0.525
0.473
0.339
0.081

0.832
0.842
0.781
0.765
0.496
∅
0.368

0.631
0.703
0.573
0.521
0.238
∅
0.149

∅
∅
∅
∅
∅
∅
∅

0.699
0.711
0.675
0.580
0.523
0.396
0.188

0.452
0.336
0.516
0.207
0.304
0.256
0.000

0.436
0.545
0.462
0.421
0.396
0.353
0.000

0.522
0.600
0.402
0.254
0.422
0.373
∅

0.900
0.848
0.891
0.882
0.869
0.782
0.271

Acc+O: Accuracy with OTHER Acc–O: Accuracy without OTHER Pron-F: micro-averaged pronoun F-score∅: this pronoun type that was never predicted by the system

Table 6: Pronoun-focused translation task: manual evaluation metrics.
4.4

Evaluation Results

The results for the manual evaluation are shownin Table 6: we show aggregate scores such as accuracy, with and without OTHER, as well as Fmaxscores for the individual pronouns. We have chosen
Acc+O to be the primary metric because it is welldefined as it is calculated on the same instances
for all participating systems, so it cannot be easily exploited by manipulating the system output inclever ways. It turns out, however, that the rankingsof our participating systems induced by this score
and the Acc–O score are exactly identical. In bothcases, the BASELINE system leads, followed relatively closely by IDIAP and UU - TIEDEMANN. Then,UU - HARDMEIER and AUTO - POST EDI T follow at aslightly larger distance, and finally A 3-108 scoresat the bottom. The micro-averaged Pron-F score
would have yielded the same ranking as well, except for the first two systems, where IDIAP would
have taken the lead from the BASELINE. This isdue to the fact that the IDIAP system has a higher
number of examples labelled BAD TRANSLATION,
while maintaining the same performance as thebaseline for the examples with acceptable translations. Rather than implying much about the qualityof the systems, this observation confirms and justifies our decision to choose a primary score thatis not susceptible to effects arising from excluded
classes.

The standard automatic MT evaluation scores(BLEU, NIST, TER, METEOR; Table 5) do not offer specific insights about pronoun translation, but
it is still useful to consider them first for an easy
overview over the submitted systems. They clearly
reveal a group of systems (IDIAP, UU - TIEDEMANN
and AUTO - POST EDI T) built with the data of theofficial BASELINE system, with very similar scores
ranging between 36.4 and 37.2 BLEU points. The
baseline itself achieves the best scores, but considering the inadequacy of BLEU for pronoun evaluation, we do not see this as a major concern in itself.
The other submissions fall behind in terms of automatic MT metrics. The UU - HARDMEIER system issimilar to the other SMT systems, but uses different
language and translation models, which evidently
do not yield the same level of raw MT performance as the baseline system. ITS 2 is a rule-based
system. Since it is well known that n-gram-based
evaluation metrics do not always do full justice
to rule-based MT approaches not using n-gram
language models (Callison-Burch et al., 2006), it
is difficult to draw definite conclusions from thissystem’s lower scores. Finally, the extremely lowscores for the A 3-108 system indicate serious problems with translation quality, an impression that weeasily confirmed by examining the system output.
8

The low scores for the ITS 2 system were partly
due to a design decision. The anaphora prediction
component of ITS 2 only generated the personalpronouns il, elle, ils and elles; this led to zero recall
for ce and ça/cela and, as a consequence, to a large
number of misses that would have been comparatively easy to predict with an n-gram model.

5

Cross-Lingual Pronoun Prediction

5.1

Data Preparation

For the second task, cross-lingual pronoun translation, we used the same bitext as for the MT baselinein the first task (Section 4.1); we pre-processed it
like before, except for lowercasing. Then, we generated the following two resources: (i) a bitext
with target pronouns identified and their translations removed, and (ii) word alignments between
the source and the target sentences in the bitext.
Since the word alignments in the training andin the testing datasets were created automatically, without manual inspection, we performed asmall study in order to investigate which alignment method performed best for pronouns. We
followed the methodology in Stymne et al. (2014),
by aligning English–French data using all IBM
models (Brown et al., 1993) and the HMM model(Vogel et al., 1996) as implemented in GIZA++(Och and Ney, 2003), as well as fast_align (Dyeret al., 2013), with a number of different symmetrization methods. IBM models 1, 2 and 3 yieldedsubpar results, so we will not discuss them.
To evaluate the alignments, we used 484 goldaligned sentences from Och and Ney (2000).5 We
used the F-score of correct sure and possible links(Fraser and Marcu, 2007) for a general evaluation,
which we will call Fall .6 In order to specificallyevaluate pronoun alignment, we used the F-scoreof the subset of links that align the two sets of pronouns we are interested in, Fpro . For all alignment
models, grow-diag-final-and symmetrization performed best on the pronoun metric, followed bygrow-diag and intersection, which also performed
best for general alignments.
Table 7 shows the results for different models
with grow-diag-final-and symmetrization. We cansee that, for all three models, the results on pronounlinks are better than those on all links. Moreover,IBM model 4 and HMM are better than fast_alignboth for general alignments and for pronoun alignments. In the final system, we chose to use IBMmodel 4 since it finds slightly more possible links
than HMM. Overall, we find the results very good.
In the best system, all pronoun links except for one
possible link were found, and there are only four
pronoun links that are not in the gold standard.

There does not seem to be a correlation between
pronoun translation quality and the choice of (a) atwo-pass approach with automatic post-editing(IDIAP, AUTO - POST EDI T) or (b) a single-pass
SMT system with some form of integrated pronoun model (UU - TIEDEMANN, UU - HARDMEIER).
Also, at the level of performance that current systems achieve, there does not seem to be an inherent advantage or disadvantage in doing explicit
anaphora resolution (as IDIAP, UU - HARDMEIER,
AUTO - POST EDI T and ITS 2 did) as opposed toconsidering unstructured context only (as in UU TIEDEMANN and the BASELINE ).
One conclusion that is supported by relatively
ample evidence in the results concerns the importance of the n-gram language model. The BASELINE
system, which only relies on n-gram modellingto choose the pronouns, achieved scores higher
than those of all competing systems. Moreover,even among the submitted systems that included
some form of pronoun model, those that reliedmost on the standard SMT models performed best.
For example, the IDIAP submission exploited the
SMT decoder’s translation hypotheses by parsing
the search graph, and UU - TIEDEMANN extended
the baseline configuration with additional n-gramstyle models. By contrast, those systems that actively overrode the choices of the baseline n-grammodel (UU - HARDMEIER and AUTO - POST EDI T)
performed much worse.
Based on these somewhat depressing results, one
might be tempted to conclude that all comparison
between the submitted systems is meaningless because all they managed to accomplish was to “disfigure” the output of a working baseline system tovarious degrees. Yet, we should point out that it
was possible for some systems to outperform thebaseline at least for some of the rarer pronouns. In
particular, the IDIAP system beat the baseline on
4 out of 6 pronoun types, including the feminine
plural pronoun elles, and the UU - TIEDEMANN system performed better on both types of feminine
pronouns, elle and elles. Results like these suggestthat all hope is not lost.

5 Downloaded from http://www.cse.unt.edu/~rada/
wpt/index.html
6 F is equivalent to 1 − AER, Alignment Error Rate (Ochalland Ney, 2003).

9

Alignment

Fall

Fpro

GIZA++, HMM
GIZA++, Model 4
fast_align

0.93
0.92
0.86

0.96
0.96
0.93

In the unlikely case that there is no alignment
link in the neighbourhood of the pronoun, the placeholder will be inserted at a similar position as thesource language position or at the end of the segment before any punctuation.
The test data were prepared in the same way
but with empty columns for the classes and thesubstitution strings. We also provided information about the document boundaries in each dataset.
For Europarl, we included file names, sentence
IDs and annotations such as SPEAKER and paragraph boundaries. For the News Commentaries, we
supplied document IDs and paragraph boundaries.
Finally, the IWSLT data included the TED talk IDs.
Table 8 shows the distribution of classes in thethree training datasets and the official test dataset.
We can see that there are significant differences
between the different genres with respect to pronoun distributions.

Table 7: F-score for all alignment links (Fall ), andfor pronoun links (Fpro ), for different alignment
models with grow-diag-final-and symmetrization.
Ultimately, we applied GIZA++ with grow-diagfinal-and symmetrization and we used fast_align asa backoff alignment method for the cases that could
not be handled by GIZA++ (sentences longer than
100 tokens and sentence pairs with unusual length
ratios). This was necessary in order to align thefull bitext without missing any sentence pair in the
discourse, as all sentences may contain valuable
information for the classifier.
We developed a script that takes the wordaligned bitext and replaces the tokens that arealigned with the English target pronouns it and they
with placeholders, keeping the information about
the substitutions for training and evaluation purposes. Note that the substitutions are always single
words. Pronouns corresponding to one of the target
classes were preferred among the aligned tokens. If
none of the tokens matched any of the classes, we
kept the shortest aligned word as the substitution
and set the class to OTHER. We marked the unaligned words with the substitution string “NONE”.
Figure 2 shows two examples of training instancesthat we created.
The final data contains five TAB-separated
columns for each aligned segment pair from thebitext: (1) the classes to be predicted in the sameorder as they appear in the text (may be empty),(2) the actual tokens that have been substituted,(3) the source language segment, (4) the target language segment with placeholders, and (5) the wordalignment. The placeholders have the format RE PLACE _XX where XX refers to the index (starting with 0) of the English token that is aligned
to the placeholder. We normalized instances ofc’ and ca to ce and ça, respectively. The substituted tokens are case-sensitive and the class OTHER
also includes empty alignments. For the latter, wedeveloped a strategy that inserts placeholders at areasonable position into the target language segment by looking at the alignment positions of thesurrounding words of the selected English pronoun
and then putting the placeholder next to the closest
link in the target sentence.

class
ça
ce
cela
elle
elles
il
ils
on
OTHER

DiscoMT
2015

IWSLT14

Training
Europarl

News

102
184
27
83
51
104
160
37
357

4,548
14,555
2,256
2,999
2,888
8,467
14,8981,410
25,394412
52,964
13,44750,254
18,543
166,873
45,985
9,871
231,230

39
2,873
1,025
4,363
1,929
8,059
7,433
566
14,969

Table 8: Distribution of classes in the DiscoMT
2015 test set and the three training datasets.
5.2

Baseline System

The baseline system tries to reproduce the most
realistic scenario for a phrase-based SMT system
assuming that the amount of information that canbe extracted from the translation table is not sufficient or is inconclusive. In that case, the pronoun
prediction would be influenced primarily by thelanguage model.
Thus, our baseline is based on a language model.
It fills the gaps by using a fixed set of pronouns(those to be predicted) and a fixed set of nonpronouns (which includes the most frequent itemsaligned with a pronoun in the provided test set) as
well as NONE (i.e., do not insert anything in the
hypothesis), with a configurable NONE penalty that
accounts for the fact that n-gram language models
tend to assign higher probability to shorter strings
than to longer ones.
10

classes
substitutions
sourcetargetalignment

ils ceils c’
Even though they were labeled whale meat , they were dolphin meat .
Même si REPLACE _2 avaient été étiquettés viande de baleine , REPLACE _8 était de la viande de dauphin .
0-0 1-1 2-2 3-3 3-4 4-5 5-8 6-6 6-7 7-9 8-10 9-11 10-16 11 -13 11-14 12-17classes
substitutions
sourcetargetalignment

ils OTHER
ils NONEBut they agreed to go along with it for a while .
Mais REPLACE _1 ont accepté de suivre REPLACE _7 pendant un temps .
0-0 1-1 1-2 2-3 3-4 4-5 5-5 6-5 7-6 8-7 9-8 10-9 11-10

Figure 2: Examples from the training data for the cross-lingual pronoun prediction task.
Development set

Test set

60

60

●

●

●

●

●

●

●

●
●
●
●

●

●

●

●

●

●

●

●

macro−averaged F−scoremacro−averaged F−score

●

●
●
●

40
●

●

n−gram order
●

20

●

6−gram

●

●

40

●
●

n−gram order
●

20

●

5−gram●6−gram5−gram●4−gram4−gram●3−gram−43−gram
−2

0

2

4

−4

NONE penalty

−2

0

2

4

NONE penalty

Figure 3: Performance of the baseline cross-lingual pronoun prediction system as a function of the NONE
penalty and the n-gram order. Shown are results on the development and on the test datasets.
The official baseline score was computed with
the NONE penalty set to an unoptimized default
value of 0. We used the same 5-gram language
model that was part of the baseline for the pronounfocused translation task, constructed with news
texts, parliament debates, and the TED talks ofthe training/development portion.

It is interesting that the performance of a 3-grammodel is very similar on the development and on
the test set. Increasing the n-gram order has almost no effect for the development set, but for thetest set it yields substantial gains in terms of both
macro-averaged F-score (see Figure 3) and accuracy (not shown here). We plan a detailed analysisof this in future work, but one hypothesis is that itis due to the test set’s better coverage of infrequent
pronouns.

After completing the evaluation, we ran additional experiments to analyze the effect of the
NONE penalty and the order of the n-gram modelon the performance of the baseline system. The
results are shown on Figure 3, where we can seethat the optimal value for the NONE penalty, bothfor the development and for the test set, would havebeen around −2. This is expected, since a negative
penalty value penalizes the omission of pronouns
in the output. The system works robustly for a wide
variety of negative penalty values, but if the penalty
is set to a positive value, which encourages pronoun
omission, the performance degrades quickly.

Overall, the language model baseline is surprisingly strong since the following (or preceding) verb
group often contains information about number,
gender, obliqueness, and animacy. It goes without
saying that much of this information is not present
in an actual MT system, which would have as muchdifficulty reconstructing number and gender information in verb groups as in argument pronouns.
Thus, to achieve a good score, systems have to use
both source-side and target-side information.
11

5.3

Submitted Systems

5.4

Evaluation

For the automatic evaluation, we developed a scoring script that calculates the following statistics:

For the cross-lingual pronoun prediction task, we
received submissions from eight groups. Some ofthem also submitted a second, contrastive run. Sixof the groups submitted system description papers,
and one of the two remaining groups formally withdrew its submission after evaluation.

• confusion matrix showing (i) the count foreach gold/predicted pair, and (ii) the sums foreach row/column;• accuracy;

All six groups with system description papers
used some form of machine learning. The main difference was whether or not they explicitly attempted to resolve pronominal coreference. Two systems relied on explicit anaphora resolution: UEDIN
and MALTA. They both applied the Stanford coreference resolver (Lee et al., 2011) on the source
language text, then projected the antecedents to thetarget language through the word alignments, andfinally obtained morphological tags with the Morfette software (Chrupała et al., 2008). The UEDIN
system (Wetzel et al., 2015) was built around a maximum entropy classifier. In addition to local context
and antecedent information, it used the NADA tool
(Bergsma and Yarowsky, 2011) to identify nonreferring pronouns and included predictions by astandard n-gram language model as a feature. The
MALTA system (Pham and van der Plas, 2015) was
based on a feed-forward neural network combinedwith word2vec continuous-space word embeddings(Mikolov et al., 2013). It used local context andantecedent information.

• precision (P), recall (R), and F-score for eachlabel;• micro-averaged P, R, F-score (note that in our
setup, micro-F is the same as accuracy);• macro-averaged P, R, F-score.
The script performs the scoring twice:• using coarse-grained labels (ce, {cela+ça},
elle, elles, il, ils, {OTHER+on});• using fine-grained labels (ce, cela, elle, elles,
il, ils, on, ça, OTHER).
The official score was the macro-averaged Fscore using fine-grained labels.
5.5Discussion

The results for the cross-lingual pronoun prediction
task are shown in Table 9. The table includes thescores for both the primary and the secondary submissions; the latter are marked with 2. The threehighest scores in each column are marked in boldface. The official score was the macro-averaged
F-score, which is reported in the second column.
As in the first subtask (the pronoun-focusedtranslation task), we find that the baseline system, BASELINE - NP 0 (here a simple n-gram-basedmodel) outperformed all the participating systems
on the official macro-averaged F-score. Note thatthe performance of the baseline depends on the
NONE penalty; we set this parameter to 0, a default
value which we did not optimize in any way.
Immediately following the baseline, there areseveral systems with macro-averaged F-scores ranging between 0.55 and 0.58 (Table 9). This seems
to mark the level of performance that is achievablewith the methods currently at our disposal.
We should note that while our baseline systemoutperformed all submissions, both primary andsecondary, in terms of macro-averaged F-score,
several systems performed better in terms of accuracy.

The other systems did not use explicit anaphoraresolution, but attempted to gather relevant information about possible antecedents by consideringa certain number of preceding, or preceding andfollowing, noun phrases. They differed in the typeof classifier and in the information sources used.
UU - TIEDEMANN (Tiedemann, 2015) used a linear support vector machine with local features andsimple surface features derived from preceding
noun phrases. WHATELLES (Callin et al., 2015)
used a neural network classifier based on work byHardmeier et al. (2013b), but replacing all (explicit or latent) anaphora resolution with information
extracted from preceding noun phrases. The IDIAP
system (Luong et al., 2015) used a Naïve Bayesclassifier and extracted features from both preceding and following noun phrases to account for the
possibility of cataphoric references. The GENEVA
system (Loáiciga, 2015) used maximum entropy
classification; unlike the other submissions, it included features derived from syntactic parse trees.
122: secondary submission

BASELINE - NP 0
UU - TIED
UEDIN
MALTA
2
MALTA
WHATELLES
UEDIN
2UU - TIED
2
GENEVAGENEVA
2
IDIAP
IDIAP
2
A 3-108( WITHDRAWN )Macro-F

Accuracy

ce

cela

elle

elles

F-score
il

ils

on

ça

OTHER

0.584
0.579
0.571
0.565
0.561
0.553
0.550
0.539
0.437
0.421
0.206
0.164
0.129
0.122

0.663
0.742
0.723
0.740
0.732
0.721
0.714
0.734
0.592
0.579
0.307
0.407
0.240
0.325

0.817
0.862
0.823
0.875
0.853
0.862
0.823
0.849
0.647
0.611
0.282
0.152
0.225
0.220

0.346
0.235
0.213
0.111
0.071
0.156
0.083
0.125
0.197
0.147
0.000
0.000
0.000
0.000

0.511
0.326
0.417
0.378
0.368
0.346
0.382
0.283
0.365
0.353
0.235
0.000
0.020
0.000

0.507
0.389
0.479
0.359
0.420
0.436
0.451
0.242
0.321
0.313
0.205
0.000
0.033
0.000

0.480
0.558
0.544
0.588
0.579
0.561
0.573
0.545
0.475
0.442
0.164
0.065
0.132
0.187

0.745
0.828
0.834
0.828
0.829
0.830
0.823
0.838
0.761
0.759
0.429
0.668
0.246
0.134

0.571
0.557
0.475
0.537
0.448
0.451
0.448
0.516
0.340
0.310
0.000
0.000
0.047
0.000

0.539
0.557
0.497
0.494
0.585
0.452
0.523
0.551
0.075
0.092
0.149
0.072
0.067
0.000

0.739
0.901
0.855
0.917
0.898
0.882
0.840
0.902
0.757
0.759
0.391
0.518
0.391
0.555

Table 9: Results for the cross-lingual pronoun prediction task.
The reason why we chose macro-averaged Fscore rather than accuracy as our primary metricis that it places more weight on the rare categories: we wanted to reward efforts to improve theperformance for the rare pronouns such as elles.
This choice was motivated by the findings of Hardmeier et al. (2013b), who observed that the performance on the rare classes strongly depended on
the classifier’s capacity to make use of coreference
information. It is worth noting that none of theirclassifiers used target language n-gram information.
Yet, in our shared task, we observed that our n-grambaseline, despite having no access to antecedent information beyond the extent of the n-gram window,
performed better than systems that did have accessto such information; this was especially true forclasses such as elle and elles, which supposedly
require knowledge about antecedents.

Finally, it is interesting to note that systems
with similar overall performance perform very
differently on individual pronoun classes. UU TIEDEMANN, which is the second-best submission
after the baseline in terms of both macro-averaged
F-score and accuracy, is very strong on all classes
except for personal pronouns, that is, the classes
ce, cela, on, and ça. In contrast, the third-bestsystem, UEDIN is much stronger on elle and elles.
Without additional experiments, it is impossible
to say whether this is due to its use of anaphoraresolution or to some other factors.

6

Conclusions

We have described the design and evaluation of theshared task at DiscoMT 2015, which included two
different, but related subtasks, focusing on the difficulty of handling pronouns in MT. We preparedand released training and testing datasets, evaluation tools, and baseline systems for both subtasks,
making it relatively easy to join. This effort was
rewarded by the attention that the task attracted
in the community. With six primary submissions
to the pronoun-focused translation task, and eightto the cross-lingual pronoun prediction task, wefeel that the acceptance of the task was high andthat our goal of establishing the state of the art inpronoun-aware MT has been accomplished.
The results suggest that the problem of pronoun
translation is far from solved. Even for crosslingual pronoun prediction, where the entire translation of the input, except for the translations of
the pronouns, is given, none of the participating
systems reached an accuracy of more than 75 % ora macro-averaged F-score of more than 60 %.

While a detailed analysis of this observationmust be deferred to future work, we can think of
two possible explanations. On the one hand, evenafter removing the pronoun translations, there remains enough information about gender and number in the inflections of the surrounding words, andn-gram models are very good at picking up on thissort of information. Thus, the presence of a nearby
adjective or participle with feminine inflection may
be enough for an n-gram model to make the rightguess about the translation of a pronoun.
On the other hand, there is evidence that n-grammodels are very good at recognising the typical,
rather than the actual, antecedent of a pronounbased on context features (Hardmeier, 2014, 137–
138). This may be another factor contributing to
the good performance of the n-gram baseline.
13

Acknowledgements

In other words, even though the actual challengeof translating the source text was completely removed from the task, and despite the focused efforts of eight groups, we still find ourselves in asituation where one pronoun in four was predictedincorrectly by the best-performing system.

The manual evaluation of the pronoun-focusedtranslation task was made possible thanks to a
grant from the European Association for MachineTranslation (EAMT). We gratefully acknowledge
the help of our annotators, Charlotta Jonasson
and Anna Lernefalk, and we are indebted to Bonnie Webber for proofreading a draft of this paper.
CH and JT were supported by the Swedish Research Council under project 2012-916 DiscourseOriented Machine Translation. The work of CH,
SS, and JT is part of the Swedish strategic research
programme eSSENCE. MC was supported by the
CRACKER project, which received funding fromthe European Union’s Horizon 2020 research andinnovation programme under grant no. 645357.

This tells us something about the difficulty ofthe task: In the real world, an MT system has togenerate hypotheses not only for the translation of
pronouns, but also for the full text. Many clues
that are successfully exploited by the pronoun prediction systems, such as word inflections in theneighbourhood of the pronouns, cannot be relied
on in an MT setting because they must be generated by the MT system itself and are likely to beabsent or incorrect before the translation process
is completed. If it is difficult to choose the correct
pronoun given the entire target language context,
this should be even more challenging in MT.

References

In both tasks, the baseline systems, whose
strongest components are standard n-gram models, outperformed all submissions on the official
metrics. This suggests that there are aspects ofthe pronoun generation problem, and possibly of ngram models, that we do not fully understand. As afirst step towards deeper analysis of the shared task
results, it will be necessary to study why n-grammodels perform better than systems specificallytargetting pronoun translation. In the pronoun prediction task, they may exploit local context clues
more aggressively, while the submitted classifiers,
designed with MT applications and unreliable context in mind, tend to make incomplete use of thisreadily available information. However, while thismay be a reason for the good performance of thebaseline in the prediction task, it does not explain
the results for the pronoun-focused translation task.

Shane Bergsma and David Yarowsky. 2011. NADA:A robust system for non-referential pronoun detection. In Proceedings of the 8th Discourse Anaphoraand Anaphor Resolution Colloquium, volume 7099
of Lecture Notes in Computer Science, pages 12–23,
Faro, Portugal.
Steven Bird, Edward Loper, and Ewan Klein.
2009. Natural Language Processing with Python.
O’Reilly, Beijing.
Ondrej Bojar, Christian Buck, Christian Federmann,
Barry Haddow, Philipp Koehn, Johannes Leveling,
Christof Monz, Pavel Pecina, Matt Post, Herve
Saint-Amand, Radu Soricut, Lucia Specia, and Aleš
Tamchyna. 2014. Findings of the 2014 workshop
on statistical machine translation. In Proceedings of
the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA.
Gerhard Boysen. 1996. Fransk grammatik. Studentlitteratur, Lund.
Peter F. Brown, Stephen A. Della Pietra, Vincent J.
Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation. Computational Linguistics, 19(2):263–311.

In any case, while this shared task has not revealed a substantially better method for pronoun
translation than a plain n-gram model, we should
certainly not conclude that n-gram models are sufficient for this task. In the pronoun-focused translation task, all systems, including the baseline, had error rates of 1 in 3 or higher, which confirms earlier
findings showing that pronoun translation is indeeda serious problem for SMT (Hardmeier and Federico, 2010; Scherrer et al., 2011). We should
therefore see the results of this shared task as anincentive to continue research on pronoun translation. We believe that our resources, methods andfindings will prove useful for this endeavour.

Jimmy Callin, Christian Hardmeier, and Jörg
Tiedemann. 2015. Part-of-speech driven crosslingual pronoun prediction with feed-forward neuralnetworks. In Proceedings of the Second Workshop
on Discourse in Machine Translation, pages 59–64,
Lisbon, Portugal.
Chris Callison-Burch, Miles Osborne, and Philipp
Koehn. 2006. Re-evaluating the role of BLEU inmachine translation research. In Proceedings of the
11th Conference of the European Chapter of the Association for Computational Linguistics, pages 249–
256, Trento, Italy.

14

Francisco Guzmán, Shafiq Joty, Lluís Màrquez, and
Preslav Nakov. 2014b. Using discourse structure
improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), pages 687–698, Baltimore, Maryland,
USA.

Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT3 : Web inventory of transcribed
and translated talks. In Proceedings of the 16th Conference of the European Association for MachineTranslation (EAMT), pages 261–268, Trento, Italy.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa
Bentivogli, and Marcello Federico. 2014. Reporton the 11th IWSLT evaluation campaign, IWSLT
2014. In Proceedings of the International Workshop
on Spoken Language Translation, Hanoi, Vietnam.

Christian Hardmeier and Marcello Federico. 2010.
Modelling pronominal anaphora in statistical machine translation. In Proceedings of the Seventh International Workshop on Spoken Language Translation (IWSLT), pages 283–289, Paris, France.

Noam Chomsky. 1981. Lectures on Government andBinding: The Pisa lectures. Mouton de Gruyter.

Christian Hardmeier, Sara Stymne, Jörg Tiedemann,
and Joakim Nivre. 2013a. Docent: A documentlevel decoder for phrase-based statistical machinetranslation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics,
pages 193–198, Sofia, Bulgaria.

Grzegorz Chrupała, Georgiana Dinu, and Josef van
Genabith.
2008.
Learning morphology withMorfette. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pages 2362–2367, Marrakech,
Morocco.
Michael Denkowski and Alon Lavie. 2011. Meteor
1.3: Automatic metric for reliable optimization andevaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91, Edinburgh, Scotland, UK.

Christian Hardmeier, Jörg Tiedemann, and Joakim
Nivre. 2013b. Latent anaphora resolution for crosslingual pronoun prediction. In Proceedings of theConference on Empirical Methods in Natural Language Processing, pages 380–391, Seattle, Washington, USA.

George Doddington. 2002. Automatic evaluationof machine translation quality using n-gram cooccurrence statistics. In Proceedings of the Second
International Conference on Human Language Technology Research, pages 138–145, San Diego, California, USA.

Christian Hardmeier. 2012. Discourse in statistical machine translation: A survey and a case study. Discours, 11.
Christian Hardmeier. 2014. Discourse in Statistical Machine Translation, volume 15 of Studia Linguistica Upsaliensia. Acta Universitatis Upsaliensis, Uppsala.

Chris Dyer, Victor Chahuneau, and Noah A. Smith.
2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 644–648,
Atlanta, Georgia, USA.

Christian Hardmeier. 2015. A document-level SMT
system with integrated pronoun prediction. In Proceedings of the Second Workshop on Discourse in
Machine Translation, pages 72–77, Lisbon, Portugal.

Alexander Fraser and Daniel Marcu. 2007. Measuring
word alignment quality for statistical machine translation. Computational Linguistics, 33(3):293–303.

Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H.
Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 690–696,
Sofia, Bulgaria.

Maurice Grevisse and André Goosse. 1993. Le bon
usage: Grammaire française. Duculot, Paris, 13eédition.

Howard Johnson, Joel Martin, George Foster, and Roland Kuhn. 2007. Improving translation quality by
discarding most of the phrasetable. In Proceedingsof the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 967–975,
Prague, Czech Republic.

Liane Guillou, Christian Hardmeier, Aaron Smith, Jörg
Tiedemann, and Bonnie Webber. 2014. ParCor
1.0: A parallel pronoun-coreference corpus to support statistical MT. In Proceedings of the Tenth
Language Resources and Evaluation Conference(LREC’14), pages 3191–3198, Reykjavík, Iceland.
Liane Guillou. 2015. Automatic post-editing for the
DiscoMT pronoun translation task. In Proceedingsof the Second Workshop on Discourse in MachineTranslation, pages 65–71, Lisbon, Portugal.

Shafiq Joty, Francisco Guzmán, Lluís Màrquez, and
Preslav Nakov. 2014. DiscoTK: Using discourse
structure for machine translation evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 402–408, Baltimore, Maryland, USA.

Francisco Guzmán, Shafiq Joty, Lluís Màrquez, Alessandro Moschitti, Preslav Nakov, and Massimo Nicosia. 2014a. Learning to differentiate better fromworse translations. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing, pages 214–220, Doha, Qatar.

Philipp Koehn, Hieu Hoang, Alexandra Birch, et al.
2007. Moses: Open source toolkit for Statistical Machine Translation. In Annual Meeting of the Associ-15

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: A method for automatic
evaluation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia,
Pennsylvania, USA.

ation for Computational Linguistics: Demonstration
session, pages 177–180, Prague, Czech Republic.
Philipp Koehn. 2005. Europarl: A corpus for statistical machine translation. In Proceedings of MT Summit X, pages 79–86, Phuket, Thailand.
Klaus Krippendorff. 2004. Measuring the reliability ofqualitative text analysis data. Quality and Quantity,
38(6):787–800.

Ngoc-Quan Pham and Lonneke van der Plas. 2015.
Predicting pronouns across languages with continuous word spaces. In Proceedings of the Second
Workshop on Discourse in Machine Translation,
pages 101–107, Lisbon, Portugal.

Ronan Le Nagard and Philipp Koehn. 2010. Aiding
pronoun translation with co-reference resolution. In
Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages
252–261, Uppsala, Sweden.

Yves Scherrer, Lorenza Russo, Jean-Philippe Goldman,
Sharid Loáiciga, Luka Nerima, and Éric Wehrli.
2011. La traduction automatique des pronoms.
Problèmes et perspectives. In Mathieu Lafourcade
and Violaine Prince, editors, Actes de la 18e conférence sur le Traitement Automatique des Langues
Naturelles, volume 2, pages 185–190, Montpellier,France.

Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky.
2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In
Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task,
pages 28–34, Portland, Oregon, USA.

William A. Scott. 1955. Reliability of content analysis:
The case of nominal scale coding. Public Opinion
Quarterly, 19(3):321–325.

Sharid Loáiciga and Eric Wehrli. 2015. Rule-based
pronominal anaphora treatment for machine translation. In Proceedings of the Second Workshop on
Discourse in Machine Translation, pages 86–93, Lisbon, Portugal.

Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A studyof translation edit rate with targeted human annotation. In AMTA 2006: Proceedings of the 7th Conference of the Association for Machine Translationin the Americas, pages 223–231, Cambridge, Massachusetts, USA.

Sharid Loáiciga. 2015. Predicting pronoun translation using syntactic, morphological and contextual
features from parallel data. In Proceedings of theSecond Workshop on Discourse in Machine Translation, pages 78–85, Lisbon, Portugal.

Sara Stymne, Jörg Tiedemann, and Joakim Nivre.
2014. Estimating word alignment quality for SMT
reordering tasks. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 275–
286, Baltimore, Maryland, USA.

Ngoc Quang Luong, Lesly Miculicich Werlen, and Andrei Popescu-Belis. 2015. Pronoun translation andprediction with or without coreference links. In Proceedings of the Second Workshop on Discourse in
Machine Translation, pages 94–100, Lisbon, Portugal.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and
Evaluation (LREC’12), pages 2214–2218, Istanbul,
Turkey.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations: Workshop Proceedings.

Jörg Tiedemann. 2015. Baseline models for pronoun
prediction and pronoun-aware translation. In Proceedings of the Second Workshop on Discourse in
Machine Translation, pages 108–114, Lisbon, Portugal.

Ruslan Mitkov and Catalina Barbu. 2003. Using bilingual corpora to improve pronoun resolution. Languages in Contrast, 4(2):201–211.

Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In Coling 1996: the 16th International Conference on Computational Linguistics,
pages 145–154, Copenhagen, Denmark.

Franz Josef Och and Hermann Ney. 2000. Improved
statistical alignment models. In Proceedings of the
38th Annual Meeting of the Association for Computational Linguistics, pages 440–447, Hongkong,
China.

Dominikus Wetzel, Adam Lopez, and Bonnie Webber.
2015. A maximum entropy classifier for crosslingual pronoun prediction. In Proceedings of theSecond Workshop on Discourse in Machine Translation, pages 115–121, Lisbon, Portugal.

Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.
Franz Josef Och. 2003. Minimum error rate training in
Statistical Machine Translation. In Proceedings ofthe 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan.

16