**1 Executive Summary** 1

This Data Management Plan (DMP) reports on the current state (as of month 6)
of the data QT21 will use and generate during its life. The DMP will be
updated during the project with new releases in months 18 and 30\.

This document follows the structure recommended for all Horizon2020 DMPs.
First by describing the **data selection methodology** before formally
describing the data. The formal data description starts **with a name and a
reference** to the data followed by a **description of the content** of the
data. Further **standards** , data **sharing** and data **archiving** have to
be addressed.

QT21 will make use of four data sets. Two of them will be produced by QT21 two
in conjunction with CRACKER for WMT. This document presents therefore four
DMPs.

The first DMP is organised around the data used and produced by the Workshop
on Machine Translation (WMT,  _http://www.statmt.org_ ) for training SMT
engines. This data set will be used by Work Packages 1 and 2 (WP1, WP2), see
section 2.

Two other DMPs are defined with respect to the work done in WP3. The related
two data sets are new and will be produced by QT21. As these data sets are
also of a new type, the section on data selection methodology goes into
details.

The first DMP for WP3 (section 3) deals with human annotations (human
posteditions and human error-annotations). This data set is made of 50.000
(resp.

25.000) 3-tuples {source;reference;human-post-edition} in 4 language pairs  2
(resp. in the 2 language pairs  3  ). From these 3-tuples, 1.000 for each of
the language-pair are extended to 4-tuples by adding error-annotations to each
segment {source;reference;human-post-edition;human-error-annotation}.

Associated to the human post-editions and human error-annotations that will be
produced, guidelines have been produced that are appended to this deliverable.
These guidelines are meant to harmonise and coordinate (and are seen as
standardisation means of) the human post-edition and error-annotation
processes.

The second DMP for WP3 (section 4) will be generated in order to train a
Statistical Machine Translation system on two different domains: Information
Technology (IT) and Pharmacy. The data set will be produced out of a mixture
of in-domain data and similar-to-in-domain data extracted from a generic
corpus using a cross-entropy based filter.

Last but not least, the WP4 DMP will produce new data for 3 WMT “translation
tasks” that will run in 2016, 2017 and 2018. This data production will be
organised jointly (shared task) with the EC funded project CRACKER (see
section 5).

Each data set described here is referenced to as a whole. For documentation
sake, we give further an indication of the data split we will make use of,
separating training data from evaluation data (see the respective “data split”
sub-section in sections 2 and 3). For development test sets, it has been
decided not to impose them and to leave it to each partner to extract what
they need from the training data set.

  2. **Data Plan for WP1-WP2**

    1. **Introduction**

  
  

  

  

WP1 and WP2 are mainly focused on improving technology for the language pairs
considered. Both WPs have no specific requirement on data. As a consequence,
both WPs will rely on existing data sets.

  2. **Data selection process – methodology**

  

  

The main issue for these WPs is comparability of results between different
technologies and methods used to improve MT and push the State-of-the-Art in
MT. Therefore partner have agreed to work on pre-defined training sets which
domains are known (for the blind test sets see section 5).

  3. **Data description: WP1-WP2**

  

  

####  2.3.1 Data set reference and name  
  
<table>  
<tr>  
<th>

**Language**

**Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

QT21-EN-DE-wmt

</td>  
<td>

WMT Data

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

QT21-EN-CS-wmt

</td>  
<td>

WMT Data

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

QT21-EN-LV-EP

</td>  
<td>

Europarl Corpus

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

QT21-EN-RO-EP

</td>  
<td>

Europarl Corpus

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

QT21-DE-EN-wmt

</td>  
<td>

WMT Data

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

QT21-CS-EN-wmt

</td>  
<td>

WMT News Data

</td> </tr> </table>

**Table 2-1 – WP1-WP2-Training data: Reference set for each language pair**

####  2.3.2 Data set description

For the German-English and Czech-English, well-established test and training
sets are available from the Workshop on Statistical Machine Translation. Using
these data sets, we are not only able to compare the performance within the
project, but also within the research community.

The data consists of translated news articles from different languages.

For English-Romanian and English-Latvian, we agree to use the Europarl domain
to evaluate the techniques developed within this work package.

In order to concentrate on method comparison, the training data is limited to
the data available for the WMT Evaluations. During the three years of the
project life, QT21 will follow the constraints given by WMT.

For the German–English pair, the parallel data consist of the Europarl Corpus
version 7, the News commentary corpus v10 and the Common Crawl Corpus. In
addition, monolingual news data is available.

For the Czech-English pair, in addition to the corpora referred to above for
the German-English pair, the CzEng 1.0  4  ( _http://ufal.mff.cuni.cz/czeng_ )
can be used to train the models.

All this data is downloadable from
_http://www.statmt.org/wmtXX/translationtask.html_ ,  XX being the year of the
WMT campaign.

For Latvian and Romanian to English, we will use the freely available Europarl
corpus to train the SMT systems.  
  
<table>  
<tr>  
<th>

**Language**

**Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

QT21-EN-DE-train

</td>  
<td>

Europarl V7 + News Commentary Corpus V10 + Common Crawl as defined in
_http://www.statmt.org/wmt15/translationtask.html_

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

QT21-EN-CS-train

</td>  
<td>

Europarl V7 + News Commentary Corpus V10 + Common Crawl + CzEng 1.0 as defined
in

_http://www.statmt.org/wmt15/translationtask.html_

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

QT21-EN-LV-train

</td>  
<td>

_http://www.statmt.org/europarl/_

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

QT21-EN-RO-train

</td>  
<td>

_http://www.statmt.org/europarl/_

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

QT21-DE-EN-train

</td>  
<td>

Europarl V7 + News Commentary Corpus V10 + Common Crawl as defined in
_http://www.statmt.org/wmt15/translationtask.html_

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

QT21-CS-EN-train

</td>  
<td>

Europarl V7 + News Commentary Corpus V10 + Common Crawl + CzEng 1.0 as defined
in

_http://www.statmt.org/wmt15/translationtask.html_

</td> </tr> </table>

**Table 2-2 – WP1-WP2-Training: Reference set for each language pair**

####  2.3.3 Standards and metadata

The data is collected over several years and available in standard formats. An
exact description can be found at  _http://www.statmt.org/wmt15/translation-
task.html_ . See Annex A for an example of the format used.

**2.3.4 Data sharing**

The data is freely available at  _http://www.statmt.org/wmt15/translation-
task.html_ .

**2.3.5 Archiving**

Following the rules set by  _http://www.statmt.org_ .

**2.3.6 Data Split**

No data split. This data will be released in one shot as described above.

  3. **Data Plan for WP3-Human annotations**

  

  

**3.1 Introduction**

The main goal of WP3 is the development of translation techniques that are
aware of the impact of specific error types on machine translation and can be
efficiently improved by learning from human feedback and corrections of
specific error types.

The success of WP3 is hence connected to the availability of large quantity of
data containing human feedback in the form of Human Post-Edition (HPE) and/or
Human Error Annotation (HEA) of MT errors. HPE is about the “what” is wrong:
it corrects translations and provides insight into what text is corrected. HEA
is about the “why” it is wrong: it identifies and names specific errors and is
thus useful for understanding why corrections are made and what types of
errors are made. HEA is 5 to 6 times more expensive than HPE.

This data also needs to contain the translated reference so that the other
work packages can work on this data also. Table 3-1 shows for each QT21
language pair the volume of human generated data (Post Editions and Error
Annotations) that the project will produce.  
  
<table>  
<tr>  
<th>

**Language Pairs**

</th>  
<th>

**Post Edition Volume**

</th>  
<th>

**Error Annotation Volume**

</th>  
<th>

**Data Set Label**

</th> </tr>  
<tr>  
<td>

EN-DE

</td>  
<td>

50.000

</td>  
<td>

1.000

</td>  
<td>

Set A

</td> </tr>  
<tr>  
<td>

EN-CS

</td>  
<td>

50.000

</td>  
<td>

1.000

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

EN-LV

</td>  
<td>

25.000

</td>  
<td>

1.000

</td>  
<td>

Set B

</td> </tr>  
<tr>  
<td>

EN-RO

</td>  
<td>

25.000

</td>  
<td>

1.000

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

DE-EN

</td>  
<td>

50.000

</td>  
<td>

1.000

</td>  
<td>

Set C

</td> </tr>  
<tr>  
<td>

CS-EN

</td>  
<td>

50.000

</td>  
<td>

1.000

</td> </tr> </table>

####  Table 3-1 – WP3- QT21 language pairs and related HPE and HEA volumes in
number of segments

During the three years of the project, WP3 will use the two typologies to
generate two sets of human-annotated data with the following content:

  1. HPE sentences: for each source sentence, the MT output, the reference and the post-edited sentence obtained by the work of professional translators will be made available; 

  

  2. HEA information: the source, target, reference and post-edited sentences will be enriched with error-annotations provided by professional translators using the harmonised error metric developed in WP3. 

  

  

The next section describes the processes required to select the appropriate TM
data from which to create HPE and HEA data.

**3.2 Generation of HPE and HEA data – methodology**

####  3.2.1 Corpus selection

The goal of WP3 is to develop new translation techniques leveraging and
learning from human feedback.

The efficiency of learning from human feedback depends very much on the
quality of the human feedback. We need a good balance of high quality MT
generated output (though not perfect) and lower quality  5  : the less
ambiguous is the human-annotation (on the MT output), the clearer the message
to the learning system.

Further, as the methods to be developed in WP3 are statistical methods, the
efficiency of these methods (learning from human feedback) also depends on the
number of similar annotations (messages) the learning system will observe: The
more repetition of error types the better. This can be best achieved when
working on a specific domain from which one can expect a higher repetitions of
errors.

The latter point has also the advantage that the consortium will be working on
data that reflects the kinds of data managed on a daily basis by Language
Service Providers and professional translators.

The WP3 data selected has to reflect the following minimal constraint set:

  * Data contains source and reference segments  6 

  

  * Data is within a narrow domain  7 

  

  * Data can be shared and referenced within the research community 

  

  * Data covers the six QT21 language pairs (Table 3-1) 

  

  * Data should contain, for each language pair, at least 50k clean and high quality source-reference segments pairs. 

  

The largest data set we have found that covers these constraints is that of
the TAUS Data Association (TDA). Table 3-2 gives the number of words for
translation memories available within TDA in two different domains that are of
interest for the WP3.

This set allowed us to define three data sets. Since within each set the
source segments are the same for each language-pair, two-by-two language-pair
comparison is possible:

**Set A** comprises bilingual segments in EN (US) - CS and EN (US) – DE in the
domain of Computer Software. The content creator is Adobe. The total number of
segments in the selected corpora are 6.5 Mio segments for EN-DE and nearly 1
Mio for EN-CS.

**Set B** comprises bilingual segments in CS – EN (UK) and DE – EN (UK) in the
domain of Pharmaceuticals and Biotechnology. The content creator is the
European Medicines Agency. For both corpora, the number of available segments
is about 450k.

**Set C** : comprises bilingual segments in EN (UK) – LV and EN (UK) – RO in
the domain of Pharmaceuticals and Biotechnology. The content creator is the
European Medicines Agency. For both corpora, the number of available segments
is about 450k.  
  
<table>  
<tr>  
<th>

**Language Pairs**

</th>  
<th>

**Computer**

**Hardware**

**(# Words)**

</th>  
<th>

**Computer**

**Software**

**(# Words)**

</th>  
<th>

**Pharma (# Words)**

</th>  
<th>

</th>  
<th>

**Data Set Label**

</th> </tr>  
<tr>  
<td>

EN-DE

</td>  
<td>

24.166.846

</td>  
<td>

83.001.203

</td>  
<td>

412.397

</td>  
<td>

</td>  
<td>

Set A

</td> </tr>  
<tr>  
<td>

EN-CS

</td>  
<td>

2.731.003

</td>  
<td>

12.470.776

</td>  
<td>

0

</td>  
<td>

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

EN-LV

</td>  
<td>

371

</td>  
<td>

198.405

</td>  
<td>

</td>  
<td>

5.812.284

</td>  
<td>

</td>  
<td>

Set B

</td> </tr>  
<tr>  
<td>

EN-RO

</td>  
<td>

1.119.292

</td>  
<td>

545.915

</td>  
<td>

</td>  
<td>

5.556.027

</td>  
<td>

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

DE-EN

</td>  
<td>

6.298.559

</td>  
<td>

1.211.718

</td>  
<td>

</td>  
<td>

6.385.014

</td>  
<td>

</td>  
<td>

Set C

</td> </tr>  
<tr>  
<td>

CS-EN

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

5.842.314

</td>  
<td>

</td> </tr> </table>

**Table 3-2 – WP3-Domain Selection within TAUS Data: Based on number of words.
We had three domains to choose from. The data sets that have been selected are
in yellow.**

####  3.2.2 Segment selection

To ensure good post-editions and good annotations, it is important that the
segments on which the MT engines will run are clean and sentence-like. For
this reason, we looked at the data provided by TAUS by adding the following
constraints:

  1. To ensure comparability between language pairs, we select source segments that are identical to both language pairs (in other words the same text has been translated into several languages) 

  

  2. Each source segment contains between 3 and 50 words  8  . 

  

  3. Both the source and the target segment end with a punctuation mark. The five selected punctuation marks are the following (see also Table 1): 

    1. Full stop ‘.’ 

  

    2. Colon ‘:’ 

  

    3. Semicolon ‘;’ 

  

    4. Question mark ‘?’ 

  

    5. Exclamation mark ‘!’ 

  
  

  

  4. The data does not contain duplicate bilingual segments: it is sorted-unique on bilingual segments. 

  

  

Constraint number 1 and 2 above participated each in a relative size reduction
of the corpora by 15%.

Further constraint number 3 contributed most importantly to the size reduction
of the corpora we are working on (relatively by about 30%).

Table 3-3 shows how punctuation is used to classify segments as sentence-like
or not. If the last character of a segment is within that character set, it is
considered a sentence. This definition can be applied to both source and
target segments or only to one of them (e.g., only to the source segment) or
to neither source nor target.

For example, the data set extracted from the TAUS data that follows the
“Punct_5” labelled punctuation is a data set where both source and target
segments end with a character within the “Punct_5” set.

It has been observed that the data sets following the “Source_Punct_5” or

“Target_Punct5” definitions are very small in size, suggesting the TAUS Data
is very clean. For this reason we will consider only the two disjointed data
sets “Punct_5” and “No_Punct_5”.  
  
<table>  
<tr>  
<th>

**Punctuation**

**Character**

**Set**

</th>  
<th>

**Label**

</th>  
<th>

**Source Segment ends in the punctuation set**

</th>  
<th>

**Target Segment ends in the punctuation set**

</th> </tr>  
<tr>  
<td>

. ; : ? ! :

</td>  
<td>

Punct_5

</td>  
<td>

Yes

</td>  
<td>

Yes

</td> </tr>  
<tr>  
<td>

No_Punct_5

</td>  
<td>

No

</td>  
<td>

No

</td> </tr>  
<tr>  
<td>

Source_Punct_5

</td>  
<td>

Yes

</td>  
<td>

No

</td> </tr>  
<tr>  
<td>

Target_Punct_5

</td>  
<td>

No

</td>  
<td>

Yes

</td> </tr> </table>

**Table 3-3 – WP3-Punctuation: Punctuation sets labelled according to on which
data type it is applied (source or target).**

Applying the constraint set above we obtained a high quality set of segments
as showed in Table 3-4 - WP3-High Quality TM (based on the punctuation set
Punct_5) that can be used for Post Editions and Annotations from which the 50k
segments to be post edited and annotated will be randomly extracted  9  .  
  
<table>  
<tr>  
<th>

**Data Set**

</th>  
<th>

**Language**

**Pair**

</th>  
<th>

**Punctuation Set**

</th>  
<th>

**Number of**

**Segments**

</th>  
<th>

**Domain**

</th>  
<th>

**Data Provider**

</th> </tr>  
<tr>  
<td>

Set A

</td>  
<td>

EN(US)-DE

</td>  
<td>

Punct_5

</td>  
<td>

80.874

</td>  
<td>

IT-Soft

</td>  
<td>

Adobe

</td> </tr>  
<tr>  
<td>

EN(US)-CS

</td>  
<td>

Punct_5

</td>  
<td>

81.352

</td>  
<td>

IT-Soft

</td>  
<td>

Adobe

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set B

</td>  
<td>

EN(UK)-LV

</td>  
<td>

Punct_5

</td>  
<td>

177.795

</td>  
<td>

Pharma

</td>  
<td>

European Medicines Agency

</td> </tr>  
<tr>  
<td>

EN(UK)-RO

</td>  
<td>

Punct_5

</td>  
<td>

179.285

</td>  
<td>

Pharma

</td>  
<td>

European Medicines Agency

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set C

</td>  
<td>

DE-EN(UK)

</td>  
<td>

Punct_5

</td>  
<td>

193.637

</td>  
<td>

Pharma

</td>  
<td>

European Medicines Agency

</td> </tr>  
<tr>  
<td>

CS-EN(UK)

</td>  
<td>

Punct_5

</td>  
<td>

193.516

</td>  
<td>

Pharma

</td>  
<td>

European Medicines Agency

</td> </tr> </table>

**Table 3-4 - WP3-High Quality TM (based on the punctuation set Punct_5) that
can be used for Post Editions and Annotations**

####  3.2.3 HPE and HEA production

Different SMT systems will be tested and the one system with the best overall
BLEU score will be selected to produce the MT segments needed. The selection
of the final 50.000 segments (resp. 25.000 segments for set B) will be done
while looking at having a variety of quality levels. This data set defines
Table 3-5.

Professional human translators will follow the guidelines described under
section 3.3.3 to produce the HPE and HEA the work package will work on.

**3.3 Data description: WP3-HPE and HEA**

####  3.3.1 Data set reference and name

#####  3.3.1.1 Translation memories  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

TAUS-EN-DE-TM4HA-IT

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

TAUS-EN-CS-TM4HA-IT

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

TAUS-EN-LV-TM4HA-Pharma

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

TAUS-EN-RO-TM4HA-Pharma

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

TAUS-DE-EN-TM4HA-Pharma

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

TAUS-CS-EN-TM4HA-Pharma

</td>  
<td>

Contact TAUS and ask for it

</td> </tr> </table>

**Table 3-5 – WP3-Translation Memories for human annotation: Reference set for
each language pair**

#####  3.3.1.2 HPE and HEA segments

Both data sets will be made available when both HPE and HEA processes will be
finalised.  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HPE-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HPE-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HPE-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HPE-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HPE-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HPE-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

######  Table 3-6 – WP3-HPE data: Reference set for each language pair  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HEA-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HEA-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HEA-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HEA-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HEA-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HEA-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

**Table 3-7 – WP3- HEA data: Reference set for each language pair**

####  3.3.2 Data set description

**Table 3-8** describes the TM segments from which HPE and HEA will be
generated.  
  
<table>  
<tr>  
<th>

**Data Set**

</th>  
<th>

**Language**

**Pair**

</th>  
<th>

**Punctuation Set**

</th>  
<th>

**Number of Segments**

</th>  
<th>

**Domain**

</th>  
<th>

**Data Provider**

</th> </tr>  
<tr>  
<td>

Set A

</td>  
<td>

EN(US)-DE

</td>  
<td>

Punct_5

</td>  
<td>

50,000

</td>  
<td>

IT-Soft

</td>  
<td>

Adobe

</td> </tr>  
<tr>  
<td>

</td>  
<td>

EN(US)-CS

</td>  
<td>

Punct_5

</td>  
<td>

50,000

</td>  
<td>

IT-Soft

</td>  
<td>

Adobe

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set B

</td>  
<td>

EN(UK)-LV

</td>  
<td>

Punct_5

</td>  
<td>

25.000

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

EN(UK)-RO

</td>  
<td>

Punct_5

</td>  
<td>

25.000

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set C

</td>  
<td>

DE-EN(UK)

</td>  
<td>

Punct_5

</td>  
<td>

50,000

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

CS-EN(UK)

</td>  
<td>

Punct_5

</td>  
<td>

50,000

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr> </table>

**Table 3-8 - WP3-High Quality TM (based on the punctuation set Punct_5) that
can be used for Post Editions and Annotations**

####  3.3.3 Standards and metadata

The TM are produced along the standards defined in section 5.3.3 that can also
be seen at  _http://www.statmt.org/wmt15/translation-task.html_ .

For HPE and HEA data, guidelines have been developed to help professional
translators producing consistent annotations between themselves. These
guidelines are based on the experience from TAUS and the work done on
QTLaunchPad.

These guidelines will evolve over time as the project will learn from MT
related errors and/or language related errors (Latvian and Romanian are new
languages to the consortium).

#####  3.3.3.1 Post-Edition

The post-edition guidelines (see Annex B) are aimed at helping QT21
participants (project managers, post-editors, evaluators) set clear
expectations and can be used as a basis on which to instruct post-editors. It
is not practical to present a set of guidelines that will cover all scenarios.
It’s better if these are used as baseline guidelines and are tailored as
required for the given purpose. Generally, these guidelines assume bi-lingual
post-edition that is ideally carried out by a paid translator but that might
in some scenarios be carried out by bilingual domain experts or volunteers.
While the QT21 project will aim at delivering 15.000 segments in six language
pairs, the guidelines presented here are not system or languagespecific, thus
can be applied throughout the whole project.

#####  3.3.3.2 Error Annotation

In QT21, annotation will always be made on segments that are also post edited
10  . This means HEA and HPE guidelines have to be harmonised, which leads to
more precise guidelines for the post-edition process when the segment has been
also annotated for errors: the specific error-annotation and related post-
edition guidelines are described in Annex C page 24.

For error-annotations, an XML form developed in the QTLaunchPad project will
be used. It groups together the results of multiple annotators and provides a
number of features.

The permissible elements and attributes are defined in the schema
(annotations.xsd) included in Annex D page 40 of this document.

The XSLT stylesheet included in Annex E page 41 can be used to convert the XML
format into an HTML output format.

Annex F page 42 gives the example of an XML file containing one annotated
segment.

Annex G page 43 gives a prose description of the XML basic elements and
attributes.

####  3.3.4 Data sharing

We have two types of data to share. The TM from TAUS and the QT21 generated
annotations.

#####  3.3.4.1 Translation memories

In order to access data from TAUS, researchers and affiliates have to register
to TAUS. Once identified as belonging an academic institution, they can access
TAUS Data for free according to the TAUS academic membership plan and policy.

In order to ease access to and refer to the data used within QT21, the data
originated from TAUS will be marked with the different labels/names as defined
in Table 3-5.

This means each person registered in TAUS Data can have direct access
according their membership plan without ambiguity to exactly the same data as
used during the life of QT21.

#####  3.3.4.2 Human post-editions and error-annotations

HPE and HEA data will be made available on Meta-Share:
_http://www.metashare.eu/_

####  3.3.5 Archiving

For the TM, the TAUS infrastructure is used.

For the HPE and HEA data, we use the Meta-Share infrastructure to make the
newly generated data available over time.  _http://www.meta-share.eu/_

####  3.3.6 Data Split

The consortium will release the data as the project advances, first releasing
2/3 of the data for training purposes and the last 1/3 of the data for
development and three different evaluation campaigns (2016, 2017 and 2018).
The data of the evaluation campaigns will be agreed between the translation
and quality estimation shared task (WP4 and WMT).

#####  3.3.6.1 Training data  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HPE-TRAIN-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HPE-TRAIN-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HPE-TRAIN-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HPE-TRAIN-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HPE-TRAIN-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HPE-TRAIN-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

######  Table 3-9 – WP3-HPE TRAIN data: Reference set for each language pair  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HEA-TRAIN-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HEA-TRAIN-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HEA-TRAIN-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HEA-TRAIN-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HEA-TRAIN-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HEA-TRAIN-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

**Table 3-10 – WP3- HEA-TRAIN data: Reference set for each language pair**

#####  3.3.6.2 Evaluation data  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HPE-EVAL-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HPE-EVAL-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HPE-EVAL-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HPE-EVAL-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HPE-EVAL-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HPE-EVAL-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

######  Table 3-11 – WP3-HPE evaluation data: Reference set for each language
pair  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

HEA-EVAL-EN-DE-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

HEA-EVAL-EN-CS-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

HEA-EVAL-EN-LV-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

HEA-EVAL-EN-RO-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

HEA-EVAL-DE-EN-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

HEA-EVAL-CS-EN-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

**Table 3-12 – WP3- HEA-EVAL data: Reference set for each language pair**

  4. **Data Plan for WP3-Domain specific training data**

  

  

**4.1 Introduction**

A crucial aspect for the production of the post-edited segments and the
errorannotations is the creation of domain-specific data to train a SMT
system. This is necessary because a generic translation system will not be
able to correctly translate specific terms or expressions, thus producing
target sentences with too many errors. This is likely to make professional
translators rewrite the translations from scratch and produce a post-edition
in a similar manner to reference translations, making error-annotation
impossible.

**4.2 Data selection process – methodology**

The best data to train a domain specific MT engine is domain specific data
(indomain data). To start with, we will select in-domain TMs from the IT and
Pharma domains. This data will come from TAUS and from OPUS. As the amount of
indomain data gathered (see Table 4-4 and Table 4-5) is not enough to train
good domain specific MT engines, data selection techniques will also be
performed to identify from a large collection of generic parallel datasets
(out-of-domain) those segments that are the closest to the specific domains.

This process, referred to as data selection, applies techniques borrowed from
Information Retrieval such as the TF-IDF used by [Lu et al. 2007], to rank
each element from the large pool of data according its similarity in terms of
topic or style to the in-domain data.

For QT21, we propose to use cross-entropy-based selection for monolingual data

[Moore and Lewis, 2010] and its extended version for bilingual texts proposed
by [Axelrod et al. 2011]. Project partners (including FBK and USFD) have prior
experience with these techniques.

Originally proposed by [Gao and Zhang, 2002], perplexity-based approaches
consist of computing the perplexity score of each sentence of a generic corpus
against an in-domain language model, and doing the same against a language
model trained on the generic corpus itself. The sentences are then ranked
according to the difference between their two perplexity scores (in-domain and
generic). Once all the generic sentences have been ranked, the size of the
subset to extract is determined by minimising the perplexity of a development
set against a language model trained on increasing amount of the sorted
corpus. According to [Moore and Lewis, 2010], when using less but more
relevant data, perplexity decreases.

All of those methods [Gao and Zhang, 2002], [Moore and Lewis, 2010] and
[Axelrod et al. 2011] have been implemented in XenC [Rousseau, 2013], a freely
available open-source tool developed during the MateCat project.

For each language pair, the in-domain corpus will be selected from the
resources listed in Table 3-4 together with the generic corpora obtained from
large collections available on the WEB (e.g. Opus, Europarl, which still need
to be defined).

**4.3 Data description: WP3-Domain specific data**

####  4.3.1 Data set reference and name

We have 3 sources of data on which to train the SMT engine that will generate
the

MT segments used as basis for HPE and HEA-EVAL. These are the Train-TAUS (for
IT and Pharma domains), the train-Opus (for the IT domain) and the train-Auto-

Extract corpora that we will generate based on the methodology described in
section 4.2.  
  
<table>  
<tr>  
<th>

**Language**

**Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

TAUS-QT21-EN-DE-train-IT

</td>  
<td>

Contact TAUS and ask for it

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

TAUS-QT21-EN-CS-train-IT

</td>  
<td>

Contact TAUS and ask for it

</td> </tr> </table>

#####  Table 4-1 – WP3-Train-TAUS: Reference set for each language pair  
  
<table>  
<tr>  
<th>

**Language**

**Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-train-Gnome

</td>  
<td>

_http://opus.lingfil.uu.se/GNOME.php_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-train-KDE4

</td>  
<td>

_http://opus.lingfil.uu.se/KDE4.php_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-train-KDEdoc

</td>  
<td>

_http://opus.lingfil.uu.se/KDEdoc.php_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-trainOpenOffice3

</td>  
<td>

_http://opus.lingfil.uu.se/OpenOffice3.p_ _hp_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-trainOpenOffice

</td>  
<td>

_http://opus.lingfil.uu.se/OpenOffice.ph_ _p_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-train-PHP

</td>  
<td>

_http://opus.lingfil.uu.se/PHP.php_

</td> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

OPUS-EN-DE-train-Ubuntu

</td>  
<td>

_http://opus.lingfil.uu.se/Ubuntu.php_

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

OPUS-EN-CS-train-Gnome

</td>  
<td>

_http://opus.lingfil.uu.se/GNOME.php_

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

OPUS-EN-CS-train-KDE4

</td>  
<td>

_http://opus.lingfil.uu.se/KDE4.php_

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

OPUS-EN-CS-train-PHP

</td>  
<td>

_http://opus.lingfil.uu.se/PHP.php_

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

OPUS-EN-CS-train-Ubuntu

</td>  
<td>

_http://opus.lingfil.uu.se/Ubuntu.php_

</td> </tr> </table>

#####  Table 4-2 – WP3-Train-OPUS for IT Domain: Reference set for each
language pair

Automatic extraction of domain-like data from general public and open corpora  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

EN🡪DE

</td>  
<td>

QT21-EN-DE-train-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪CS

</td>  
<td>

QT21-EN-CS-train-IT

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪LV

</td>  
<td>

QT21-EN-LV-train-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

EN🡪RO

</td>  
<td>

QT21-EN-RO-train-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

DE🡪EN

</td>  
<td>

QT21-DE-EN-train-Pharma

</td>  
<td>

TBA

</td> </tr>  
<tr>  
<td>

CS🡪EN

</td>  
<td>

QT21-CS-EN-train-Pharma

</td>  
<td>

TBA

</td> </tr> </table>

**Table 4-3 – WP3-Train-auto-extract: Reference set for each language pair**

####  4.3.2 Data set description

Table 4-4 describes more specifically the domain specific TM QT21 receives
from TAUS for the IT domain.  
  
<table>  
<tr>  
<th>

**Data Set**

</th>  
<th>

**Language**

**Pair**

</th>  
<th>

**Constraint on segments**

</th>  
<th>

**Number of Segments**

</th>  
<th>

**Domain**

</th>  
<th>

**Data Provider**

</th> </tr>  
<tr>  
<td>

Set A

</td>  
<td>

EN-DE

</td>  
<td>

No

</td>  
<td>

7.974.430

</td>  
<td>

IT-Soft+HW

</td>  
<td>

Various except for Adobe

</td> </tr>  
<tr>  
<td>

EN-CS

</td>  
<td>

No

</td>  
<td>

1.260.696

</td>  
<td>

IT-Soft +

HW

</td>  
<td>

Various except for Adobe

</td> </tr> </table>

#####  Table 4-4 - WP3-TAUS training data (all data)

Table 4-5 describes more precisely additional domain specific data QT21 will
get from OPUS, both for the IT and pharma domains. This data is complimentary
to the one in Table 4-4. For sets B and C (Pharma domain)  
  
<table>  
<tr>  
<th>

**Data Set**

</th>  
<th>

**Language**

**Pair**

</th>  
<th>

**Constraint on segments**

</th>  
<th>

**Number of Segments**

</th>  
<th>

**Domain**

</th>  
<th>

**Data Provider**

</th> </tr>  
<tr>  
<td>

Set A

</td>  
<td>

EN-DE

</td>  
<td>

No

</td>  
<td>

310.285

</td>  
<td>

IT

</td>  
<td>

OPUS

</td> </tr>  
<tr>  
<td>

EN-CS

</td>  
<td>

No

</td>  
<td>

125.309

</td>  
<td>

IT

</td>  
<td>

OPUS

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set B

</td>  
<td>

EN-LV

</td>  
<td>

No

</td>  
<td>

1.005.272

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

EN-RO

</td>  
<td>

No

</td>  
<td>

969.499

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td>  
<td>

</td> </tr>  
<tr>  
<td>

Set C

</td>  
<td>

DE-EN

</td>  
<td>

No

</td>  
<td>

1.058.752

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr>  
<tr>  
<td>

CS-EN

</td>  
<td>

No

</td>  
<td>

1.003.385

</td>  
<td>

Pharma

</td>  
<td>

European

Medicines Agency

</td> </tr> </table>

#####  Table 4-5 - WP3-OPUS+EMEA training data

In the next version of this deliverable we shall be able to describe the data
obtained from the data selection technique described section 4.2.

####  4.3.3 Standards and metadata

The TM are produced along the standards defined in section 5.3.3 that can also
be seen at  _http://www.statmt.org/wmt15/translation-task.html_ .

###  4.3.4 Data sharing

In order to access data from TAUS, researchers and affiliates have to register
to TAUS. Once identified as belonging an academic institution, they can access
TAUS Data for free according to the TAUS academic membership plan and policy.

In order to ease access to and refer to the data used within QT21, the data
originated from TAUS will be marked with the different labels/names as defined
in Table 4-1.

This means each person registered in TAUS Data can have direct access
according their membership plan without ambiguity to exactly the same data as
used during the life of QT21.

The other data are available through Meta-Share  _http://www.meta-share.eu/_ .

###  4.3.5 Archiving

For the TAUS Data, TAUS has its own archiving system.

For the other data, we use the Meta-Share infrastructure to make the newly
generated data available over time  _http://www.meta-share.eu/_ .

###  4.3.6 Data Split

No data split. This data will be released in one shot as described above.

#  5 Data Plan for WP4

##  5.1 Introduction

This data plan concerns only the shared task with WMT. We refer here only to
the

WMT test sets that WP1, WP2 and WP3 will make use of for their blind
evaluations.

In WP4 we organise together with CRACKER three annual shared task campaigns: a
translation shared task, a quality estimation shared task, and a metrics
shared task. These tasks continue a successful series of shared tasks held
with the Workshop on Statistical Machine Translation (WMT) in previous years.

We aim to create around 6000 sentences of human-translated text for each year
of the translation task, in two language pairs. This text will be used as an
evaluation set or be split into separate sets for system development and
evaluation.

Collaboration with other projects such as CRACKER will enable us to cover more
than just two languages in the shared tasks. The core language pairs are
GermanEnglish and Czech-English, but other challenging language pairs will be
introduced each year. We typically have three to five language pairs for WMT
shared tasks.

##  5.2 Data selection process – methodology

We crawl monolingual sources from online news sites. We then create manual
translations of crawled monolingual data to be used as test sets for the
shared tasks.

##  5.3 Data description: Evaluation data WP4

###  5.3.1 Data set reference and name

For WMT’16, the data set will be defined during the project meeting prior to
WMT’15.  
  
<table>  
<tr>  
<th>

**Language Pair**

</th>  
<th>

**Name**

</th>  
<th>

**Reference**

</th> </tr>  
<tr>  
<td>

TBA

</td>  
<td>

WMT Test Sets

</td>  
<td>

Will be  _http://www.statmt.org/wmt16/_

</td> </tr> </table>

**Table 5-1 – WP4-Test: Reference set for each language pair will be
announced**

###  5.3.2 Data set description

For German and Czech, we will use the yearly blind official evaluation sets of
the

WMT Evaluations that are being produced by QT21 and CRACKER together. They are
small parallel data sets used for testing MT systems, and are typically
created by translating a selection of crawled articles from online news sites.

For Latvian and Romanian, new evaluation test sets will be defined and
created, probably together with WMT for WMT’16 and WMT’17.

###  5.3.3 Standards and metadata

WMT test sets are typically distributed in an SGML format which is compatible
with common machine translation evaluation tools such as the NIST scoring tool
(mtevalv13a.pl).The text encoding is Unicode (UTF-8).

Metadata such as language codes and document identifiers are provided in the
SGML documents. See Annex A for an example of the format used.

###  5.3.4 Data sharing

The data will be made available from the appropriate WMT website (i.e.

_http://www.statmt.org/wmt15/_ for 2015).

###  5.3.5 Archiving

The data will remain available for download from  _http://www.statmt.org/_ .
This website is currently hosted at the University of Edinburgh.