Repository: mjpost/sacrebleu (language: python)

Issue:
Sacrebleu 2.0 dies when encountering an empty reference line
Sacrebleu 2.0.0 dies if there is an empty line in the reference. You can reproduce this with this command:

```bash
 $ sacrebleu -t mtedx/test -l el-en --echo ref | sacrebleu -t mtedx/test -l el-en
Traceback (most recent call last):
  File "/home/mattpost/local/bin/sacrebleu", line 8, in <module>
    sys.exit(main())
  File "/home/mattpost/local/lib/python3.8/site-packages/sacrebleu/sacrebleu.py", line 482, in main
    metrics[name] = METRICS[name](**metric_args)
  File "/home/mattpost/local/lib/python3.8/site-packages/sacrebleu/metrics/bleu.py", line 197, in __init__
    self._ref_cache = self._cache_references(references)
  File "/home/mattpost/local/lib/python3.8/site-packages/sacrebleu/metrics/base.py", line 333, in _cache_references
    raise RuntimeError("Empty or `None` reference sentence found.")
RuntimeError: Empty or `None` reference sentence found.
```
Default params for calculating bleu
Hi, what exactly are the default set of parameters (tokenize, smooth_method, smooth_value) to calculate BLEU? I see default smooth_method is `floor` and `tokenize='none'` (which uses 13a tokenizer) in this repo while huggingface `datasets` has default smooth_method=`exp` and `tokenize=None` (which uses BaseTokenizer)

would like to know the default one used in research.

And also, whats the difference b/w `raw_corpus_bleu` and `corpus_bleu` methods in `compat.py`?
Unexpected behavior when using zh/ja sacrebleu in Python 
# Problem 

The current python logic don't do the `zh` and `ja-mecab` as expected by user, here's the minimal reproducible example:

```python
TOKENIZERS = ['none', 'zh', '13a', 'char', 'intl', 'ja-mecab', 'spm']
TOKENIZER_DEFAULT = '13a'
_TOKENIZER_MAP = {
    'zh': 'zh',
    'ja': 'ja-mecab',
}

tokenize = '13a' # '13a'
trg_lang = 'ja-mecab'

if tokenize is None:
    best_tokenizer = TOKENIZER_DEFAULT
    if trg_lang in _TOKENIZER_MAP:
        best_tokenizer = _TOKENIZER_MAP[trg_lang]
else:
    best_tokenizer = tokenize
    if trg_lang == 'zh' and best_tokenizer != 'zh':
        print("You should use the 'zh' tokenizer for Chinese.")
    if trg_lang == 'ja' and best_tokenizer != 'ja-mecab':
        print("You should use the 'ja-mecab' tokenizer for Japanese.")

print(best_tokenizer)
```

[out]:

```
'13a'
```

Because the default arguments is set where, `tokenize = '13a'` 

```python
class BLEU(metric):
    def __init__(self, lowercase: bool = False,
                 force: bool = False,
                 tokenize: Optional[str] = '13a',
                 smooth_method: str = 'exp',
                 smooth_value: Optional[float] = None,
                 max_ngram_order: int = MAX_NGRAM_ORDER,
                 effective_order: bool = False,
                 trg_lang: str = '',
                 references: Optional[Sequence[Sequence[str]]] = None):
```

So currently users get stymied at:

```python
bleu = BLEU(trg_lang='ja-mecab')

refs = [# First set of references
        ['犬が男を噛んだ。', 'それは予想外ではなかった。', '男が最初に彼を噛んだ。'],
        # Second set of references
        ['犬が男を噛んだ。', '誰も驚かなかった。', '男が犬を噛んだ。'],
       ]
sys = ['犬が男を噛んだ。', "それは驚くべきことではなかった。", '男はちょうど彼を噛んだ。']

bleu.corpus_score(sys, refs)
```

[out]:

```
BLEU = 0.00 33.3/0.0/0.0/0.0 (BP = 1.000 ratio = 1.000 hyp_len = 3 ref_len = 3)
```

And had to manually do:

```python
bleu = BLEU(trg_lang='ja-mecab', tokenize='ja-mecab')
bleu.corpus_score(sys, refs)
```

to get:

```
BLEU = 63.04 80.0/68.2/57.9/50.0 (BP = 1.000 ratio = 1.087 hyp_len = 25 ref_len = 23)
```

----

# Proposal 1: Keep the warning but implicitly set the `best_tokenizer` according to the `trg_lang`

```python
if tokenize is None:
    best_tokenizer = self._TOKENIZER_MAP[trg_lang] if self.trg_lang in self._TOKENIZER_MAP else self.TOKENIZER_DEFAULT
else:
    best_tokenizer = tokenize
    if self.trg_lang == 'zh' and best_tokenizer != 'zh':
        best_tokenizer = 'zh'
        sacrelogger.warning(
            "For Chinese, you should initialize the scorer as such `bleu = BLEU(trg_lang='zh', tokenize='zh')`")
    if self.trg_lang == 'ja' and best_tokenizer != 'ja-mecab':
        best_tokenizer = 'ja-mecab'
        sacrelogger.warning(
            "For Japanese, you should initialize the scorer as such `bleu = BLEU(trg_lang='ja-mecab', tokenize='ja-mecab')`")
```

# Proposal 2: Keep the warning but explicitly tell user that sacrebleu is using `13a` instead of a `zh`/`ja` tokenizer

```python
if tokenize is None:
    best_tokenizer = self._TOKENIZER_MAP[trg_lang] if self.trg_lang in self._TOKENIZER_MAP else self.TOKENIZER_DEFAULT
else:
    best_tokenizer = tokenize
    if self.trg_lang == 'zh' and best_tokenizer != 'zh':
        best_tokenizer = 'zh'
        sacrelogger.warning(
            str("You might be seeing weird BLEU scores because you're using mt-eval's '13a' tokenizer.\n"
                  "For Chinese, you should initialize the scorer as such `bleu = BLEU(trg_lang='zh', tokenize='zh')`") )
    if self.trg_lang == 'ja' and best_tokenizer != 'ja-mecab':
        best_tokenizer = 'ja-mecab'
        sacrelogger.warning(
            str("You might be seeing weird BLEU scores because you're using mt-eval's '13a' tokenizer.\n"
            "For Japanese, you should initialize the scorer as such `bleu = BLEU(trg_lang='ja-mecab', tokenize='ja-mecab')`") )
```




Discussion / hints:
@ozancaglayan I tagged you, but let me know if you can't process this.
Hi there,

This is a side-effect of variable number of references introduced in #132 . The reason it was not failing with 1.5.x is probably due to this feature being available only through the API call and not the CLI utility. We now globally filter the set of references to reject empty or `None`s.

I would suggest fixing the references if possible as an empty line in references does not make much sense, what do you think?
At the very least, this error is way too cryptic. We should catch the exception and report to the user.

However, I think that we should support empty references, in part because there are many in our existing datasets. Without fixing this, those are unusable. Also, it is not clear how to fix them, especially for the average user.

Can’t we just do a check and convert `None`→ ``?
Honestly, I can't wrap my head around for this problem :) Let me think loudly.

Initially, the `''` check wasn't there and it was only filtering out the `None`s but if I understand correctly you proposed to add that bit as well ( See https://github.com/mjpost/sacrebleu/pull/132/commits/2b5052a211539570bbf7e71f618affecbad78d1f#r556098399)

That made sense because explicit `None`s could only appear if the user uses the API and injects them deliberately to their list of refs. This would mean that the CLI would not work with variable number of references. We allow multiple references through CLI in two ways (i) with a list of files and (ii) with a tab-separated stream. For (i) you can give 100 ref files and some of their lines can be missing to allow var. number of refs. For (ii), essentially the same thing: some columns would be empty strings. So still no `None`s at all to check and convert to `''`.

BTW, I am not sure why we throw that exception. It was again introduced in #132 probably for some sanity check @tuetschek ? I can't guess out of my head what happens if we remove that actually.

In the end, all these should be very well documented and tested to not cause ambiguities and weird results in the future.


We should be able to retain the feature of multiple tab-delimited references, no?

For example, consider ref1:

```
The first sentence.

```

and reference two:

```

The second sentence.
```

(I really wish Markdown supported tables).

Pasted, this would give

```
This is the first sentence\t
\tThis is the second sentence
```

Which is fine. To make this not throw an error, we just need those empty fields to be `""` instead of `None`, right? So we would change [line 330](https://github.com/mjpost/sacrebleu/blob/078c440168c6adc89ba75fe6d63f0d922d42bcfe/sacrebleu/metrics/base.py#L330) from this:

```python
            lines = [x for x in refs if x is not None and x != ""]
```

to this:

```python
            lines = [x for x in refs if x is not None]
```
But now I have a vague memory where we had to add the second check for some reason. Do you remember why?

I did test this. It fixes the command above, and still works in situations where we have multiple references:

```bash
$ sacrebleu -t wmt17/ms -l zh-en --echo src | ./sacrebleu.py -t wmt17/ms -l zh-en
{
 "name": "BLEU",
 "score": 0.0,
 "signature": "nrefs:3|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0",
 "verbose_score": "20.5/17.3/6.6/3.0 (BP = 0.000 ratio = 0.066 hyp_len = 3289 ref_len = 50034)",
 "nrefs": "3",
 "case": "mixed",
 "eff": "no",
 "tok": "13a",
 "smooth": "exp",
 "version": "2.0.0"
}
```

Actually, though, this doesn't work if you try to give the *references* as a tab-delimited stream. How does that work? Here, I dump the three references to disk, then try:

```bash
$ sacrebleu -t wmt17/ms -l zh-en --echo ref > refs
sacrebleu -t wmt17/ms -l zh-en --echo src | ./sacrebleu.py refs
{
 "name": "BLEU",
 "score": 0.0,
 "signature": "nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0",
 "verbose_score": "20.8/17.3/6.6/3.0 (BP = 0.000 ratio = 0.020 hyp_len = 3289 ref_len = 166361)",
 "nrefs": "1",
 "case": "mixed",
 "eff": "no",
 "tok": "13a",
 "smooth": "exp",
 "version": "2.0.0"
}
```

Note there is just one reference here, even though `refs` has three, tab-delimited. Maybe I am just forgetting the syntax for multiple tab-delimited refs?
For the last part, to make things even complicated, we have a historical `--num-refs` param that needs to be provided if the references are given in a tab-delimited file.. If you pass `--num-refs 3`, you get the same result.
> But now I have a vague memory where we had to add the second check for some reason. Do you remember why?
> 

The second check was added there to allow skipping empty reference lines since `None` reference lines are not something that can be curated through CLI. `None` placeholders can only be injected through the API use.

Now you could (or I could) say, why do we even filter out empty lines there. Then I can't directly guess if having an empty ref vs. not having it changes BLEU, TER or chrF scores computed at all.


I think we should first decide whether we want to support empty reference segments in case of a single reference (i.e. ignoring the use case of variable number of references).

> we should support empty references, in part because there are many in our existing datasets.

I can imagine multiple reasons for empty (segments in) references and I am not sure if we want to support all/any of them.

1. The source sentence is also empty. This could happen e.g.
    a.  to mark paragraph boundaries in plain text
    b.  in multi-parallel corpora where some translations for some languages are missing, but we want to keep all languages alignable by line numbers
2. The reference is wrong.
    a. The translation for a given segment is missing (but a correct translation is not empty).
    b. There was a technical error (e.g. a wrong sentence alignment between the source and reference files).
3. The correct translation of a given segment should be empty (according to the reference translator).
    a. The segment is a "filler word" (oh, ehm...) and the goal of the task is translation and text cleaning.
    b. The correct translation of a given segment should be empty only within a given document context, for example to prevent repeating the same sentence again.

The two empty lines in `-t mtedx/test -l el-en --echo ref` seem to be the case of 3b (although, the repetition is not exact: *You **speak** it very well.* vs. *You **know** it very well.*).

SacreBLEU can be used for evaluating document-level translation and translation-like tasks (such as text normalization, phonetic transcription etc.), so I agree there are reasons to support both 3a and 3b (at the cost of making everything a bit more complicated, e.g. not catching 2b).

We don't want to support 2b (sacrebleu should warn users about such wrong references) and even for 2a, I would say it would be better to mark such (temporarily) missing segments explicitly with a special token (MISSING) or exclude such segments from both source and target (esp. if sentence-level translation is expected).

I am not sure about 1a and 1b. It could be a nice feature for some users. For example, I would use 1a, but only if it results in exactly the same BLEU as when filtering out the empty lines (from both src and ref)<del>, which is not the case now because of the brevity penalty</del>.
Some tests:

```
$ cat hyps
The dog bit the man.
It wasn't surprising.
The man had just bitten him.

$ cat ref1
The dog bit the man.
It was not unexpected.
The man bit him first.

$ cat ref1_missing_line

It was not unexpected.
The man bit him first.

$ cat ref2
The dog had bit the man.
No one was surprised.
The man had bitten the dog.
```

```
cat hyps | sacrebleu ref1 -f text
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 45.1 70.6/42.9/36.4/37.5 (BP = 1.000 ratio = 1.000 hyp_len = 17 ref_len = 17)
# after relaxing x != ''
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 45.1 70.6/42.9/36.4/37.5 (BP = 1.000 ratio = 1.000 hyp_len = 17 ref_len = 17)

cat hyps | sacrebleu ref2 -f text
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 28.3 70.6/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)
# after relaxing x != ''
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 28.3 70.6/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)

cat hyps | sacrebleu ref1 ref2 -f text
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 48.5 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)
# after relaxing x != ''
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 48.5 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

# Wrong as we did not pass -nr 2
cat hyps | sacrebleu <(paste ref1 ref2) -f text
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 16.8 82.4/50.0/45.5/37.5 (BP = 0.327 ratio = 0.472 hyp_len = 17 ref_len = 36)
# after relaxing x != ''
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 16.8 82.4/50.0/45.5/37.5 (BP = 0.327 ratio = 0.472 hyp_len = 17 ref_len = 36)

cat hyps | sacrebleu <(paste ref1 ref2) -f text -nr 2
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 48.5 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)
# after relaxing x != ''
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 48.5 82.4/50.0/45.5/37.5 (BP = 0.943 ratio = 0.944 hyp_len = 17 ref_len = 18)

cat hyps | sacrebleu ref1_missing_line
RuntimeError: Empty or `None` reference sentence found.
# after relaxing x != ''
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 7.7 35.3/7.1/4.5/3.1 (BP = 1.000 ratio = 1.545 hyp_len = 17 ref_len = 11)

# do the same with multi refs
cat hyps | sacrebleu ref1_missing_line ref2 -f text
BLEU|nrefs:var|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 29.4 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)
# after relaxing x != '' ** variable ref counting broke
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 29.4 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)

# do the same with TSV
cat hyps | sacrebleu <(paste ref1_missing_line ref2) -f text -nr 2
BLEU|nrefs:var|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 29.4 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)
# after relaxing x != '' ** variable ref counting broke
BLEU|nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 29.4 82.4/42.9/27.3/12.5 (BP = 0.889 ratio = 0.895 hyp_len = 17 ref_len = 19)
```
Just a note: I think that this check was introduced so that no one is using empty references by mistake. "Simulating" variable number of references by using empty references could change the brevity penalty.
What do we do about this?



Fix patch:
diff --git a/.github/workflows/check-build.yml b/.github/workflows/check-build.yml
index ae92ac08..6f55a008 100644
--- a/.github/workflows/check-build.yml
+++ b/.github/workflows/check-build.yml
@@ -6,6 +6,11 @@ on:
 env:
   PYTHONUTF8: "1"
 
+# only run one at a time per branch
+concurrency:
+  group: check-build-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   check-build:
     runs-on: ${{ matrix.os }}
@@ -35,6 +40,7 @@ jobs:
           python -m pip install --upgrade pip
           pip install pytest
           pip install .[ja]
+          pip install .[ko]
       - name: Python pytest test suite
         run: python3 -m pytest
       - name: CLI bash test suite
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 419cb820..478fd3eb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,13 +1,32 @@
 # Release Notes
 
-- 2.1.0 (TBD)
+- 2.2.0 (2022-07-25)
   Features:
-  - Added `-tok spm` for multilingual SPM tokenization
+  - Added WMT21 datasets (thanks to @BrighXiaoHan)
+  - `--echo` now exposes document metadata where available (e.g., docid, genre, origlang)
+  - Bugfix: allow empty references (#161)
+  - Adds a Korean tokenizer (thanks to @NoUnique)
+
+  Under the hood:
+  - Moderate code refactoring
+  - Processed files have adopted a more sensible internal naming scheme under ~/.sacrebleu
+    (e.g., wmt17_ms.zh-en.src instead of zh-en.zh)
+  - Processed file extensions correspond to the values passed to `--echo` (e.g., "src")
+  - Now explicitly representing NoneTokenizer
+  - Got rid of the ".lock" lockfile for downloading (using the tarball itself)
+
+  Many thanks to @BrightXiaoHan (https://github.com/BrightXiaoHan) for the bulk of
+  the code contributions in this release.
+
+- 2.1.0 (2022-05-19)
+  Features:
+  - Added `-tok spm` for multilingual SPM tokenization (#168)
     (thanks to Naman Goyal and James Cross at Facebook)
 
   Fixes:
   - Handle potential memory usage issues due to LRU caching in tokenizers (#167)
   - Bugfix: BLEU.corpus_score() now using max_ngram_order (#173)
+  - Upgraded ja-mecab to 1.0.5 (#196)
 
 - 2.0.0 (2021-07-18)
   - Build: Add Windows and OS X testing to Travis CI.
diff --git a/README.md b/README.md
index 35eff82e..1427ddb7 100644
--- a/README.md
+++ b/README.md
@@ -59,6 +59,11 @@ following command instead, to perform a full installation with dependencies:
 
     pip install "sacrebleu[ja]"
 
+In order to install Korean tokenizer support through `pymecab-ko`, you need to run the
+following command instead, to perform a full installation with dependencies:
+
+    pip install "sacrebleu[ko]"
+
 # Command-line Usage
 
 You can get a list of available test sets with `sacrebleu --list`. Please see [DATASETS.md](DATASETS.md)
@@ -68,17 +73,15 @@ for an up-to-date list of supported datasets.
 
 ### Downloading test sets
 
-Download the **source** for one of the pre-defined test sets:
+Downloading is triggered when you request a test set. If the dataset is not available, it is downloaded
+and unpacked.
 
-```
-$ sacrebleu -t wmt17 -l en-de --echo src | head -n1
-28-Year-Old Chef Found Dead at San Francisco Mall
-```
+E.g., you can use the following commands to download the source, pass it through your translation system
+in `translate.sh`, and then score it:
 
-Download the **reference** for one of the pre-defined test sets:
 ```
-$ sacrebleu -t wmt17 -l en-de --echo ref | head -n1
-28-jähriger Koch in San Francisco Mall tot aufgefunden
+$ sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en
+$ cat wmt17.en-de.en | translate.sh | sacrebleu -t wmt17 -l en-de
 ```
 
 ### JSON output
@@ -194,8 +197,8 @@ BLEU related arguments:
                         Smoothing method: exponential decay, floor (increment zero counts), add-k (increment num/denom by k for n>1), or none. (Default: exp)
   --smooth-value BLEU_SMOOTH_VALUE, -sv BLEU_SMOOTH_VALUE
                         The smoothing value. Only valid for floor and add-k. (Defaults: floor: 0.1, add-k: 1)
-  --tokenize {none,zh,13a,char,intl,ja-mecab}, -tok {none,zh,13a,char,intl,ja-mecab}
-                        Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, `ja-mecab` for Japanese and `13a` (mteval) otherwise.
+  --tokenize {none,zh,13a,char,intl,ja-mecab,ko-mecab}, -tok {none,zh,13a,char,intl,ja-mecab,ko-mecab}
+                        Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, `ja-mecab` for Japanese, `ko-mecab` for Korean and `13a` (mteval) otherwise.
   --lowercase, -lc      If True, enables case-insensitivity. (Default: False)
   --force               Insist that your tokenized input is actually detokenized.
 
@@ -220,6 +223,26 @@ TER related arguments (The defaults replicate TERCOM's behavior):
 ### Version Signatures
 As you may have noticed, sacreBLEU generates version strings such as `BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0` for reproducibility reasons. It's strongly recommended to share these signatures in your papers!
 
+### Outputting other metadata
+
+Sacrebleu knows about metadata for some test sets, and you can output it like this:
+
+```
+$ sacrebleu -t wmt21 -l en-de --echo src docid ref | head 2
+Couple MACED at California dog park for not wearing face masks while having lunch (VIDEO) - RT USA News	rt.com.131279	Paar in Hundepark in Kalifornien mit Pfefferspray besprüht, weil es beim Mittagessen keine Masken trug (VIDEO) - RT USA News
+There's mask-shaming and then there's full on assault.	rt.com.131279	Masken-Shaming ist eine Sache, Körperverletzung eine andere.
+```
+
+If multiple fields are requested, they are output as tab-separated columns (a TSV).
+
+To see the available fields, add `--echo asdf` (or some other garbage data):
+
+```
+$ sacrebleu -t wmt21 -l en-de --echo asdf
+sacreBLEU: No such field asdf in test set wmt21 for language pair en-de.
+sacreBLEU: available fields for wmt21/en-de: src, ref:A, ref, docid, origlang
+```
+
 ## Translationese Support
 
 If you are interested in the translationese effect, you can evaluate BLEU on a subset of sentences
@@ -247,11 +270,12 @@ but it expects that you pass through the entire translated test set.
    - `intl` applies international tokenization and mimics the `mteval-v14` script from Moses
    - `zh` separates out **Chinese** characters and tokenizes the non-Chinese parts using `13a` tokenizer
    - `ja-mecab` tokenizes **Japanese** inputs using the [MeCab](https://pypi.org/project/mecab-python3) morphological analyzer
+   - `ko-mecab` tokenizes **Korean** inputs using the [MeCab-ko](https://pypi.org/project/mecab-ko) morphological analyzer
    - `spm` uses the SentencePiece model built from the Flores-101 dataset (https://github.com/facebookresearch/flores#list-of-languages). Note: the canonical .spm file will be automatically fetched if not found locally.
 - You can switch tokenizers using the `--tokenize` flag of sacreBLEU. Alternatively, if you provide language-pair strings
-  using `--language-pair/-l`, `zh` and `ja-mecab` tokenizers will be used if the target language is `zh` or `ja`, respectively.
+  using `--language-pair/-l`, `zh`, `ja-mecab` and `ko-mecab` tokenizers will be used if the target language is `zh` or `ja` or `ko`, respectively.
 - **Note that** there's no automatic language detection from the hypotheses so you need to make sure that you are correctly
-  selecting the tokenizer for **Japanese** and **Chinese**.
+  selecting the tokenizer for **Japanese**, **Korean** and **Chinese**.
 
 
 Default 13a tokenizer will produce poor results for Japanese:
diff --git a/sacrebleu/__init__.py b/sacrebleu/__init__.py
index e5f57049..ca9016e0 100644
--- a/sacrebleu/__init__.py
+++ b/sacrebleu/__init__.py
@@ -14,7 +14,7 @@
 # express or implied. See the License for the specific language governing
 # permissions and limitations under the License.
 
-__version__ = '2.1.0'
+__version__ = '2.2.0'
 __description__ = 'Hassle-free computation of shareable, comparable, and reproducible BLEU, chrF, and TER scores'
 
 
diff --git a/sacrebleu/compat.py b/sacrebleu/compat.py
index acb47c20..cce90e9f 100644
--- a/sacrebleu/compat.py
+++ b/sacrebleu/compat.py
@@ -15,6 +15,8 @@ def corpus_bleu(hypotheses: Sequence[str],
                 tokenize=BLEU.TOKENIZER_DEFAULT,
                 use_effective_order=False) -> BLEUScore:
     """Computes BLEU for a corpus against a single (or multiple) reference(s).
+    This is the main CLI entry point for computing BLEU between a system output
+    and a reference sentence.
 
     :param hypotheses: A sequence of hypothesis strings.
     :param references: A sequence of reference documents with document being
@@ -42,11 +44,16 @@ def raw_corpus_bleu(hypotheses: Sequence[str],
     This convenience function assumes a particular set of arguments i.e.
     it disables tokenization and applies a `floor` smoothing with value `0.1`.
 
+    This convenience call does not apply any tokenization at all,
+    neither to the system output nor the reference. It just computes
+    BLEU on the "raw corpus" (hence the name).
+
     :param hypotheses: A sequence of hypothesis strings.
     :param references: A sequence of reference documents with document being
         defined as a sequence of reference strings.
     :param smooth_value: The smoothing value for `floor`. If not given, the default of 0.1 is used.
     :return: Returns a `BLEUScore` object.
+
     """
     return corpus_bleu(
         hypotheses, references, smooth_method='floor',
diff --git a/sacrebleu/dataset.py b/sacrebleu/dataset.py
deleted file mode 100644
index 5b4d57ae..00000000
--- a/sacrebleu/dataset.py
+++ /dev/null
@@ -1,892 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-# Copyright 2017--2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License"). You may not
-# use this file except in compliance with the License. A copy of the License
-# is located at
-#
-#     http://aws.amazon.com/apache2.0/
-#
-# or in the "license" file accompanying this file. This file is distributed on
-# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
-# express or implied. See the License for the specific language governing
-# permissions and limitations under the License.
-
-
-# This defines data locations.
-# At the top level are test sets.
-# Beneath each test set, we define the location to download the test data.
-# The other keys are each language pair contained in the tarball, and the respective locations of the source and reference data within each.
-# Many of these are *.sgm files, which are processed to produced plain text that can be used by this script.
-# The canonical location of unpacked, processed data is $SACREBLEU_DIR/$TEST/$SOURCE-$TARGET.{$SOURCE,$TARGET}
-DATASETS = {
-    "mtedx/valid": {
-        "data": ['https://raw.githubusercontent.com/esalesky/mtedx-eval/main/valid.tar.gz'],
-        "description": 'mTEDx evaluation data, valid: http://openslr.org/100',
-        "citation": "@misc{salesky2021multilingual,\n      title={The Multilingual TEDx Corpus for Speech Recognition and Translation}, \n      author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post},\n      year={2021},\n      eprint={2102.01757},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}",
-        "md5": ['40618171614c50e6cbb5e5bbceee0635'],
-        "el-en": ['valid/mtedx-valid-elen.el', 'valid/mtedx-valid-elen.en'],
-        "es-en": ['valid/mtedx-valid-esen.es', 'valid/mtedx-valid-esen.en'],
-        "es-fr": ['valid/mtedx-valid-esfr.es', 'valid/mtedx-valid-esfr.fr'],
-        "es-it": ['valid/mtedx-valid-esit.es', 'valid/mtedx-valid-esit.it'],
-        "es-pt": ['valid/mtedx-valid-espt.es', 'valid/mtedx-valid-espt.pt'],
-        "fr-en": ['valid/mtedx-valid-fren.fr', 'valid/mtedx-valid-fren.en'],
-        "fr-es": ['valid/mtedx-valid-fres.fr', 'valid/mtedx-valid-fres.es'],
-        "fr-pt": ['valid/mtedx-valid-frpt.fr', 'valid/mtedx-valid-frpt.pt'],
-        "it-en": ['valid/mtedx-valid-iten.it', 'valid/mtedx-valid-iten.en'],
-        "it-es": ['valid/mtedx-valid-ites.it', 'valid/mtedx-valid-ites.es'],
-        "pt-en": ['valid/mtedx-valid-pten.pt', 'valid/mtedx-valid-pten.en'],
-        "pt-es": ['valid/mtedx-valid-ptes.pt', 'valid/mtedx-valid-ptes.es'],
-        "ru-en": ['valid/mtedx-valid-ruen.ru', 'valid/mtedx-valid-ruen.en']
-    },
-    "mtedx/test": {
-        "data": ['https://raw.githubusercontent.com/esalesky/mtedx-eval/main/test.tar.gz'],
-        "description": 'mTEDx evaluation data, test: http://openslr.org/100',
-        "citation": "@misc{salesky2021multilingual,\n      title={The Multilingual TEDx Corpus for Speech Recognition and Translation}, \n      author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post},\n      year={2021},\n      eprint={2102.01757},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}",
-        "md5": ['fa4cb1548c210ec424d7d6bc9a3675a7'],
-        "el-en": ['test/mtedx-test-elen.el', 'test/mtedx-test-elen.en'],
-        "es-en": ['test/mtedx-test-esen.es', 'test/mtedx-test-esen.en'],
-        "es-fr": ['test/mtedx-test-esfr.es', 'test/mtedx-test-esfr.fr'],
-        "es-it": ['test/mtedx-test-esit.es', 'test/mtedx-test-esit.it'],
-        "es-pt": ['test/mtedx-test-espt.es', 'test/mtedx-test-espt.pt'],
-        "fr-en": ['test/mtedx-test-fren.fr', 'test/mtedx-test-fren.en'],
-        "fr-es": ['test/mtedx-test-fres.fr', 'test/mtedx-test-fres.es'],
-        "fr-pt": ['test/mtedx-test-frpt.fr', 'test/mtedx-test-frpt.pt'],
-        "it-en": ['test/mtedx-test-iten.it', 'test/mtedx-test-iten.en'],
-        "it-es": ['test/mtedx-test-ites.it', 'test/mtedx-test-ites.es'],
-        "pt-en": ['test/mtedx-test-pten.pt', 'test/mtedx-test-pten.en'],
-        "pt-es": ['test/mtedx-test-ptes.pt', 'test/mtedx-test-ptes.es'],
-        "ru-en": ['test/mtedx-test-ruen.ru', 'test/mtedx-test-ruen.en']
-    },
-    "wmt20/robust/set1": {
-        "data": ["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
-        'description': "WMT20 robustness task, set 1",
-        "en-ja": ["robustness20-3-sets/robustness20-set1-enja.en", "robustness20-3-sets/robustness20-set1-enja.ja"],
-        "en-de": ["robustness20-3-sets/robustness20-set1-ende.en", "robustness20-3-sets/robustness20-set1-ende.de"],
-    },
-    "wmt20/robust/set2": {
-        "data": ["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
-        'description': "WMT20 robustness task, set 2",
-        "en-ja": ["robustness20-3-sets/robustness20-set2-enja.en", "robustness20-3-sets/robustness20-set2-enja.ja"],
-        "ja-en": ["robustness20-3-sets/robustness20-set2-jaen.ja", "robustness20-3-sets/robustness20-set2-jaen.en"],
-    },
-    "wmt20/robust/set3": {
-        "data": ["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
-        'description': "WMT20 robustness task, set 3",
-        "de-en": ["robustness20-3-sets/robustness20-set3-deen.de", "robustness20-3-sets/robustness20-set3-deen.en"],
-    },
-    "wmt20/tworefs": {
-        'data': ['http://data.statmt.org/wmt20/translation-task/test.tgz'],
-        'description': 'WMT20 news test sets with two references',
-        'md5': ['3b1f777cfd2fb15ccf66e9bfdb2b1699'],
-        'de-en': ['sgm/newstest2020-deen-src.de.sgm', 'sgm/newstest2020-deen-ref.en.sgm', 'sgm/newstestB2020-deen-ref.en.sgm'],
-        'en-de': ['sgm/newstest2020-ende-src.en.sgm', 'sgm/newstest2020-ende-ref.de.sgm', 'sgm/newstestB2020-ende-ref.de.sgm'],
-        'en-zh': ['sgm/newstest2020-enzh-src.en.sgm', 'sgm/newstest2020-enzh-ref.zh.sgm', 'sgm/newstestB2020-enzh-ref.zh.sgm'],
-        'ru-en': ['sgm/newstest2020-ruen-src.ru.sgm', 'sgm/newstest2020-ruen-ref.en.sgm', 'sgm/newstestB2020-ruen-ref.en.sgm'],
-        'zh-en': ['sgm/newstest2020-zhen-src.zh.sgm', 'sgm/newstest2020-zhen-ref.en.sgm', 'sgm/newstestB2020-zhen-ref.en.sgm'],
-    },
-    "wmt20": {
-        'data': ['http://data.statmt.org/wmt20/translation-task/test.tgz'],
-        'description': 'Official evaluation data for WMT20',
-        'md5': ['3b1f777cfd2fb15ccf66e9bfdb2b1699'],
-        'cs-en': ['sgm/newstest2020-csen-src.cs.sgm', 'sgm/newstest2020-csen-ref.en.sgm'],
-        'de-en': ['sgm/newstest2020-deen-src.de.sgm', 'sgm/newstest2020-deen-ref.en.sgm'],
-        'de-fr': ['sgm/newstest2020-defr-src.de.sgm', 'sgm/newstest2020-defr-ref.fr.sgm'],
-        'en-cs': ['sgm/newstest2020-encs-src.en.sgm', 'sgm/newstest2020-encs-ref.cs.sgm'],
-        'en-de': ['sgm/newstest2020-ende-src.en.sgm', 'sgm/newstest2020-ende-ref.de.sgm'],
-        'en-iu': ['sgm/newstest2020-eniu-src.en.sgm', 'sgm/newstest2020-eniu-ref.iu.sgm'],
-        'en-ja': ['sgm/newstest2020-enja-src.en.sgm', 'sgm/newstest2020-enja-ref.ja.sgm'],
-        'en-km': ['sgm/newstest2020-enkm-src.en.sgm', 'sgm/newstest2020-enkm-ref.km.sgm'],
-        'en-pl': ['sgm/newstest2020-enpl-src.en.sgm', 'sgm/newstest2020-enpl-ref.pl.sgm'],
-        'en-ps': ['sgm/newstest2020-enps-src.en.sgm', 'sgm/newstest2020-enps-ref.ps.sgm'],
-        'en-ru': ['sgm/newstest2020-enru-src.en.sgm', 'sgm/newstest2020-enru-ref.ru.sgm'],
-        'en-ta': ['sgm/newstest2020-enta-src.en.sgm', 'sgm/newstest2020-enta-ref.ta.sgm'],
-        'en-zh': ['sgm/newstest2020-enzh-src.en.sgm', 'sgm/newstest2020-enzh-ref.zh.sgm'],
-        'fr-de': ['sgm/newstest2020-frde-src.fr.sgm', 'sgm/newstest2020-frde-ref.de.sgm'],
-        'iu-en': ['sgm/newstest2020-iuen-src.iu.sgm', 'sgm/newstest2020-iuen-ref.en.sgm'],
-        'ja-en': ['sgm/newstest2020-jaen-src.ja.sgm', 'sgm/newstest2020-jaen-ref.en.sgm'],
-        'km-en': ['sgm/newstest2020-kmen-src.km.sgm', 'sgm/newstest2020-kmen-ref.en.sgm'],
-        'pl-en': ['sgm/newstest2020-plen-src.pl.sgm', 'sgm/newstest2020-plen-ref.en.sgm'],
-        'ps-en': ['sgm/newstest2020-psen-src.ps.sgm', 'sgm/newstest2020-psen-ref.en.sgm'],
-        'ru-en': ['sgm/newstest2020-ruen-src.ru.sgm', 'sgm/newstest2020-ruen-ref.en.sgm'],
-        'ta-en': ['sgm/newstest2020-taen-src.ta.sgm', 'sgm/newstest2020-taen-ref.en.sgm'],
-        'zh-en': ['sgm/newstest2020-zhen-src.zh.sgm', 'sgm/newstest2020-zhen-ref.en.sgm'],
-    },
-    'mtnt2019': {
-        'data': ['http://www.cs.cmu.edu/~pmichel1/hosting/MTNT2019.tar.gz'],
-        'description': 'Test set for the WMT 19 robustness shared task',
-        'md5': ['78a672e1931f106a8549023c0e8af8f6'],
-        'en-fr': ['2:MTNT2019/en-fr.final.tsv', '3:MTNT2019/en-fr.final.tsv'],
-        'fr-en': ['2:MTNT2019/fr-en.final.tsv', '3:MTNT2019/fr-en.final.tsv'],
-        'en-ja': ['2:MTNT2019/en-ja.final.tsv', '3:MTNT2019/en-ja.final.tsv'],
-        'ja-en': ['2:MTNT2019/ja-en.final.tsv', '3:MTNT2019/ja-en.final.tsv'],
-    },
-    'mtnt1.1/test': {
-        'data': ['https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz'],
-        'description': 'Test data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/',
-        'citation': '@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
-        'md5': ['8ce1831ac584979ba8cdcd9d4be43e1d'],
-        'en-fr': ['1:MTNT/test/test.en-fr.tsv', '2:MTNT/test/test.en-fr.tsv'],
-        'fr-en': ['1:MTNT/test/test.fr-en.tsv', '2:MTNT/test/test.fr-en.tsv'],
-        'en-ja': ['1:MTNT/test/test.en-ja.tsv', '2:MTNT/test/test.en-ja.tsv'],
-        'ja-en': ['1:MTNT/test/test.ja-en.tsv', '2:MTNT/test/test.ja-en.tsv'],
-    },
-    'mtnt1.1/valid': {
-        'data': ['https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz'],
-        'description': 'Validation data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/',
-        'citation': '@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
-        'md5': ['8ce1831ac584979ba8cdcd9d4be43e1d'],
-        'en-fr': ['1:MTNT/valid/valid.en-fr.tsv', '2:MTNT/valid/valid.en-fr.tsv'],
-        'fr-en': ['1:MTNT/valid/valid.fr-en.tsv', '2:MTNT/valid/valid.fr-en.tsv'],
-        'en-ja': ['1:MTNT/valid/valid.en-ja.tsv', '2:MTNT/valid/valid.en-ja.tsv'],
-        'ja-en': ['1:MTNT/valid/valid.ja-en.tsv', '2:MTNT/valid/valid.ja-en.tsv'],
-    },
-    'mtnt1.1/train': {
-        'data': ['https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz'],
-        'description': 'Training data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/',
-        'citation': '@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
-        'md5': ['8ce1831ac584979ba8cdcd9d4be43e1d'],
-        'en-fr': ['1:MTNT/train/train.en-fr.tsv', '2:MTNT/train/train.en-fr.tsv'],
-        'fr-en': ['1:MTNT/train/train.fr-en.tsv', '2:MTNT/train/train.fr-en.tsv'],
-        'en-ja': ['1:MTNT/train/train.en-ja.tsv', '2:MTNT/train/train.en-ja.tsv'],
-        'ja-en': ['1:MTNT/train/train.ja-en.tsv', '2:MTNT/train/train.ja-en.tsv'],
-    },
-    'wmt20/dev': {
-        'data': ['http://data.statmt.org/wmt20/translation-task/dev.tgz'],
-        'description': 'Development data for tasks new to 2020.',
-        'md5': ['037f2b37aab74febbb1b2307dc2afb54'],
-        'iu-en': ['dev/newsdev2020-iuen-src.iu.sgm', 'dev/newsdev2020-iuen-ref.en.sgm'],
-        'en-iu': ['dev/newsdev2020-eniu-src.en.sgm', 'dev/newsdev2020-eniu-ref.iu.sgm'],
-        'ja-en': ['dev/newsdev2020-jaen-src.ja.sgm', 'dev/newsdev2020-jaen-ref.en.sgm'],
-        'en-ja': ['dev/newsdev2020-enja-src.en.sgm', 'dev/newsdev2020-enja-ref.ja.sgm'],
-        'pl-en': ['dev/newsdev2020-plen-src.pl.sgm', 'dev/newsdev2020-plen-ref.en.sgm'],
-        'en-pl': ['dev/newsdev2020-enpl-src.en.sgm', 'dev/newsdev2020-enpl-ref.pl.sgm'],
-        'ta-en': ['dev/newsdev2020-taen-src.ta.sgm', 'dev/newsdev2020-taen-ref.en.sgm'],
-        'en-ta': ['dev/newsdev2020-enta-src.en.sgm', 'dev/newsdev2020-enta-ref.ta.sgm'],
-    },
-    'wmt19': {
-        'data': ['http://data.statmt.org/wmt19/translation-task/test.tgz'],
-        'description': 'Official evaluation data.',
-        'md5': ['84de7162d158e28403103b01aeefc39a'],
-        'citation': r"""@proceedings{ws-2019-machine,
-    title = "Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)",
-    editor = "Bojar, Ond{\v{r}}ej  and
-      Chatterjee, Rajen  and
-      Federmann, Christian  and
-      Fishel, Mark  and
-      Graham, Yvette  and
-      Haddow, Barry  and
-      Huck, Matthias  and
-      Yepes, Antonio Jimeno  and
-      Koehn, Philipp  and
-      Martins, Andr{\'e}  and
-      Monz, Christof  and
-      Negri, Matteo  and
-      N{\'e}v{\'e}ol, Aur{\'e}lie  and
-      Neves, Mariana  and
-      Post, Matt  and
-      Turchi, Marco  and
-      Verspoor, Karin",
-    month = aug,
-    year = "2019",
-    address = "Florence, Italy",
-    publisher = "Association for Computational Linguistics",
-    url = "https://www.aclweb.org/anthology/W19-5200",
-}""",
-        'cs-de': ['sgm/newstest2019-csde-src.cs.sgm', 'sgm/newstest2019-csde-ref.de.sgm'],
-        'de-cs': ['sgm/newstest2019-decs-src.de.sgm', 'sgm/newstest2019-decs-ref.cs.sgm'],
-        'de-en': ['sgm/newstest2019-deen-src.de.sgm', 'sgm/newstest2019-deen-ref.en.sgm'],
-        'de-fr': ['sgm/newstest2019-defr-src.de.sgm', 'sgm/newstest2019-defr-ref.fr.sgm'],
-        'en-cs': ['sgm/newstest2019-encs-src.en.sgm', 'sgm/newstest2019-encs-ref.cs.sgm'],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', 'sgm/newstest2019-ende-ref.de.sgm'],
-        'en-fi': ['sgm/newstest2019-enfi-src.en.sgm', 'sgm/newstest2019-enfi-ref.fi.sgm'],
-        'en-gu': ['sgm/newstest2019-engu-src.en.sgm', 'sgm/newstest2019-engu-ref.gu.sgm'],
-        'en-kk': ['sgm/newstest2019-enkk-src.en.sgm', 'sgm/newstest2019-enkk-ref.kk.sgm'],
-        'en-lt': ['sgm/newstest2019-enlt-src.en.sgm', 'sgm/newstest2019-enlt-ref.lt.sgm'],
-        'en-ru': ['sgm/newstest2019-enru-src.en.sgm', 'sgm/newstest2019-enru-ref.ru.sgm'],
-        'en-zh': ['sgm/newstest2019-enzh-src.en.sgm', 'sgm/newstest2019-enzh-ref.zh.sgm'],
-        'fi-en': ['sgm/newstest2019-fien-src.fi.sgm', 'sgm/newstest2019-fien-ref.en.sgm'],
-        'fr-de': ['sgm/newstest2019-frde-src.fr.sgm', 'sgm/newstest2019-frde-ref.de.sgm'],
-        'gu-en': ['sgm/newstest2019-guen-src.gu.sgm', 'sgm/newstest2019-guen-ref.en.sgm'],
-        'kk-en': ['sgm/newstest2019-kken-src.kk.sgm', 'sgm/newstest2019-kken-ref.en.sgm'],
-        'lt-en': ['sgm/newstest2019-lten-src.lt.sgm', 'sgm/newstest2019-lten-ref.en.sgm'],
-        'ru-en': ['sgm/newstest2019-ruen-src.ru.sgm', 'sgm/newstest2019-ruen-ref.en.sgm'],
-        'zh-en': ['sgm/newstest2019-zhen-src.zh.sgm', 'sgm/newstest2019-zhen-ref.en.sgm'],
-    },
-    'wmt19/dev': {
-        'data': ['http://data.statmt.org/wmt19/translation-task/dev.tgz'],
-        'description': 'Development data for tasks new to 2019.',
-        'md5': ['f2ec7af5947c19e0cacb3882eb208002'],
-        'lt-en': ['dev/newsdev2019-lten-src.lt.sgm', 'dev/newsdev2019-lten-ref.en.sgm'],
-        'en-lt': ['dev/newsdev2019-enlt-src.en.sgm', 'dev/newsdev2019-enlt-ref.lt.sgm'],
-        'gu-en': ['dev/newsdev2019-guen-src.gu.sgm', 'dev/newsdev2019-guen-ref.en.sgm'],
-        'en-gu': ['dev/newsdev2019-engu-src.en.sgm', 'dev/newsdev2019-engu-ref.gu.sgm'],
-        'kk-en': ['dev/newsdev2019-kken-src.kk.sgm', 'dev/newsdev2019-kken-ref.en.sgm'],
-        'en-kk': ['dev/newsdev2019-enkk-src.en.sgm', 'dev/newsdev2019-enkk-ref.kk.sgm'],
-    },
-    "wmt19/google/ar": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-ar.ref"],
-        "description": "Additional high-quality reference for WMT19/en-de.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "d66d9e91548ced0ac476f2390e32e2de"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-ar.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    "wmt19/google/arp": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-arp.ref"],
-        "description": "Additional paraphrase of wmt19/google/ar.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "c70ea808cf2bff621ad7a8fddd4deca9"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-arp.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    "wmt19/google/wmtp": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-wmtp.ref"],
-        "description": "Additional paraphrase of the official WMT19 reference.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "587c660ee5fd44727f0db025b71c6a82"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-wmtp.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    "wmt19/google/hqr": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqr.ref"],
-        "description": "Best human selected-reference between wmt19 and wmt19/google/ar.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "d9221135f62d7152de041f5bfc8efaea"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-hqr.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    "wmt19/google/hqp": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqp.ref"],
-        "description": "Best human-selected reference between wmt19/google/arp and wmt19/google/wmtp.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "b7c3a07a59c8eccea5367e9ec5417a8a"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-hqp.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    "wmt19/google/hqall": {
-        "data": ["http://data.statmt.org/wmt19/translation-task/test.tgz",
-                 "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqall.ref"],
-        "description": "Best human-selected reference among original official reference and the Google reference and paraphrases.",
-        "md5": ["84de7162d158e28403103b01aeefc39a", "edecf10ced59e10b703a6fbcf1fa9dfa"],
-        'en-de': ['sgm/newstest2019-ende-src.en.sgm', '../wmt19-ende-hqall.ref'],
-        "citation": "@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
-    },
-    'wmt18': {
-        'data': ['http://data.statmt.org/wmt18/translation-task/test.tgz'],
-        'md5': ['f996c245ecffea23d0006fa4c34e9064'],
-        'description': 'Official evaluation data.',
-        'citation': '@inproceedings{bojar-etal-2018-findings,\n    title = "Findings of the 2018 Conference on Machine Translation ({WMT}18)",\n    author = "Bojar, Ond{\v{r}}ej  and\n      Federmann, Christian  and\n      Fishel, Mark  and\n      Graham, Yvette  and\n      Haddow, Barry  and\n      Koehn, Philipp  and\n      Monz, Christof",\n    booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers",\n    month = oct,\n    year = "2018",\n    address = "Belgium, Brussels",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W18-6401",\n    pages = "272--303",\n}',
-        'cs-en': ['test/newstest2018-csen-src.cs.sgm', 'test/newstest2018-csen-ref.en.sgm'],
-        'de-en': ['test/newstest2018-deen-src.de.sgm', 'test/newstest2018-deen-ref.en.sgm'],
-        'en-cs': ['test/newstest2018-encs-src.en.sgm', 'test/newstest2018-encs-ref.cs.sgm'],
-        'en-de': ['test/newstest2018-ende-src.en.sgm', 'test/newstest2018-ende-ref.de.sgm'],
-        'en-et': ['test/newstest2018-enet-src.en.sgm', 'test/newstest2018-enet-ref.et.sgm'],
-        'en-fi': ['test/newstest2018-enfi-src.en.sgm', 'test/newstest2018-enfi-ref.fi.sgm'],
-        'en-ru': ['test/newstest2018-enru-src.en.sgm', 'test/newstest2018-enru-ref.ru.sgm'],
-        'et-en': ['test/newstest2018-eten-src.et.sgm', 'test/newstest2018-eten-ref.en.sgm'],
-        'fi-en': ['test/newstest2018-fien-src.fi.sgm', 'test/newstest2018-fien-ref.en.sgm'],
-        'ru-en': ['test/newstest2018-ruen-src.ru.sgm', 'test/newstest2018-ruen-ref.en.sgm'],
-        'en-tr': ['test/newstest2018-entr-src.en.sgm', 'test/newstest2018-entr-ref.tr.sgm'],
-        'tr-en': ['test/newstest2018-tren-src.tr.sgm', 'test/newstest2018-tren-ref.en.sgm'],
-        'en-zh': ['test/newstest2018-enzh-src.en.sgm', 'test/newstest2018-enzh-ref.zh.sgm'],
-        'zh-en': ['test/newstest2018-zhen-src.zh.sgm', 'test/newstest2018-zhen-ref.en.sgm'],
-    },
-    'wmt18/test-ts': {
-        'data': ['http://data.statmt.org/wmt18/translation-task/test-ts.tgz'],
-        'md5': ['5c621a34d512cc2dd74162ae7d00b320'],
-        'description': 'Official evaluation sources with extra test sets interleaved.',
-        'cs-en': ['test-ts/newstest2018-csen-src-ts.cs.sgm'],
-        'de-en': ['test-ts/newstest2018-deen-src-ts.de.sgm'],
-        'en-cs': ['test-ts/newstest2018-encs-src-ts.en.sgm'],
-        'en-de': ['test-ts/newstest2018-ende-src-ts.en.sgm'],
-        'en-et': ['test-ts/newstest2018-enet-src-ts.en.sgm'],
-        'en-fi': ['test-ts/newstest2018-enfi-src-ts.en.sgm'],
-        'en-ru': ['test-ts/newstest2018-enru-src-ts.en.sgm'],
-        'et-en': ['test-ts/newstest2018-eten-src-ts.et.sgm'],
-        'fi-en': ['test-ts/newstest2018-fien-src-ts.fi.sgm'],
-        'ru-en': ['test-ts/newstest2018-ruen-src-ts.ru.sgm'],
-        'en-tr': ['test-ts/newstest2018-entr-src-ts.en.sgm'],
-        'tr-en': ['test-ts/newstest2018-tren-src-ts.tr.sgm'],
-        'en-zh': ['test-ts/newstest2018-enzh-src-ts.en.sgm'],
-        'zh-en': ['test-ts/newstest2018-zhen-src-ts.zh.sgm'],
-    },
-    'wmt18/dev': {
-        'data': ['http://data.statmt.org/wmt18/translation-task/dev.tgz'],
-        'md5': ['486f391da54a7a3247f02ebd25996f24'],
-        'description': 'Development data (Estonian<>English).',
-        'et-en': ['dev/newsdev2018-eten-src.et.sgm', 'dev/newsdev2018-eten-ref.en.sgm'],
-        'en-et': ['dev/newsdev2018-enet-src.en.sgm', 'dev/newsdev2018-enet-ref.et.sgm'],
-    },
-    'wmt17': {
-        'data': ['http://data.statmt.org/wmt17/translation-task/test.tgz'],
-        'md5': ['86a1724c276004aa25455ae2a04cef26'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{bojar-EtAl:2017:WMT1,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Graham, Yvette  and  Haddow, Barry  and  Huang, Shujian  and  Huck, Matthias  and  Koehn, Philipp  and  Liu, Qun  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Post, Matt  and  Rubino, Raphael  and  Specia, Lucia  and  Turchi, Marco},\n  title     = {Findings of the 2017 Conference on Machine Translation (WMT17)},\n  booktitle = {Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers},\n  month     = {September},\n  year      = {2017},\n  address   = {Copenhagen, Denmark},\n  publisher = {Association for Computational Linguistics},\n  pages     = {169--214},\n  url       = {http://www.aclweb.org/anthology/W17-4717}\n}',
-        'cs-en': ['test/newstest2017-csen-src.cs.sgm', 'test/newstest2017-csen-ref.en.sgm'],
-        'de-en': ['test/newstest2017-deen-src.de.sgm', 'test/newstest2017-deen-ref.en.sgm'],
-        'en-cs': ['test/newstest2017-encs-src.en.sgm', 'test/newstest2017-encs-ref.cs.sgm'],
-        'en-de': ['test/newstest2017-ende-src.en.sgm', 'test/newstest2017-ende-ref.de.sgm'],
-        'en-fi': ['test/newstest2017-enfi-src.en.sgm', 'test/newstest2017-enfi-ref.fi.sgm'],
-        'en-lv': ['test/newstest2017-enlv-src.en.sgm', 'test/newstest2017-enlv-ref.lv.sgm'],
-        'en-ru': ['test/newstest2017-enru-src.en.sgm', 'test/newstest2017-enru-ref.ru.sgm'],
-        'en-tr': ['test/newstest2017-entr-src.en.sgm', 'test/newstest2017-entr-ref.tr.sgm'],
-        'en-zh': ['test/newstest2017-enzh-src.en.sgm', 'test/newstest2017-enzh-ref.zh.sgm'],
-        'fi-en': ['test/newstest2017-fien-src.fi.sgm', 'test/newstest2017-fien-ref.en.sgm'],
-        'lv-en': ['test/newstest2017-lven-src.lv.sgm', 'test/newstest2017-lven-ref.en.sgm'],
-        'ru-en': ['test/newstest2017-ruen-src.ru.sgm', 'test/newstest2017-ruen-ref.en.sgm'],
-        'tr-en': ['test/newstest2017-tren-src.tr.sgm', 'test/newstest2017-tren-ref.en.sgm'],
-        'zh-en': ['test/newstest2017-zhen-src.zh.sgm', 'test/newstest2017-zhen-ref.en.sgm'],
-    },
-    'wmt17/B': {
-        'data': ['http://data.statmt.org/wmt17/translation-task/test.tgz'],
-        'md5': ['86a1724c276004aa25455ae2a04cef26'],
-        'description': 'Additional reference for EN-FI and FI-EN.',
-        'en-fi': ['test/newstestB2017-enfi-src.en.sgm', 'test/newstestB2017-enfi-ref.fi.sgm'],
-    },
-    'wmt17/tworefs': {
-        'data': ['http://data.statmt.org/wmt17/translation-task/test.tgz'],
-        'md5': ['86a1724c276004aa25455ae2a04cef26'],
-        'description': 'Systems with two references.',
-        'en-fi': ['test/newstest2017-enfi-src.en.sgm', 'test/newstest2017-enfi-ref.fi.sgm', 'test/newstestB2017-enfi-ref.fi.sgm'],
-    },
-    'wmt17/improved': {
-        'data': ['http://data.statmt.org/wmt17/translation-task/test-update-1.tgz'],
-        'md5': ['91dbfd5af99bc6891a637a68e04dfd41'],
-        'description': 'Improved zh-en and en-zh translations.',
-        'en-zh': ['newstest2017-enzh-src.en.sgm', 'newstest2017-enzh-ref.zh.sgm'],
-        'zh-en': ['newstest2017-zhen-src.zh.sgm', 'newstest2017-zhen-ref.en.sgm'],
-    },
-    'wmt17/dev': {
-        'data': ['http://data.statmt.org/wmt17/translation-task/dev.tgz'],
-        'md5': ['9b1aa63c1cf49dccdd20b962fe313989'],
-        'description': 'Development sets released for new languages in 2017.',
-        'en-lv': ['dev/newsdev2017-enlv-src.en.sgm', 'dev/newsdev2017-enlv-ref.lv.sgm'],
-        'en-zh': ['dev/newsdev2017-enzh-src.en.sgm', 'dev/newsdev2017-enzh-ref.zh.sgm'],
-        'lv-en': ['dev/newsdev2017-lven-src.lv.sgm', 'dev/newsdev2017-lven-ref.en.sgm'],
-        'zh-en': ['dev/newsdev2017-zhen-src.zh.sgm', 'dev/newsdev2017-zhen-ref.en.sgm'],
-    },
-    'wmt17/ms': {
-        'data': ['https://github.com/MicrosoftTranslator/Translator-HumanParityData/archive/master.zip',
-                 'http://data.statmt.org/wmt17/translation-task/test-update-1.tgz'],
-        'md5': ['18fdaa7a3c84cf6ef688da1f6a5fa96f', '91dbfd5af99bc6891a637a68e04dfd41'],
-        'description': 'Additional Chinese-English references from Microsoft Research.',
-        'citation': '@inproceedings{achieving-human-parity-on-automatic-chinese-to-english-news-translation,\n  author = {Hassan Awadalla, Hany and Aue, Anthony and Chen, Chang and Chowdhary, Vishal and Clark, Jonathan and Federmann, Christian and Huang, Xuedong and Junczys-Dowmunt, Marcin and Lewis, Will and Li, Mu and Liu, Shujie and Liu, Tie-Yan and Luo, Renqian and Menezes, Arul and Qin, Tao and Seide, Frank and Tan, Xu and Tian, Fei and Wu, Lijun and Wu, Shuangzhi and Xia, Yingce and Zhang, Dongdong and Zhang, Zhirui and Zhou, Ming},\n  title = {Achieving Human Parity on Automatic Chinese to English News Translation},\n  booktitle = {},\n  year = {2018},\n  month = {March},\n  abstract = {Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question naturally arises whether such systems can approach or achieve parity with human translations. In this paper, we first address the problem of how to define and accurately measure human parity in translation. We then describe Microsoft’s machine translation system and measure the quality of its translations on the widely used WMT 2017 news translation task from Chinese to English. We find that our latest neural machine translation system has reached a new state-of-the-art, and that the translation quality is at human parity when compared to professional human translations. We also find that it significantly exceeds the quality of crowd-sourced non-professional translations.},\n  publisher = {},\n  url = {https://www.microsoft.com/en-us/research/publication/achieving-human-parity-on-automatic-chinese-to-english-news-translation/},\n  address = {},\n  pages = {},\n  journal = {},\n  volume = {},\n  chapter = {},\n  isbn = {},\n}',
-        'zh-en': ['newstest2017-zhen-src.zh.sgm', 'newstest2017-zhen-ref.en.sgm', 'Translator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-HT.txt', 'Translator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-PE.txt'],
-    },
-    'wmt16': {
-        'data': ['http://data.statmt.org/wmt16/translation-task/test.tgz'],
-        'md5': ['3d809cd0c2c86adb2c67034d15c4e446'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{bojar-EtAl:2016:WMT1,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Graham, Yvette  and  Haddow, Barry  and  Huck, Matthias  and  Jimeno Yepes, Antonio  and  Koehn, Philipp  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Neveol, Aurelie  and  Neves, Mariana  and  Popel, Martin  and  Post, Matt  and  Rubino, Raphael  and  Scarton, Carolina  and  Specia, Lucia  and  Turchi, Marco  and  Verspoor, Karin  and  Zampieri, Marcos},\n  title     = {Findings of the 2016 Conference on Machine Translation},\n  booktitle = {Proceedings of the First Conference on Machine Translation},\n  month     = {August},\n  year      = {2016},\n  address   = {Berlin, Germany},\n  publisher = {Association for Computational Linguistics},\n  pages     = {131--198},\n  url       = {http://www.aclweb.org/anthology/W/W16/W16-2301}\n}',
-        'cs-en': ['test/newstest2016-csen-src.cs.sgm', 'test/newstest2016-csen-ref.en.sgm'],
-        'de-en': ['test/newstest2016-deen-src.de.sgm', 'test/newstest2016-deen-ref.en.sgm'],
-        'en-cs': ['test/newstest2016-encs-src.en.sgm', 'test/newstest2016-encs-ref.cs.sgm'],
-        'en-de': ['test/newstest2016-ende-src.en.sgm', 'test/newstest2016-ende-ref.de.sgm'],
-        'en-fi': ['test/newstest2016-enfi-src.en.sgm', 'test/newstest2016-enfi-ref.fi.sgm'],
-        'en-ro': ['test/newstest2016-enro-src.en.sgm', 'test/newstest2016-enro-ref.ro.sgm'],
-        'en-ru': ['test/newstest2016-enru-src.en.sgm', 'test/newstest2016-enru-ref.ru.sgm'],
-        'en-tr': ['test/newstest2016-entr-src.en.sgm', 'test/newstest2016-entr-ref.tr.sgm'],
-        'fi-en': ['test/newstest2016-fien-src.fi.sgm', 'test/newstest2016-fien-ref.en.sgm'],
-        'ro-en': ['test/newstest2016-roen-src.ro.sgm', 'test/newstest2016-roen-ref.en.sgm'],
-        'ru-en': ['test/newstest2016-ruen-src.ru.sgm', 'test/newstest2016-ruen-ref.en.sgm'],
-        'tr-en': ['test/newstest2016-tren-src.tr.sgm', 'test/newstest2016-tren-ref.en.sgm'],
-    },
-    'wmt16/B': {
-        'data': ['http://data.statmt.org/wmt16/translation-task/test.tgz'],
-        'md5': ['3d809cd0c2c86adb2c67034d15c4e446'],
-        'description': 'Additional reference for EN-FI.',
-        'en-fi': ['test/newstest2016-enfi-src.en.sgm', 'test/newstestB2016-enfi-ref.fi.sgm'],
-    },
-    'wmt16/tworefs': {
-        'data': ['http://data.statmt.org/wmt16/translation-task/test.tgz'],
-        'md5': ['3d809cd0c2c86adb2c67034d15c4e446'],
-        'description': 'EN-FI with two references.',
-        'en-fi': ['test/newstest2016-enfi-src.en.sgm', 'test/newstest2016-enfi-ref.fi.sgm', 'test/newstestB2016-enfi-ref.fi.sgm'],
-    },
-    'wmt16/dev': {
-        'data': ['http://data.statmt.org/wmt16/translation-task/dev.tgz'],
-        'md5': ['4a3dc2760bb077f4308cce96b06e6af6'],
-        'description': 'Development sets released for new languages in 2016.',
-        'en-ro': ['dev/newsdev2016-enro-src.en.sgm', 'dev/newsdev2016-enro-ref.ro.sgm'],
-        'en-tr': ['dev/newsdev2016-entr-src.en.sgm', 'dev/newsdev2016-entr-ref.tr.sgm'],
-        'ro-en': ['dev/newsdev2016-roen-src.ro.sgm', 'dev/newsdev2016-roen-ref.en.sgm'],
-        'tr-en': ['dev/newsdev2016-tren-src.tr.sgm', 'dev/newsdev2016-tren-ref.en.sgm']
-    },
-    'wmt15': {
-        'data': ['http://statmt.org/wmt15/test.tgz'],
-        'md5': ['67e3beca15e69fe3d36de149da0a96df'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{bojar-EtAl:2015:WMT,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Haddow, Barry  and  Huck, Matthias  and  Hokamp, Chris  and  Koehn, Philipp  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Post, Matt  and  Scarton, Carolina  and  Specia, Lucia  and  Turchi, Marco},\n  title     = {Findings of the 2015 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},\n  month     = {September},\n  year      = {2015},\n  address   = {Lisbon, Portugal},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--46},\n  url       = {http://aclweb.org/anthology/W15-3001}\n}',
-        'en-fr': ['test/newsdiscusstest2015-enfr-src.en.sgm', 'test/newsdiscusstest2015-enfr-ref.fr.sgm'],
-        'fr-en': ['test/newsdiscusstest2015-fren-src.fr.sgm', 'test/newsdiscusstest2015-fren-ref.en.sgm'],
-        'cs-en': ['test/newstest2015-csen-src.cs.sgm', 'test/newstest2015-csen-ref.en.sgm'],
-        'de-en': ['test/newstest2015-deen-src.de.sgm', 'test/newstest2015-deen-ref.en.sgm'],
-        'en-cs': ['test/newstest2015-encs-src.en.sgm', 'test/newstest2015-encs-ref.cs.sgm'],
-        'en-de': ['test/newstest2015-ende-src.en.sgm', 'test/newstest2015-ende-ref.de.sgm'],
-        'en-fi': ['test/newstest2015-enfi-src.en.sgm', 'test/newstest2015-enfi-ref.fi.sgm'],
-        'en-ru': ['test/newstest2015-enru-src.en.sgm', 'test/newstest2015-enru-ref.ru.sgm'],
-        'fi-en': ['test/newstest2015-fien-src.fi.sgm', 'test/newstest2015-fien-ref.en.sgm'],
-        'ru-en': ['test/newstest2015-ruen-src.ru.sgm', 'test/newstest2015-ruen-ref.en.sgm']
-    },
-    'wmt14': {
-        'data': ['http://statmt.org/wmt14/test-filtered.tgz'],
-        'md5': ['84c597844c1542e29c2aff23aaee4310'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{bojar-EtAl:2014:W14-33,\n  author    = {Bojar, Ondrej  and  Buck, Christian  and  Federmann, Christian  and  Haddow, Barry  and  Koehn, Philipp  and  Leveling, Johannes  and  Monz, Christof  and  Pecina, Pavel  and  Post, Matt  and  Saint-Amand, Herve  and  Soricut, Radu  and  Specia, Lucia  and  Tamchyna, Ale\\v{s}},\n  title     = {Findings of the 2014 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2014},\n  address   = {Baltimore, Maryland, USA},\n  publisher = {Association for Computational Linguistics},\n  pages     = {12--58},\n  url       = {http://www.aclweb.org/anthology/W/W14/W14-3302}\n}',
-        'cs-en': ['test/newstest2014-csen-src.cs.sgm', 'test/newstest2014-csen-ref.en.sgm'],
-        'en-cs': ['test/newstest2014-csen-src.en.sgm', 'test/newstest2014-csen-ref.cs.sgm'],
-        'de-en': ['test/newstest2014-deen-src.de.sgm', 'test/newstest2014-deen-ref.en.sgm'],
-        'en-de': ['test/newstest2014-deen-src.en.sgm', 'test/newstest2014-deen-ref.de.sgm'],
-        'en-fr': ['test/newstest2014-fren-src.en.sgm', 'test/newstest2014-fren-ref.fr.sgm'],
-        'fr-en': ['test/newstest2014-fren-src.fr.sgm', 'test/newstest2014-fren-ref.en.sgm'],
-        'en-hi': ['test/newstest2014-hien-src.en.sgm', 'test/newstest2014-hien-ref.hi.sgm'],
-        'hi-en': ['test/newstest2014-hien-src.hi.sgm', 'test/newstest2014-hien-ref.en.sgm'],
-        'en-ru': ['test/newstest2014-ruen-src.en.sgm', 'test/newstest2014-ruen-ref.ru.sgm'],
-        'ru-en': ['test/newstest2014-ruen-src.ru.sgm', 'test/newstest2014-ruen-ref.en.sgm']
-    },
-    'wmt14/full': {
-        'data': ['http://statmt.org/wmt14/test-full.tgz'],
-        'md5': ['a8cd784e006feb32ac6f3d9ec7eb389a'],
-        'description': 'Evaluation data released after official evaluation for further research.',
-        'cs-en': ['test-full/newstest2014-csen-src.cs.sgm', 'test-full/newstest2014-csen-ref.en.sgm'],
-        'en-cs': ['test-full/newstest2014-csen-src.en.sgm', 'test-full/newstest2014-csen-ref.cs.sgm'],
-        'de-en': ['test-full/newstest2014-deen-src.de.sgm', 'test-full/newstest2014-deen-ref.en.sgm'],
-        'en-de': ['test-full/newstest2014-deen-src.en.sgm', 'test-full/newstest2014-deen-ref.de.sgm'],
-        'en-fr': ['test-full/newstest2014-fren-src.en.sgm', 'test-full/newstest2014-fren-ref.fr.sgm'],
-        'fr-en': ['test-full/newstest2014-fren-src.fr.sgm', 'test-full/newstest2014-fren-ref.en.sgm'],
-        'en-hi': ['test-full/newstest2014-hien-src.en.sgm', 'test-full/newstest2014-hien-ref.hi.sgm'],
-        'hi-en': ['test-full/newstest2014-hien-src.hi.sgm', 'test-full/newstest2014-hien-ref.en.sgm'],
-        'en-ru': ['test-full/newstest2014-ruen-src.en.sgm', 'test-full/newstest2014-ruen-ref.ru.sgm'],
-        'ru-en': ['test-full/newstest2014-ruen-src.ru.sgm', 'test-full/newstest2014-ruen-ref.en.sgm']
-    },
-    'wmt13': {
-        'data': ['http://statmt.org/wmt13/test.tgz'],
-        'md5': ['48eca5d02f637af44e85186847141f67'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{bojar-EtAl:2013:WMT,\n  author    = {Bojar, Ond\\v{r}ej  and  Buck, Christian  and  Callison-Burch, Chris  and  Federmann, Christian  and  Haddow, Barry  and  Koehn, Philipp  and  Monz, Christof  and  Post, Matt  and  Soricut, Radu  and  Specia, Lucia},\n  title     = {Findings of the 2013 {Workshop on Statistical Machine Translation}},\n  booktitle = {Proceedings of the Eighth Workshop on Statistical Machine Translation},\n  month     = {August},\n  year      = {2013},\n  address   = {Sofia, Bulgaria},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--44},\n  url       = {http://www.aclweb.org/anthology/W13-2201}\n}',
-        'cs-en': ['test/newstest2013-src.cs.sgm', 'test/newstest2013-src.en.sgm'],
-        'en-cs': ['test/newstest2013-src.en.sgm', 'test/newstest2013-src.cs.sgm'],
-        'de-en': ['test/newstest2013-src.de.sgm', 'test/newstest2013-src.en.sgm'],
-        'en-de': ['test/newstest2013-src.en.sgm', 'test/newstest2013-src.de.sgm'],
-        'es-en': ['test/newstest2013-src.es.sgm', 'test/newstest2013-src.en.sgm'],
-        'en-es': ['test/newstest2013-src.en.sgm', 'test/newstest2013-src.es.sgm'],
-        'fr-en': ['test/newstest2013-src.fr.sgm', 'test/newstest2013-src.en.sgm'],
-        'en-fr': ['test/newstest2013-src.en.sgm', 'test/newstest2013-src.fr.sgm'],
-        'ru-en': ['test/newstest2013-src.ru.sgm', 'test/newstest2013-src.en.sgm'],
-        'en-ru': ['test/newstest2013-src.en.sgm', 'test/newstest2013-src.ru.sgm']
-    },
-    'wmt12': {
-        'data': ['http://statmt.org/wmt12/test.tgz'],
-        'md5': ['608232d34ebc4ba2ff70fead45674e47'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{callisonburch-EtAl:2012:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Post, Matt  and  Soricut, Radu  and  Specia, Lucia},\n  title     = {Findings of the 2012 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Seventh Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2012},\n  address   = {Montr{\'e}al, Canada},\n  publisher = {Association for Computational Linguistics},\n  pages     = {10--51},\n  url       = {http://www.aclweb.org/anthology/W12-3102}\n}',
-        'cs-en': ['test/newstest2012-src.cs.sgm', 'test/newstest2012-src.en.sgm'],
-        'en-cs': ['test/newstest2012-src.en.sgm', 'test/newstest2012-src.cs.sgm'],
-        'de-en': ['test/newstest2012-src.de.sgm', 'test/newstest2012-src.en.sgm'],
-        'en-de': ['test/newstest2012-src.en.sgm', 'test/newstest2012-src.de.sgm'],
-        'es-en': ['test/newstest2012-src.es.sgm', 'test/newstest2012-src.en.sgm'],
-        'en-es': ['test/newstest2012-src.en.sgm', 'test/newstest2012-src.es.sgm'],
-        'fr-en': ['test/newstest2012-src.fr.sgm', 'test/newstest2012-src.en.sgm'],
-        'en-fr': ['test/newstest2012-src.en.sgm', 'test/newstest2012-src.fr.sgm']
-    },
-    'wmt11': {
-        'data': ['http://statmt.org/wmt11/test.tgz'],
-        'md5': ['b0c9680adf32d394aefc2b24e3a5937e'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{callisonburch-EtAl:2011:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Zaidan, Omar},\n  title     = {Findings of the 2011 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation},\n  month     = {July},\n  year      = {2011},\n  address   = {Edinburgh, Scotland},\n  publisher = {Association for Computational Linguistics},\n  pages     = {22--64},\n  url       = {http://www.aclweb.org/anthology/W11-2103}\n}',
-        'cs-en': ['newstest2011-src.cs.sgm', 'newstest2011-src.en.sgm'],
-        'en-cs': ['newstest2011-src.en.sgm', 'newstest2011-src.cs.sgm'],
-        'de-en': ['newstest2011-src.de.sgm', 'newstest2011-src.en.sgm'],
-        'en-de': ['newstest2011-src.en.sgm', 'newstest2011-src.de.sgm'],
-        'fr-en': ['newstest2011-src.fr.sgm', 'newstest2011-src.en.sgm'],
-        'en-fr': ['newstest2011-src.en.sgm', 'newstest2011-src.fr.sgm'],
-        'es-en': ['newstest2011-src.es.sgm', 'newstest2011-src.en.sgm'],
-        'en-es': ['newstest2011-src.en.sgm', 'newstest2011-src.es.sgm']
-    },
-    'wmt10': {
-        'data': ['http://statmt.org/wmt10/test.tgz'],
-        'md5': ['491cb885a355da5a23ea66e7b3024d5c'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{callisonburch-EtAl:2010:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Peterson, Kay  and  Przybocki, Mark  and  Zaidan, Omar},\n  title     = {Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation},\n  booktitle = {Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR},\n  month     = {July},\n  year      = {2010},\n  address   = {Uppsala, Sweden},\n  publisher = {Association for Computational Linguistics},\n  pages     = {17--53},\n  note      = {Revised August 2010},\n  url       = {http://www.aclweb.org/anthology/W10-1703}\n}',
-        'cs-en': ['test/newstest2010-src.cz.sgm', 'test/newstest2010-src.en.sgm'],
-        'en-cs': ['test/newstest2010-src.en.sgm', 'test/newstest2010-src.cz.sgm'],
-        'de-en': ['test/newstest2010-src.de.sgm', 'test/newstest2010-src.en.sgm'],
-        'en-de': ['test/newstest2010-src.en.sgm', 'test/newstest2010-src.de.sgm'],
-        'es-en': ['test/newstest2010-src.es.sgm', 'test/newstest2010-src.en.sgm'],
-        'en-es': ['test/newstest2010-src.en.sgm', 'test/newstest2010-src.es.sgm'],
-        'fr-en': ['test/newstest2010-src.fr.sgm', 'test/newstest2010-src.en.sgm'],
-        'en-fr': ['test/newstest2010-src.en.sgm', 'test/newstest2010-src.fr.sgm']
-    },
-    'wmt09': {
-        'data': ['http://statmt.org/wmt09/test.tgz'],
-        'md5': ['da227abfbd7b666ec175b742a0d27b37'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{callisonburch-EtAl:2009:WMT-09,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Schroeder, Josh},\n  title     = {Findings of the 2009 {W}orkshop on {S}tatistical {M}achine {T}ranslation},\n  booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation},\n  month     = {March},\n  year      = {2009},\n  address   = {Athens, Greece},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--28},\n  url       = {http://www.aclweb.org/anthology/W/W09/W09-0401}\n}',
-        'cs-en': ['test/newstest2009-src.cz.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-cs': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.cz.sgm'],
-        'de-en': ['test/newstest2009-src.de.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-de': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.de.sgm'],
-        'es-en': ['test/newstest2009-src.es.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-es': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.es.sgm'],
-        'fr-en': ['test/newstest2009-src.fr.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-fr': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.fr.sgm'],
-        'hu-en': ['test/newstest2009-src.hu.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-hu': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.hu.sgm'],
-        'it-en': ['test/newstest2009-src.it.sgm', 'test/newstest2009-src.en.sgm'],
-        'en-it': ['test/newstest2009-src.en.sgm', 'test/newstest2009-src.it.sgm']
-    },
-    'wmt08': {
-        'data': ['http://statmt.org/wmt08/test.tgz'],
-        'md5': ['0582e4e894a3342044059c894e1aea3d'],
-        'description': 'Official evaluation data.',
-        'citation': '@InProceedings{callisonburch-EtAl:2008:WMT,\n  author    = {Callison-Burch, Chris  and  Fordyce, Cameron  and  Koehn, Philipp  and  Monz, Christof  and  Schroeder, Josh},\n  title     = {Further Meta-Evaluation of Machine Translation},\n  booktitle = {Proceedings of the Third Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2008},\n  address   = {Columbus, Ohio},\n  publisher = {Association for Computational Linguistics},\n  pages     = {70--106},\n  url       = {http://www.aclweb.org/anthology/W/W08/W08-0309}\n}',
-        'cs-en': ['test/newstest2008-src.cz.sgm', 'test/newstest2008-src.en.sgm'],
-        'en-cs': ['test/newstest2008-src.en.sgm', 'test/newstest2008-src.cz.sgm'],
-        'de-en': ['test/newstest2008-src.de.sgm', 'test/newstest2008-src.en.sgm'],
-        'en-de': ['test/newstest2008-src.en.sgm', 'test/newstest2008-src.de.sgm'],
-        'es-en': ['test/newstest2008-src.es.sgm', 'test/newstest2008-src.en.sgm'],
-        'en-es': ['test/newstest2008-src.en.sgm', 'test/newstest2008-src.es.sgm'],
-        'fr-en': ['test/newstest2008-src.fr.sgm', 'test/newstest2008-src.en.sgm'],
-        'en-fr': ['test/newstest2008-src.en.sgm', 'test/newstest2008-src.fr.sgm'],
-        'hu-en': ['test/newstest2008-src.hu.sgm', 'test/newstest2008-src.en.sgm'],
-        'en-hu': ['test/newstest2008-src.en.sgm', 'test/newstest2008-src.hu.sgm']
-    },
-    'wmt08/nc': {
-        'data': ['http://statmt.org/wmt08/test.tgz'],
-        'md5': ['0582e4e894a3342044059c894e1aea3d'],
-        'description': 'Official evaluation data (news commentary).',
-        'cs-en': ['test/nc-test2008-src.cz.sgm', 'test/nc-test2008-src.en.sgm'],
-        'en-cs': ['test/nc-test2008-src.en.sgm', 'test/nc-test2008-src.cz.sgm']
-    },
-    'wmt08/europarl': {
-        'data': ['http://statmt.org/wmt08/test.tgz'],
-        'md5': ['0582e4e894a3342044059c894e1aea3d'],
-        'description': 'Official evaluation data (Europarl).',
-        'de-en': ['test/test2008-src.de.sgm', 'test/test2008-src.en.sgm'],
-        'en-de': ['test/test2008-src.en.sgm', 'test/test2008-src.de.sgm'],
-        'es-en': ['test/test2008-src.es.sgm', 'test/test2008-src.en.sgm'],
-        'en-es': ['test/test2008-src.en.sgm', 'test/test2008-src.es.sgm'],
-        'fr-en': ['test/test2008-src.fr.sgm', 'test/test2008-src.en.sgm'],
-        'en-fr': ['test/test2008-src.en.sgm', 'test/test2008-src.fr.sgm']
-    },
-    'iwslt17': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ar/en-ar.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ar/en/ar-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ja/en-ja.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ja/en/ja-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ko/en-ko.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ko/en/ko-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/zh/en/zh-en.tgz'],
-        'md5': ["1849bcc3b006dc0642a8843b11aa7192",
-                "79bf7a2ef02d226875f55fb076e7e473",
-                "b68e7097b179491f6c466ef41ad72b9b",
-                "e3f5b2a075a2da1a395c8b60bf1e9be1",
-                "ecdc6bc4ab4c8984e919444f3c05183a",
-                "4b5141d14b98706c081371e2f8afe0ca",
-                "d957ee79de1f33c89077d37c5a2c5b06",
-                "c213e8bb918ebf843543fe9fd2e33db2",
-                "59f6a81c707378176e9ad8bb8d811f5f",
-                "7e580af973bb389ec1d1378a1850742f",
-                "975a858783a0ebec8c57d83ddd5bd381",
-                "cc51d9b7fe1ff2af858c6a0dd80b8815"],
-        'description': 'Official evaluation data for IWSLT.',
-        'citation': '@InProceedings{iwslt2017,\n  author    = {Cettolo, Mauro and Federico, Marcello and Bentivogli, Luisa and Niehues, Jan and Stüker, Sebastian and Sudoh, Katsuitho and Yoshino, Koichiro and Federmann, Christian},\n  title     = {Overview of the IWSLT 2017 Evaluation Campaign},\n  booktitle = {14th International Workshop on Spoken Language Translation},\n  month     = {December},\n  year      = {2017},\n  address   = {Tokyo, Japan},\n  pages     = {2--14},\n  url       = {http://workshop2017.iwslt.org/downloads/iwslt2017_proceeding_v2.pdf}\n}',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2017.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2017.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2017.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2017.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2017.en-de.en.xml', 'de-en/IWSLT17.TED.tst2017.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2017.de-en.de.xml', 'en-de/IWSLT17.TED.tst2017.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2017.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2017.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2017.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2017.en-zh.en.xml'],
-        'en-ar': ['en-ar/IWSLT17.TED.tst2017.en-ar.en.xml', 'ar-en/IWSLT17.TED.tst2017.ar-en.ar.xml'],
-        'ar-en': ['ar-en/IWSLT17.TED.tst2017.ar-en.ar.xml', 'en-ar/IWSLT17.TED.tst2017.en-ar.en.xml'],
-        'en-ja': ['en-ja/IWSLT17.TED.tst2017.en-ja.en.xml', 'ja-en/IWSLT17.TED.tst2017.ja-en.ja.xml'],
-        'ja-en': ['ja-en/IWSLT17.TED.tst2017.ja-en.ja.xml', 'en-ja/IWSLT17.TED.tst2017.en-ja.en.xml'],
-        'en-ko': ['en-ko/IWSLT17.TED.tst2017.en-ko.en.xml', 'ko-en/IWSLT17.TED.tst2017.ko-en.ko.xml'],
-        'ko-en': ['ko-en/IWSLT17.TED.tst2017.ko-en.ko.xml', 'en-ko/IWSLT17.TED.tst2017.en-ko.en.xml'],
-    },
-    'iwslt17/tst2016': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/zh/en/zh-en.tgz'],
-        "md5": ["1849bcc3b006dc0642a8843b11aa7192",
-                "79bf7a2ef02d226875f55fb076e7e473",
-                "b68e7097b179491f6c466ef41ad72b9b",
-                "e3f5b2a075a2da1a395c8b60bf1e9be1",
-                "975a858783a0ebec8c57d83ddd5bd381",
-                "cc51d9b7fe1ff2af858c6a0dd80b8815"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2016.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2016.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2016.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2016.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2016.en-de.en.xml', 'de-en/IWSLT17.TED.tst2016.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2016.de-en.de.xml', 'en-de/IWSLT17.TED.tst2016.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2016.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2016.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2016.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2016.en-zh.en.xml'],
-    },
-    'iwslt17/tst2015': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2015.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2015.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2015.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2015.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2015.en-de.en.xml', 'de-en/IWSLT17.TED.tst2015.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2015.de-en.de.xml', 'en-de/IWSLT17.TED.tst2015.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2015.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2015.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2015.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2015.en-zh.en.xml'],
-    },
-    'iwslt17/tst2014': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2014.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2014.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2014.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2014.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2014.en-de.en.xml', 'de-en/IWSLT17.TED.tst2014.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2014.de-en.de.xml', 'en-de/IWSLT17.TED.tst2014.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2014.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2014.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2014.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2014.en-zh.en.xml'],
-    },
-    'iwslt17/tst2013': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2013.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2013.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2013.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2013.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2013.en-de.en.xml', 'de-en/IWSLT17.TED.tst2013.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2013.de-en.de.xml', 'en-de/IWSLT17.TED.tst2013.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2013.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2013.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2013.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2013.en-zh.en.xml'],
-    },
-    'iwslt17/tst2012': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2012.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2012.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2012.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2012.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2012.en-de.en.xml', 'de-en/IWSLT17.TED.tst2012.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2012.de-en.de.xml', 'en-de/IWSLT17.TED.tst2012.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2012.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2012.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2012.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2012.en-zh.en.xml'],
-    },
-    'iwslt17/tst2011': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2011.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2011.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2011.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2011.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2011.en-de.en.xml', 'de-en/IWSLT17.TED.tst2011.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2011.de-en.de.xml', 'en-de/IWSLT17.TED.tst2011.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2011.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2011.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2011.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2011.en-zh.en.xml'],
-    },
-    'iwslt17/tst2010': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.tst2010.en-fr.en.xml', 'fr-en/IWSLT17.TED.tst2010.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.tst2010.fr-en.fr.xml', 'en-fr/IWSLT17.TED.tst2010.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.tst2010.en-de.en.xml', 'de-en/IWSLT17.TED.tst2010.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.tst2010.de-en.de.xml', 'en-de/IWSLT17.TED.tst2010.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.tst2010.en-zh.en.xml', 'zh-en/IWSLT17.TED.tst2010.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.tst2010.zh-en.zh.xml', 'en-zh/IWSLT17.TED.tst2010.en-zh.en.xml'],
-    },
-    'iwslt17/dev2010': {
-        'data': ['https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz',
-                 'https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz'],
-        "md5": ["d8a32cfc002a4f12b17429cfa78050e6",
-                "ca2b94d694150d4d6c5dc64c200fa589",
-                "3cf07ebe305312b12f7f1a4d5f8f8377",
-                "19927da9de0f40348cad9c0fc61642ac",
-                "575b788dad6c5b9c5cee636f9ac1094a",
-                "1c0ae40171d52593df8a6963d3828116"],
-        'description': 'Development data for IWSLT 2017.',
-        'en-fr': ['en-fr/IWSLT17.TED.dev2010.en-fr.en.xml', 'fr-en/IWSLT17.TED.dev2010.fr-en.fr.xml'],
-        'fr-en': ['fr-en/IWSLT17.TED.dev2010.fr-en.fr.xml', 'en-fr/IWSLT17.TED.dev2010.en-fr.en.xml'],
-        'en-de': ['en-de/IWSLT17.TED.dev2010.en-de.en.xml', 'de-en/IWSLT17.TED.dev2010.de-en.de.xml'],
-        'de-en': ['de-en/IWSLT17.TED.dev2010.de-en.de.xml', 'en-de/IWSLT17.TED.dev2010.en-de.en.xml'],
-        'en-zh': ['en-zh/IWSLT17.TED.dev2010.en-zh.en.xml', 'zh-en/IWSLT17.TED.dev2010.zh-en.zh.xml'],
-        'zh-en': ['zh-en/IWSLT17.TED.dev2010.zh-en.zh.xml', 'en-zh/IWSLT17.TED.dev2010.en-zh.en.xml'],
-    },
-    'multi30k/2016': {
-        'data': ['https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz'],
-        "md5": ['9cf8f22d57fee2ca2af3c682dfdc525b'],
-        'description': '2016 flickr test set of Multi30k dataset',
-        'citation': '@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}',
-        'en-fr': ['test_2016_flickr.en', 'test_2016_flickr.fr'],
-        'en-de': ['test_2016_flickr.en', 'test_2016_flickr.de'],
-        'en-cs': ['test_2016_flickr.en', 'test_2016_flickr.cs'],
-    },
-    'multi30k/2017': {
-        'data': ['https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz'],
-        "md5": ['9cf8f22d57fee2ca2af3c682dfdc525b'],
-        'description': '2017 flickr test set of Multi30k dataset',
-        'citation': '@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}\n\n@InProceedings{elliott-etal-2017-findings,\n    title = "Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description",\n    author = {Elliott, Desmond  and Frank, Stella  and Barrault, Lo{\\"\\i}c  and Bougares, Fethi  and Specia, Lucia},\n    booktitle = "Proceedings of the Second Conference on Machine Translation",\n    month = sep,\n    year = "2017",\n    address = "Copenhagen, Denmark",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W17-4718",\n    doi = "10.18653/v1/W17-4718",\n    pages = "215--233",\n}\n',
-        'en-fr': ['test_2017_flickr.en', 'test_2017_flickr.fr'],
-        'en-de': ['test_2017_flickr.en', 'test_2017_flickr.de'],
-    },
-    'multi30k/2018': {
-        'data': ['https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz'],
-        "md5": ['9cf8f22d57fee2ca2af3c682dfdc525b'],
-        'description': '2018 flickr test set of Multi30k dataset. See https://competitions.codalab.org/competitions/19917 for evaluation.',
-        'citation': '@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}\n\n@InProceedings{barrault-etal-2018-findings,\n    title = "Findings of the Third Shared Task on Multimodal Machine Translation",\n    author = {Barrault, Lo{\\"\\i}c  and Bougares, Fethi  and Specia, Lucia  and Lala, Chiraag  and Elliott, Desmond  and Frank, Stella},\n    booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers",\n    month = oct,\n    year = "2018",\n    address = "Belgium, Brussels",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W18-6402",\n    doi = "10.18653/v1/W18-6402",\n    pages = "304--323",\n}\n',
-        'en-fr': ['test_2018_flickr.en'],
-        'en-de': ['test_2018_flickr.en'],
-    },
-}
-
-# Detailed document metadata annotation in form DocumentID -> CountryCode - Domain - OptionalFinegrainedCountryCode
-# While the annotation is subjective with many unclear cases, it may provide useful insights
-# when applied on large data (TODO: annotate all documents from recent WMT years, at least for origlang=en, consider renaming "world" to "other").
-_SUBSETS = {
-    'wmt18': 'rt.com.68098=US-crime guardian.181611=US-politics bbc.310963=GB-sport washpost.116881=US-politics scotsman.104228=GB-sport timemagazine.75207=OTHER-world-ID '
-             'euronews-en.117981=OTHER-crime-AE smh.com.au.242810=US-crime msnbc.53726=US-politics euronews-en.117983=US-politics msnbc.53894=US-crime theglobeandmail.com.62700=US-business '
-             'bbc.310870=OTHER-world-AF reuters.196698=US-politics latimes.231739=US-sport thelocal.51929=OTHER-world-SE cbsnews.198694=US-politics reuters.196718=OTHER-sport-RU '
-             'abcnews.255599=EU-sport nytimes.127256=US-entertainment scotsman.104225=GB-politics dailymail.co.uk.233026=GB-scitech independent.181088=GB-entertainment '
-             'brisbanetimes.com.au.181614=OTHER-business-AU washpost.116837=US-politics dailymail.co.uk.232928=GB-world thelocal.51916=OTHER-politics-IT bbc.310871=US-crime '
-             'nytimes.127392=EU-business-DE euronews-en.118001=EU-scitech-FR washpost.116866=OTHER-crime-MX dailymail.co.uk.233025=OTHER-scitech-CA latimes.231829=US-crime '
-             'guardian.181662=US-entertainment msnbc.53731=US-crime rt.com.68127=OTHER-sport-RU latimes.231782=US-business latimes.231840=US-sport reuters.196711=OTHER-scitech '
-             'guardian.181666=GB-entertainment novinite.com.24019=US-politics smh.com.au.242750=OTHER-scitech guardian.181610=US-politics telegraph.364393=OTHER-crime-ZA '
-             'novinite.com.23995=EU-world dailymail.co.uk.233028=GB-scitech independent.181071=GB-sport telegraph.364538=GB-scitech timemagazine.75193=US-politics '
-             'independent.181096=US-entertainment upi.140602=OTHER-world-AF bbc.310946=GB-business independent.181052=EU-sport ',
-    'wmt19': 'bbc.381790=GB-politics rt.com.91337=OTHER-politics-MK nytimes.184853=US-world upi.176266=US-crime guardian.221754=GB-business dailymail.co.uk.298595=GB-business '
-             'cnbc.com.6790=US-politics nytimes.184837=OTHER-world-ID upi.176249=GB-sport euronews-en.153835=OTHER-world-ID dailymail.co.uk.298732=GB-crime telegraph.405401=GB-politics '
-             'newsweek.51331=OTHER-crime-CN abcnews.306815=US-world cbsnews.248384=US-politics reuters.218882=GB-politics cbsnews.248387=US-crime abcnews.306764=OTHER-world-MX '
-             'reuters.218888=EU-politics bbc.381780=GB-crime bbc.381746=GB-sport euronews-en.153800=EU-politics bbc.381679=GB-crime bbc.381735=GB-crime newsweek.51338=US-world '
-             'bbc.381765=GB-crime cnn.304489=US-politics reuters.218863=OTHER-world-ID nytimes.184860=OTHER-world-ID cnn.304404=US-crime bbc.381647=US-entertainment '
-             'abcnews.306758=OTHER-politics-MX cnbc.com.6772=US-business reuters.218932=OTHER-politics-MK upi.176251=GB-sport reuters.218921=US-sport cnn.304447=US-politics '
-             'guardian.221679=GB-politics scotsman.133765=GB-sport scotsman.133804=GB-entertainment guardian.221762=OTHER-politics-BO cnbc.com.6769=US-politics '
-             'dailymail.co.uk.298692=EU-entertainment scotsman.133744=GB-world reuters.218911=US-sport newsweek.51310=US-politics independent.226301=US-sport reuters.218923=EU-sport '
-             'reuters.218861=US-politics dailymail.co.uk.298759=US-world scotsman.133791=GB-sport cbsnews.248484=EU-scitech dailymail.co.uk.298630=US-scitech '
-             'newsweek.51329=US-entertainment bbc.381701=GB-crime dailymail.co.uk.298738=GB-entertainment bbc.381669=OTHER-world-CN foxnews.94512=US-politics '
-             'guardian.221718=GB-entertainment dailymail.co.uk.298686=GB-politics cbsnews.248471=US-politics newsweek.51318=US-entertainment rt.com.91335=US-politics '
-             'newsweek.51300=US-politics cnn.304478=US-politics upi.176275=US-politics telegraph.405422=OTHER-world-ID reuters.218933=US-politics newsweek.51328=US-politics '
-             'newsweek.51307=US-business bbc.381692=GB-world independent.226346=GB-entertainment bbc.381646=GB-sport reuters.218914=US-sport scotsman.133758=EU-sport '
-             'rt.com.91350=EU-world scotsman.133773=GB-scitech rt.com.91334=EU-crime bbc.381680=GB-politics guardian.221756=US-politics scotsman.133783=GB-politics cnn.304521=US-sport '
-             'dailymail.co.uk.298622=GB-politics bbc.381789=GB-sport dailymail.co.uk.298644=GB-business dailymail.co.uk.298602=GB-world scotsman.133753=GB-sport '
-             'independent.226317=GB-entertainment nytimes.184862=US-politics thelocal.65969=OTHER-world-SY nytimes.184825=US-politics cnbc.com.6784=US-politics nytimes.184804=US-politics '
-             'nytimes.184830=US-politics scotsman.133801=GB-sport cnbc.com.6770=US-business bbc.381760=GB-crime reuters.218865=OTHER-world-ID newsweek.51339=US-crime '
-             'euronews-en.153797=OTHER-world-ID abcnews.306774=US-crime dailymail.co.uk.298696=GB-politics abcnews.306755=US-politics reuters.218909=US-crime '
-             'independent.226349=OTHER-sport-RU newsweek.51330=US-politics bbc.381705=GB-sport newsweek.51340=OTHER-world-ID cbsnews.248411=OTHER-world-FM abcnews.306776=US-crime '
-             'bbc.381694=GB-entertainment rt.com.91356=US-world telegraph.405430=GB-entertainment telegraph.405404=EU-world bbc.381749=GB-world telegraph.405413=US-politics '
-             'bbc.381736=OTHER-politics-KP cbsnews.248394=US-politics nytimes.184822=US-world telegraph.405408=US-politics euronews-en.153799=OTHER-politics-SY '
-             'euronews-en.153826=EU-sport cnn.304400=US-world',
-}
-
-SUBSETS = {k: {d.split('=')[0]: d.split('=')[1] for d in v.split()} for (k, v) in _SUBSETS.items()}
-COUNTRIES = sorted(list({v.split('-')[0] for v in SUBSETS['wmt19'].values()}))
-DOMAINS = sorted(list({v.split('-')[1] for v in SUBSETS['wmt19'].values()}))
-
-
-if __name__ == '__main__':
-    import sys
-    try:
-        cmd = sys.argv[1]
-    except IndexError:
-        print(f'Usage: {sys.argv[0]} --check | --dump')
-        sys.exit(1)
-
-    if cmd == '--check':
-        import urllib.request
-        import hashlib
-        url_md5 = {}
-
-        for key, value in DATASETS.items():
-            md5_hashes = value.get('md5', None)
-            if md5_hashes is not None:
-                assert len(value['data']) == len(md5_hashes)
-                pairs = zip(value['data'], md5_hashes)
-                for url, md5_hash in pairs:
-                    url_md5[url] = md5_hash
-
-        for url, md5_hash in url_md5.items():
-            try:
-                print('Downloading ', url)
-                with urllib.request.urlopen(url) as f:
-                    data = f.read()
-            except Exception as exc:
-                raise(exc)
-
-            if hashlib.md5(data).hexdigest() != md5_hash:
-                print('MD5 check failed for', url)
-    elif cmd == '--dump':
-        import re
-        # Dumps a table in markdown format
-        print(f'| {"Dataset":<30} | {"Description":<115} |')
-        header = '| ' + '-' * 30 + ' | ' + '-' * 115 + ' |'
-        print(header)
-        for name, dset in DATASETS.items():
-            desc = re.sub(r'(http[s]?:\/\/\S+)', r'[URL](\1)', str(dset['description']))
-            print(f'| {name:<30} | {desc:<115} |')
diff --git a/sacrebleu/dataset/__init__.py b/sacrebleu/dataset/__init__.py
new file mode 100644
index 00000000..34bbb441
--- /dev/null
+++ b/sacrebleu/dataset/__init__.py
@@ -0,0 +1,1855 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright 2017--2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You may not
+# use this file except in compliance with the License. A copy of the License
+# is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is distributed on
+# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+
+# Detailed document metadata annotation in form DocumentID -> CountryCode - Domain - OptionalFinegrainedCountryCode
+# While the annotation is subjective with many unclear cases, it may provide useful insights
+# when applied on large data (TODO: annotate all documents from recent WMT years, at least for origlang=en, consider renaming "world" to "other").
+_SUBSETS = {
+    "wmt18": "rt.com.68098=US-crime guardian.181611=US-politics bbc.310963=GB-sport washpost.116881=US-politics scotsman.104228=GB-sport timemagazine.75207=OTHER-world-ID "
+    "euronews-en.117981=OTHER-crime-AE smh.com.au.242810=US-crime msnbc.53726=US-politics euronews-en.117983=US-politics msnbc.53894=US-crime theglobeandmail.com.62700=US-business "
+    "bbc.310870=OTHER-world-AF reuters.196698=US-politics latimes.231739=US-sport thelocal.51929=OTHER-world-SE cbsnews.198694=US-politics reuters.196718=OTHER-sport-RU "
+    "abcnews.255599=EU-sport nytimes.127256=US-entertainment scotsman.104225=GB-politics dailymail.co.uk.233026=GB-scitech independent.181088=GB-entertainment "
+    "brisbanetimes.com.au.181614=OTHER-business-AU washpost.116837=US-politics dailymail.co.uk.232928=GB-world thelocal.51916=OTHER-politics-IT bbc.310871=US-crime "
+    "nytimes.127392=EU-business-DE euronews-en.118001=EU-scitech-FR washpost.116866=OTHER-crime-MX dailymail.co.uk.233025=OTHER-scitech-CA latimes.231829=US-crime "
+    "guardian.181662=US-entertainment msnbc.53731=US-crime rt.com.68127=OTHER-sport-RU latimes.231782=US-business latimes.231840=US-sport reuters.196711=OTHER-scitech "
+    "guardian.181666=GB-entertainment novinite.com.24019=US-politics smh.com.au.242750=OTHER-scitech guardian.181610=US-politics telegraph.364393=OTHER-crime-ZA "
+    "novinite.com.23995=EU-world dailymail.co.uk.233028=GB-scitech independent.181071=GB-sport telegraph.364538=GB-scitech timemagazine.75193=US-politics "
+    "independent.181096=US-entertainment upi.140602=OTHER-world-AF bbc.310946=GB-business independent.181052=EU-sport ",
+    "wmt19": "bbc.381790=GB-politics rt.com.91337=OTHER-politics-MK nytimes.184853=US-world upi.176266=US-crime guardian.221754=GB-business dailymail.co.uk.298595=GB-business "
+    "cnbc.com.6790=US-politics nytimes.184837=OTHER-world-ID upi.176249=GB-sport euronews-en.153835=OTHER-world-ID dailymail.co.uk.298732=GB-crime telegraph.405401=GB-politics "
+    "newsweek.51331=OTHER-crime-CN abcnews.306815=US-world cbsnews.248384=US-politics reuters.218882=GB-politics cbsnews.248387=US-crime abcnews.306764=OTHER-world-MX "
+    "reuters.218888=EU-politics bbc.381780=GB-crime bbc.381746=GB-sport euronews-en.153800=EU-politics bbc.381679=GB-crime bbc.381735=GB-crime newsweek.51338=US-world "
+    "bbc.381765=GB-crime cnn.304489=US-politics reuters.218863=OTHER-world-ID nytimes.184860=OTHER-world-ID cnn.304404=US-crime bbc.381647=US-entertainment "
+    "abcnews.306758=OTHER-politics-MX cnbc.com.6772=US-business reuters.218932=OTHER-politics-MK upi.176251=GB-sport reuters.218921=US-sport cnn.304447=US-politics "
+    "guardian.221679=GB-politics scotsman.133765=GB-sport scotsman.133804=GB-entertainment guardian.221762=OTHER-politics-BO cnbc.com.6769=US-politics "
+    "dailymail.co.uk.298692=EU-entertainment scotsman.133744=GB-world reuters.218911=US-sport newsweek.51310=US-politics independent.226301=US-sport reuters.218923=EU-sport "
+    "reuters.218861=US-politics dailymail.co.uk.298759=US-world scotsman.133791=GB-sport cbsnews.248484=EU-scitech dailymail.co.uk.298630=US-scitech "
+    "newsweek.51329=US-entertainment bbc.381701=GB-crime dailymail.co.uk.298738=GB-entertainment bbc.381669=OTHER-world-CN foxnews.94512=US-politics "
+    "guardian.221718=GB-entertainment dailymail.co.uk.298686=GB-politics cbsnews.248471=US-politics newsweek.51318=US-entertainment rt.com.91335=US-politics "
+    "newsweek.51300=US-politics cnn.304478=US-politics upi.176275=US-politics telegraph.405422=OTHER-world-ID reuters.218933=US-politics newsweek.51328=US-politics "
+    "newsweek.51307=US-business bbc.381692=GB-world independent.226346=GB-entertainment bbc.381646=GB-sport reuters.218914=US-sport scotsman.133758=EU-sport "
+    "rt.com.91350=EU-world scotsman.133773=GB-scitech rt.com.91334=EU-crime bbc.381680=GB-politics guardian.221756=US-politics scotsman.133783=GB-politics cnn.304521=US-sport "
+    "dailymail.co.uk.298622=GB-politics bbc.381789=GB-sport dailymail.co.uk.298644=GB-business dailymail.co.uk.298602=GB-world scotsman.133753=GB-sport "
+    "independent.226317=GB-entertainment nytimes.184862=US-politics thelocal.65969=OTHER-world-SY nytimes.184825=US-politics cnbc.com.6784=US-politics nytimes.184804=US-politics "
+    "nytimes.184830=US-politics scotsman.133801=GB-sport cnbc.com.6770=US-business bbc.381760=GB-crime reuters.218865=OTHER-world-ID newsweek.51339=US-crime "
+    "euronews-en.153797=OTHER-world-ID abcnews.306774=US-crime dailymail.co.uk.298696=GB-politics abcnews.306755=US-politics reuters.218909=US-crime "
+    "independent.226349=OTHER-sport-RU newsweek.51330=US-politics bbc.381705=GB-sport newsweek.51340=OTHER-world-ID cbsnews.248411=OTHER-world-FM abcnews.306776=US-crime "
+    "bbc.381694=GB-entertainment rt.com.91356=US-world telegraph.405430=GB-entertainment telegraph.405404=EU-world bbc.381749=GB-world telegraph.405413=US-politics "
+    "bbc.381736=OTHER-politics-KP cbsnews.248394=US-politics nytimes.184822=US-world telegraph.405408=US-politics euronews-en.153799=OTHER-politics-SY "
+    "euronews-en.153826=EU-sport cnn.304400=US-world",
+}
+
+SUBSETS = {
+    k: {d.split("=")[0]: d.split("=")[1] for d in v.split()}
+    for (k, v) in _SUBSETS.items()
+}
+COUNTRIES = sorted(list({v.split("-")[0] for v in SUBSETS["wmt19"].values()}))
+DOMAINS = sorted(list({v.split("-")[1] for v in SUBSETS["wmt19"].values()}))
+
+
+# This defines data locations.
+# At the top level are test sets.
+# Beneath each test set, we define the location to download the test data.
+# The other keys are each language pair contained in the tarball, and the respective locations of the source and reference data within each.
+# Many of these are *.sgm files, which are processed to produced plain text that can be used by this script.
+# The canonical location of unpacked, processed data is $SACREBLEU_DIR/$TEST/$SOURCE-$TARGET.{$SOURCE,$TARGET}
+from .fake_sgml import FakeSGMLDataset, WMTAdditionDataset
+from .iwslt_xml import IWSLTXMLDataset
+from .plain_text import PlainTextDataset
+from .tsv import TSVDataset
+from .wmt_xml import WMTXMLDataset
+
+DATASETS = {
+    # wmt
+    "wmt21": WMTXMLDataset(
+        "wmt21",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21.",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "de-fr": ["test/newstest2021.de-fr.xml"],
+            "en-de": ["test/newstest2021.en-de.xml"],
+            "en-ha": ["test/newstest2021.en-ha.xml"],
+            "en-is": ["test/newstest2021.en-is.xml"],
+            "en-ja": ["test/newstest2021.en-ja.xml"],
+            "fr-de": ["test/newstest2021.fr-de.xml"],
+            "ha-en": ["test/newstest2021.ha-en.xml"],
+            "is-en": ["test/newstest2021.is-en.xml"],
+            "ja-en": ["test/newstest2021.ja-en.xml"],
+            "zh-en": ["test/newstest2021.zh-en.xml"],
+            "en-zh": ["test/newstest2021.en-zh.xml"],
+            "cs-en": ["test/newstest2021.cs-en.xml"],
+            "de-en": ["test/newstest2021.de-en.xml"],
+            "en-cs": ["test/newstest2021.en-cs.xml"],
+            "en-ru": ["test/newstest2021.en-ru.xml"],
+            "ru-en": ["test/newstest2021.ru-en.xml"],
+            "bn-hi": ["test/florestest2021.bn-hi.xml"],
+            "hi-bn": ["test/florestest2021.hi-bn.xml"],
+            "xh-zu": ["test/florestest2021.xh-zu.xml"],
+            "zu-xh": ["test/florestest2021.zu-xh.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["A"],
+    ),
+    "wmt21/B": WMTXMLDataset(
+        "wmt21/B",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21 with reference B.",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "cs-en": ["test/newstest2021.cs-en.xml"],
+            "de-en": ["test/newstest2021.de-en.xml"],
+            "en-cs": ["test/newstest2021.en-cs.xml"],
+            "en-ru": ["test/newstest2021.en-ru.xml"],
+            "en-zh": ["test/newstest2021.en-zh.xml"],
+            "ru-en": ["test/newstest2021.ru-en.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["B"],
+    ),
+    "wmt21/AB": WMTXMLDataset(
+        "wmt21/AB",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21 with references A and B.",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "cs-en": ["test/newstest2021.cs-en.xml"],
+            "de-en": ["test/newstest2021.de-en.xml"],
+            "en-de": ["test/newstest2021.en-de.xml"],
+            "en-cs": ["test/newstest2021.en-cs.xml"],
+            "en-ru": ["test/newstest2021.en-ru.xml"],
+            "en-zh": ["test/newstest2021.en-zh.xml"],
+            "ru-en": ["test/newstest2021.ru-en.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["A", "B"],
+    ),
+    "wmt21/C": WMTXMLDataset(
+        "wmt21/C",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21 with reference C",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "en-de": ["test/newstest2021.en-de.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["C"],
+    ),
+    "wmt21/AC": WMTXMLDataset(
+        "wmt21/AC",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21 with references A and C",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "en-de": ["test/newstest2021.en-de.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["A", "C"],
+    ),
+    "wmt21/D": WMTXMLDataset(
+        "wmt21/D",
+        data=["http://data.statmt.org/wmt21/translation-task/test.tgz"],
+        description="Official evaluation data for WMT21 with reference D",
+        md5=["32e7ab995bc318414375d60f0269af92"],
+        langpairs={
+            "en-de": ["test/newstest2021.en-de.xml"],
+        },
+        # the reference to use with this dataset
+        refs=["D"],
+    ),
+    "wmt21/dev": WMTXMLDataset(
+        "wmt21/dev",
+        data=["http://data.statmt.org/wmt21/translation-task/dev.tgz"],
+        description="Development data for WMT21，if multiple references are available, the first one is used.",
+        md5=["165da59ac8dfb5b7cafd7e90b1cac672"],
+        langpairs={
+            "en-ha": ["dev/xml/newsdev2021.en-ha.xml"],
+            "ha-en": ["dev/xml/newsdev2021.ha-en.xml"],
+            "en-is": ["dev/xml/newsdev2021.en-is.xml"],
+            "is-en": ["dev/xml/newsdev2021.is-en.xml"],
+        },
+    ),
+    "wmt20/tworefs": FakeSGMLDataset(
+        "wmt20/tworefs",
+        data=["http://data.statmt.org/wmt20/translation-task/test.tgz"],
+        description="WMT20 news test sets with two references",
+        md5=["3b1f777cfd2fb15ccf66e9bfdb2b1699"],
+        langpairs={
+            "de-en": [
+                "sgm/newstest2020-deen-src.de.sgm",
+                "sgm/newstest2020-deen-ref.en.sgm",
+                "sgm/newstestB2020-deen-ref.en.sgm",
+            ],
+            "en-de": [
+                "sgm/newstest2020-ende-src.en.sgm",
+                "sgm/newstest2020-ende-ref.de.sgm",
+                "sgm/newstestB2020-ende-ref.de.sgm",
+            ],
+            "en-zh": [
+                "sgm/newstest2020-enzh-src.en.sgm",
+                "sgm/newstest2020-enzh-ref.zh.sgm",
+                "sgm/newstestB2020-enzh-ref.zh.sgm",
+            ],
+            "ru-en": [
+                "sgm/newstest2020-ruen-src.ru.sgm",
+                "sgm/newstest2020-ruen-ref.en.sgm",
+                "sgm/newstestB2020-ruen-ref.en.sgm",
+            ],
+            "zh-en": [
+                "sgm/newstest2020-zhen-src.zh.sgm",
+                "sgm/newstest2020-zhen-ref.en.sgm",
+                "sgm/newstestB2020-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt20": FakeSGMLDataset(
+        "wmt20",
+        data=["http://data.statmt.org/wmt20/translation-task/test.tgz"],
+        description="Official evaluation data for WMT20",
+        md5=["3b1f777cfd2fb15ccf66e9bfdb2b1699"],
+        langpairs={
+            "cs-en": [
+                "sgm/newstest2020-csen-src.cs.sgm",
+                "sgm/newstest2020-csen-ref.en.sgm",
+            ],
+            "de-en": [
+                "sgm/newstest2020-deen-src.de.sgm",
+                "sgm/newstest2020-deen-ref.en.sgm",
+            ],
+            "de-fr": [
+                "sgm/newstest2020-defr-src.de.sgm",
+                "sgm/newstest2020-defr-ref.fr.sgm",
+            ],
+            "en-cs": [
+                "sgm/newstest2020-encs-src.en.sgm",
+                "sgm/newstest2020-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "sgm/newstest2020-ende-src.en.sgm",
+                "sgm/newstest2020-ende-ref.de.sgm",
+            ],
+            "en-iu": [
+                "sgm/newstest2020-eniu-src.en.sgm",
+                "sgm/newstest2020-eniu-ref.iu.sgm",
+            ],
+            "en-ja": [
+                "sgm/newstest2020-enja-src.en.sgm",
+                "sgm/newstest2020-enja-ref.ja.sgm",
+            ],
+            "en-km": [
+                "sgm/newstest2020-enkm-src.en.sgm",
+                "sgm/newstest2020-enkm-ref.km.sgm",
+            ],
+            "en-pl": [
+                "sgm/newstest2020-enpl-src.en.sgm",
+                "sgm/newstest2020-enpl-ref.pl.sgm",
+            ],
+            "en-ps": [
+                "sgm/newstest2020-enps-src.en.sgm",
+                "sgm/newstest2020-enps-ref.ps.sgm",
+            ],
+            "en-ru": [
+                "sgm/newstest2020-enru-src.en.sgm",
+                "sgm/newstest2020-enru-ref.ru.sgm",
+            ],
+            "en-ta": [
+                "sgm/newstest2020-enta-src.en.sgm",
+                "sgm/newstest2020-enta-ref.ta.sgm",
+            ],
+            "en-zh": [
+                "sgm/newstest2020-enzh-src.en.sgm",
+                "sgm/newstest2020-enzh-ref.zh.sgm",
+            ],
+            "fr-de": [
+                "sgm/newstest2020-frde-src.fr.sgm",
+                "sgm/newstest2020-frde-ref.de.sgm",
+            ],
+            "iu-en": [
+                "sgm/newstest2020-iuen-src.iu.sgm",
+                "sgm/newstest2020-iuen-ref.en.sgm",
+            ],
+            "ja-en": [
+                "sgm/newstest2020-jaen-src.ja.sgm",
+                "sgm/newstest2020-jaen-ref.en.sgm",
+            ],
+            "km-en": [
+                "sgm/newstest2020-kmen-src.km.sgm",
+                "sgm/newstest2020-kmen-ref.en.sgm",
+            ],
+            "pl-en": [
+                "sgm/newstest2020-plen-src.pl.sgm",
+                "sgm/newstest2020-plen-ref.en.sgm",
+            ],
+            "ps-en": [
+                "sgm/newstest2020-psen-src.ps.sgm",
+                "sgm/newstest2020-psen-ref.en.sgm",
+            ],
+            "ru-en": [
+                "sgm/newstest2020-ruen-src.ru.sgm",
+                "sgm/newstest2020-ruen-ref.en.sgm",
+            ],
+            "ta-en": [
+                "sgm/newstest2020-taen-src.ta.sgm",
+                "sgm/newstest2020-taen-ref.en.sgm",
+            ],
+            "zh-en": [
+                "sgm/newstest2020-zhen-src.zh.sgm",
+                "sgm/newstest2020-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt20/dev": FakeSGMLDataset(
+        "wmt20/dev",
+        data=["http://data.statmt.org/wmt20/translation-task/dev.tgz"],
+        description="Development data for tasks new to 2020.",
+        md5=["037f2b37aab74febbb1b2307dc2afb54"],
+        langpairs={
+            "iu-en": [
+                "dev/newsdev2020-iuen-src.iu.sgm",
+                "dev/newsdev2020-iuen-ref.en.sgm",
+            ],
+            "en-iu": [
+                "dev/newsdev2020-eniu-src.en.sgm",
+                "dev/newsdev2020-eniu-ref.iu.sgm",
+            ],
+            "ja-en": [
+                "dev/newsdev2020-jaen-src.ja.sgm",
+                "dev/newsdev2020-jaen-ref.en.sgm",
+            ],
+            "en-ja": [
+                "dev/newsdev2020-enja-src.en.sgm",
+                "dev/newsdev2020-enja-ref.ja.sgm",
+            ],
+            "pl-en": [
+                "dev/newsdev2020-plen-src.pl.sgm",
+                "dev/newsdev2020-plen-ref.en.sgm",
+            ],
+            "en-pl": [
+                "dev/newsdev2020-enpl-src.en.sgm",
+                "dev/newsdev2020-enpl-ref.pl.sgm",
+            ],
+            "ta-en": [
+                "dev/newsdev2020-taen-src.ta.sgm",
+                "dev/newsdev2020-taen-ref.en.sgm",
+            ],
+            "en-ta": [
+                "dev/newsdev2020-enta-src.en.sgm",
+                "dev/newsdev2020-enta-ref.ta.sgm",
+            ],
+        },
+    ),
+    "wmt20/robust/set1": PlainTextDataset(
+        "wmt20/robust/set1",
+        data=["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
+        md5=["a12ac9ebe89b72195041518dffc4a9d5"],
+        description="WMT20 robustness task, set 1",
+        langpairs={
+            "en-ja": [
+                "robustness20-3-sets/robustness20-set1-enja.en",
+                "robustness20-3-sets/robustness20-set1-enja.ja",
+            ],
+            "en-de": [
+                "robustness20-3-sets/robustness20-set1-ende.en",
+                "robustness20-3-sets/robustness20-set1-ende.de",
+            ],
+        },
+    ),
+    "wmt20/robust/set2": PlainTextDataset(
+        "wmt20/robust/set2",
+        data=["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
+        md5=["a12ac9ebe89b72195041518dffc4a9d5"],
+        description="WMT20 robustness task, set 2",
+        langpairs={
+            "en-ja": [
+                "robustness20-3-sets/robustness20-set2-enja.en",
+                "robustness20-3-sets/robustness20-set2-enja.ja",
+            ],
+            "ja-en": [
+                "robustness20-3-sets/robustness20-set2-jaen.ja",
+                "robustness20-3-sets/robustness20-set2-jaen.en",
+            ],
+        },
+    ),
+    "wmt20/robust/set3": PlainTextDataset(
+        "wmt20/robust/set3",
+        data=["http://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip"],
+        md5=["a12ac9ebe89b72195041518dffc4a9d5"],
+        description="WMT20 robustness task, set 3",
+        langpairs={
+            "de-en": [
+                "robustness20-3-sets/robustness20-set3-deen.de",
+                "robustness20-3-sets/robustness20-set3-deen.en",
+            ],
+        },
+    ),
+    "wmt19": FakeSGMLDataset(
+        "wmt19",
+        data=["http://data.statmt.org/wmt19/translation-task/test.tgz"],
+        description="Official evaluation data.",
+        md5=["84de7162d158e28403103b01aeefc39a"],
+        citation=r"""@proceedings{ws-2019-machine,
+    title = "Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)",
+    editor = "Bojar, Ond{\v{r}}ej  and
+      Chatterjee, Rajen  and
+      Federmann, Christian  and
+      Fishel, Mark  and
+      Graham, Yvette  and
+      Haddow, Barry  and
+      Huck, Matthias  and
+      Yepes, Antonio Jimeno  and
+      Koehn, Philipp  and
+      Martins, Andr{\'e}  and
+      Monz, Christof  and
+      Negri, Matteo  and
+      N{\'e}v{\'e}ol, Aur{\'e}lie  and
+      Neves, Mariana  and
+      Post, Matt  and
+      Turchi, Marco  and
+      Verspoor, Karin",
+    month = aug,
+    year = "2019",
+    address = "Florence, Italy",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/W19-5200",
+}""",
+        langpairs={
+            "cs-de": [
+                "sgm/newstest2019-csde-src.cs.sgm",
+                "sgm/newstest2019-csde-ref.de.sgm",
+            ],
+            "de-cs": [
+                "sgm/newstest2019-decs-src.de.sgm",
+                "sgm/newstest2019-decs-ref.cs.sgm",
+            ],
+            "de-en": [
+                "sgm/newstest2019-deen-src.de.sgm",
+                "sgm/newstest2019-deen-ref.en.sgm",
+            ],
+            "de-fr": [
+                "sgm/newstest2019-defr-src.de.sgm",
+                "sgm/newstest2019-defr-ref.fr.sgm",
+            ],
+            "en-cs": [
+                "sgm/newstest2019-encs-src.en.sgm",
+                "sgm/newstest2019-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "sgm/newstest2019-ende-src.en.sgm",
+                "sgm/newstest2019-ende-ref.de.sgm",
+            ],
+            "en-fi": [
+                "sgm/newstest2019-enfi-src.en.sgm",
+                "sgm/newstest2019-enfi-ref.fi.sgm",
+            ],
+            "en-gu": [
+                "sgm/newstest2019-engu-src.en.sgm",
+                "sgm/newstest2019-engu-ref.gu.sgm",
+            ],
+            "en-kk": [
+                "sgm/newstest2019-enkk-src.en.sgm",
+                "sgm/newstest2019-enkk-ref.kk.sgm",
+            ],
+            "en-lt": [
+                "sgm/newstest2019-enlt-src.en.sgm",
+                "sgm/newstest2019-enlt-ref.lt.sgm",
+            ],
+            "en-ru": [
+                "sgm/newstest2019-enru-src.en.sgm",
+                "sgm/newstest2019-enru-ref.ru.sgm",
+            ],
+            "en-zh": [
+                "sgm/newstest2019-enzh-src.en.sgm",
+                "sgm/newstest2019-enzh-ref.zh.sgm",
+            ],
+            "fi-en": [
+                "sgm/newstest2019-fien-src.fi.sgm",
+                "sgm/newstest2019-fien-ref.en.sgm",
+            ],
+            "fr-de": [
+                "sgm/newstest2019-frde-src.fr.sgm",
+                "sgm/newstest2019-frde-ref.de.sgm",
+            ],
+            "gu-en": [
+                "sgm/newstest2019-guen-src.gu.sgm",
+                "sgm/newstest2019-guen-ref.en.sgm",
+            ],
+            "kk-en": [
+                "sgm/newstest2019-kken-src.kk.sgm",
+                "sgm/newstest2019-kken-ref.en.sgm",
+            ],
+            "lt-en": [
+                "sgm/newstest2019-lten-src.lt.sgm",
+                "sgm/newstest2019-lten-ref.en.sgm",
+            ],
+            "ru-en": [
+                "sgm/newstest2019-ruen-src.ru.sgm",
+                "sgm/newstest2019-ruen-ref.en.sgm",
+            ],
+            "zh-en": [
+                "sgm/newstest2019-zhen-src.zh.sgm",
+                "sgm/newstest2019-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt19/dev": FakeSGMLDataset(
+        "wmt19/dev",
+        data=["http://data.statmt.org/wmt19/translation-task/dev.tgz"],
+        description="Development data for tasks new to 2019.",
+        md5=["f2ec7af5947c19e0cacb3882eb208002"],
+        langpairs={
+            "lt-en": [
+                "dev/newsdev2019-lten-src.lt.sgm",
+                "dev/newsdev2019-lten-ref.en.sgm",
+            ],
+            "en-lt": [
+                "dev/newsdev2019-enlt-src.en.sgm",
+                "dev/newsdev2019-enlt-ref.lt.sgm",
+            ],
+            "gu-en": [
+                "dev/newsdev2019-guen-src.gu.sgm",
+                "dev/newsdev2019-guen-ref.en.sgm",
+            ],
+            "en-gu": [
+                "dev/newsdev2019-engu-src.en.sgm",
+                "dev/newsdev2019-engu-ref.gu.sgm",
+            ],
+            "kk-en": [
+                "dev/newsdev2019-kken-src.kk.sgm",
+                "dev/newsdev2019-kken-ref.en.sgm",
+            ],
+            "en-kk": [
+                "dev/newsdev2019-enkk-src.en.sgm",
+                "dev/newsdev2019-enkk-ref.kk.sgm",
+            ],
+        },
+    ),
+    "wmt19/google/ar": WMTAdditionDataset(
+        "wmt19/google/ar",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-ar.ref",
+        ],
+        description="Additional high-quality reference for WMT19/en-de.",
+        md5=["84de7162d158e28403103b01aeefc39a", "d66d9e91548ced0ac476f2390e32e2de"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_ar.wmt19-ende-ar.ref"],
+        },
+    ),
+    "wmt19/google/arp": WMTAdditionDataset(
+        "wmt19/google/arp",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-arp.ref",
+        ],
+        description="Additional paraphrase of wmt19/google/ar.",
+        md5=["84de7162d158e28403103b01aeefc39a", "c70ea808cf2bff621ad7a8fddd4deca9"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_arp.wmt19-ende-arp.ref"],
+        },
+    ),
+    "wmt19/google/wmtp": WMTAdditionDataset(
+        "wmt19/google/wmtp",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-wmtp.ref",
+        ],
+        description="Additional paraphrase of the official WMT19 reference.",
+        md5=["84de7162d158e28403103b01aeefc39a", "587c660ee5fd44727f0db025b71c6a82"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_wmtp.wmt19-ende-wmtp.ref"],
+        },
+    ),
+    "wmt19/google/hqr": WMTAdditionDataset(
+        "wmt19/google/hqr",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqr.ref",
+        ],
+        description="Best human selected-reference between wmt19 and wmt19/google/ar.",
+        md5=["84de7162d158e28403103b01aeefc39a", "d9221135f62d7152de041f5bfc8efaea"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_hqr.wmt19-ende-hqr.ref"],
+        },
+    ),
+    "wmt19/google/hqp": WMTAdditionDataset(
+        "wmt19/google/hqp",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqp.ref",
+        ],
+        description="Best human-selected reference between wmt19/google/arp and wmt19/google/wmtp.",
+        md5=["84de7162d158e28403103b01aeefc39a", "b7c3a07a59c8eccea5367e9ec5417a8a"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_hqp.wmt19-ende-hqp.ref"],
+        },
+    ),
+    "wmt19/google/hqall": WMTAdditionDataset(
+        "wmt19/google/hqall",
+        data=[
+            "http://data.statmt.org/wmt19/translation-task/test.tgz",
+            "https://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqall.ref",
+        ],
+        description="Best human-selected reference among original official reference and the Google reference and paraphrases.",
+        md5=["84de7162d158e28403103b01aeefc39a", "edecf10ced59e10b703a6fbcf1fa9dfa"],
+        citation="@misc{freitag2020bleu,\n    title={{BLEU} might be Guilty but References are not Innocent},\n    author={Markus Freitag and David Grangier and Isaac Caswell},\n    year={2020},\n    eprint={2004.06063},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}",
+        langpairs={
+            "en-de": ["sgm/newstest2019-ende-src.en.sgm", "wmt19_google_hqall.wmt19-ende-hqall.ref"],
+        },
+    ),
+    "wmt18": FakeSGMLDataset(
+        "wmt18",
+        data=["http://data.statmt.org/wmt18/translation-task/test.tgz"],
+        md5=["f996c245ecffea23d0006fa4c34e9064"],
+        description="Official evaluation data.",
+        citation='@inproceedings{bojar-etal-2018-findings,\n    title = "Findings of the 2018 Conference on Machine Translation ({WMT}18)",\n    author = "Bojar, Ond{\v{r}}ej  and\n      Federmann, Christian  and\n      Fishel, Mark  and\n      Graham, Yvette  and\n      Haddow, Barry  and\n      Koehn, Philipp  and\n      Monz, Christof",\n    booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers",\n    month = oct,\n    year = "2018",\n    address = "Belgium, Brussels",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W18-6401",\n    pages = "272--303",\n}',
+        langpairs={
+            "cs-en": [
+                "test/newstest2018-csen-src.cs.sgm",
+                "test/newstest2018-csen-ref.en.sgm",
+            ],
+            "de-en": [
+                "test/newstest2018-deen-src.de.sgm",
+                "test/newstest2018-deen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test/newstest2018-encs-src.en.sgm",
+                "test/newstest2018-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "test/newstest2018-ende-src.en.sgm",
+                "test/newstest2018-ende-ref.de.sgm",
+            ],
+            "en-et": [
+                "test/newstest2018-enet-src.en.sgm",
+                "test/newstest2018-enet-ref.et.sgm",
+            ],
+            "en-fi": [
+                "test/newstest2018-enfi-src.en.sgm",
+                "test/newstest2018-enfi-ref.fi.sgm",
+            ],
+            "en-ru": [
+                "test/newstest2018-enru-src.en.sgm",
+                "test/newstest2018-enru-ref.ru.sgm",
+            ],
+            "et-en": [
+                "test/newstest2018-eten-src.et.sgm",
+                "test/newstest2018-eten-ref.en.sgm",
+            ],
+            "fi-en": [
+                "test/newstest2018-fien-src.fi.sgm",
+                "test/newstest2018-fien-ref.en.sgm",
+            ],
+            "ru-en": [
+                "test/newstest2018-ruen-src.ru.sgm",
+                "test/newstest2018-ruen-ref.en.sgm",
+            ],
+            "en-tr": [
+                "test/newstest2018-entr-src.en.sgm",
+                "test/newstest2018-entr-ref.tr.sgm",
+            ],
+            "tr-en": [
+                "test/newstest2018-tren-src.tr.sgm",
+                "test/newstest2018-tren-ref.en.sgm",
+            ],
+            "en-zh": [
+                "test/newstest2018-enzh-src.en.sgm",
+                "test/newstest2018-enzh-ref.zh.sgm",
+            ],
+            "zh-en": [
+                "test/newstest2018-zhen-src.zh.sgm",
+                "test/newstest2018-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt18/test-ts": FakeSGMLDataset(
+        "wmt18/test-ts",
+        data=["http://data.statmt.org/wmt18/translation-task/test-ts.tgz"],
+        md5=["5c621a34d512cc2dd74162ae7d00b320"],
+        description="Official evaluation sources with extra test sets interleaved.",
+        langpairs={
+            "cs-en": ["test-ts/newstest2018-csen-src-ts.cs.sgm", "test-ts/newstest2018-csen-ref-ts.en.sgm"],
+            "de-en": ["test-ts/newstest2018-deen-src-ts.de.sgm", "test-ts/newstest2018-deen-ref-ts.en.sgm"],
+            "en-cs": ["test-ts/newstest2018-encs-src-ts.en.sgm", "test-ts/newstest2018-encs-ref-ts.cs.sgm"],
+            "en-de": ["test-ts/newstest2018-ende-src-ts.en.sgm", "test-ts/newstest2018-ende-ref-ts.de.sgm"],
+            "en-et": ["test-ts/newstest2018-enet-src-ts.en.sgm", "test-ts/newstest2018-enet-ref-ts.et.sgm"],
+            "en-fi": ["test-ts/newstest2018-enfi-src-ts.en.sgm", "test-ts/newstest2018-enfi-ref-ts.fi.sgm"],
+            "en-ru": ["test-ts/newstest2018-enru-src-ts.en.sgm", "test-ts/newstest2018-enru-ref-ts.ru.sgm"],
+            "et-en": ["test-ts/newstest2018-eten-src-ts.et.sgm", "test-ts/newstest2018-eten-ref-ts.en.sgm"],
+            "fi-en": ["test-ts/newstest2018-fien-src-ts.fi.sgm", "test-ts/newstest2018-fien-ref-ts.en.sgm"],
+            "ru-en": ["test-ts/newstest2018-ruen-src-ts.ru.sgm", "test-ts/newstest2018-ruen-ref-ts.en.sgm"],
+            "en-tr": ["test-ts/newstest2018-entr-src-ts.en.sgm", "test-ts/newstest2018-entr-ref-ts.tr.sgm"],
+            "tr-en": ["test-ts/newstest2018-tren-src-ts.tr.sgm", "test-ts/newstest2018-tren-ref-ts.en.sgm"],
+            "en-zh": ["test-ts/newstest2018-enzh-src-ts.en.sgm", "test-ts/newstest2018-enzh-ref-ts.zh.sgm"],
+            "zh-en": ["test-ts/newstest2018-zhen-src-ts.zh.sgm", "test-ts/newstest2018-zhen-ref-ts.en.sgm"],
+        },
+    ),
+    "wmt18/dev": FakeSGMLDataset(
+        "wmt18/dev",
+        data=["http://data.statmt.org/wmt18/translation-task/dev.tgz"],
+        md5=["486f391da54a7a3247f02ebd25996f24"],
+        description="Development data (Estonian<>English).",
+        langpairs={
+            "et-en": [
+                "dev/newsdev2018-eten-src.et.sgm",
+                "dev/newsdev2018-eten-ref.en.sgm",
+            ],
+            "en-et": [
+                "dev/newsdev2018-enet-src.en.sgm",
+                "dev/newsdev2018-enet-ref.et.sgm",
+            ],
+        },
+    ),
+    "wmt17": FakeSGMLDataset(
+        "wmt17",
+        data=["http://data.statmt.org/wmt17/translation-task/test.tgz"],
+        md5=["86a1724c276004aa25455ae2a04cef26"],
+        description="Official evaluation data.",
+        citation="@InProceedings{bojar-EtAl:2017:WMT1,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Graham, Yvette  and  Haddow, Barry  and  Huang, Shujian  and  Huck, Matthias  and  Koehn, Philipp  and  Liu, Qun  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Post, Matt  and  Rubino, Raphael  and  Specia, Lucia  and  Turchi, Marco},\n  title     = {Findings of the 2017 Conference on Machine Translation (WMT17)},\n  booktitle = {Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers},\n  month     = {September},\n  year      = {2017},\n  address   = {Copenhagen, Denmark},\n  publisher = {Association for Computational Linguistics},\n  pages     = {169--214},\n  url       = {http://www.aclweb.org/anthology/W17-4717}\n}",
+        langpairs={
+            "cs-en": [
+                "test/newstest2017-csen-src.cs.sgm",
+                "test/newstest2017-csen-ref.en.sgm",
+            ],
+            "de-en": [
+                "test/newstest2017-deen-src.de.sgm",
+                "test/newstest2017-deen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test/newstest2017-encs-src.en.sgm",
+                "test/newstest2017-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "test/newstest2017-ende-src.en.sgm",
+                "test/newstest2017-ende-ref.de.sgm",
+            ],
+            "en-fi": [
+                "test/newstest2017-enfi-src.en.sgm",
+                "test/newstest2017-enfi-ref.fi.sgm",
+            ],
+            "en-lv": [
+                "test/newstest2017-enlv-src.en.sgm",
+                "test/newstest2017-enlv-ref.lv.sgm",
+            ],
+            "en-ru": [
+                "test/newstest2017-enru-src.en.sgm",
+                "test/newstest2017-enru-ref.ru.sgm",
+            ],
+            "en-tr": [
+                "test/newstest2017-entr-src.en.sgm",
+                "test/newstest2017-entr-ref.tr.sgm",
+            ],
+            "en-zh": [
+                "test/newstest2017-enzh-src.en.sgm",
+                "test/newstest2017-enzh-ref.zh.sgm",
+            ],
+            "fi-en": [
+                "test/newstest2017-fien-src.fi.sgm",
+                "test/newstest2017-fien-ref.en.sgm",
+            ],
+            "lv-en": [
+                "test/newstest2017-lven-src.lv.sgm",
+                "test/newstest2017-lven-ref.en.sgm",
+            ],
+            "ru-en": [
+                "test/newstest2017-ruen-src.ru.sgm",
+                "test/newstest2017-ruen-ref.en.sgm",
+            ],
+            "tr-en": [
+                "test/newstest2017-tren-src.tr.sgm",
+                "test/newstest2017-tren-ref.en.sgm",
+            ],
+            "zh-en": [
+                "test/newstest2017-zhen-src.zh.sgm",
+                "test/newstest2017-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt17/B": FakeSGMLDataset(
+        "wmt17/B",
+        data=["http://data.statmt.org/wmt17/translation-task/test.tgz"],
+        md5=["86a1724c276004aa25455ae2a04cef26"],
+        description="Additional reference for EN-FI and FI-EN.",
+        langpairs={
+            "en-fi": [
+                "test/newstestB2017-enfi-src.en.sgm",
+                "test/newstestB2017-enfi-ref.fi.sgm",
+            ],
+        },
+    ),
+    "wmt17/tworefs": FakeSGMLDataset(
+        "wmt17/tworefs",
+        data=["http://data.statmt.org/wmt17/translation-task/test.tgz"],
+        md5=["86a1724c276004aa25455ae2a04cef26"],
+        description="Systems with two references.",
+        langpairs={
+            "en-fi": [
+                "test/newstest2017-enfi-src.en.sgm",
+                "test/newstest2017-enfi-ref.fi.sgm",
+                "test/newstestB2017-enfi-ref.fi.sgm",
+            ],
+        },
+    ),
+    "wmt17/improved": FakeSGMLDataset(
+        "wmt17/improved",
+        data=["http://data.statmt.org/wmt17/translation-task/test-update-1.tgz"],
+        md5=["91dbfd5af99bc6891a637a68e04dfd41"],
+        description="Improved zh-en and en-zh translations.",
+        langpairs={
+            "en-zh": ["newstest2017-enzh-src.en.sgm", "newstest2017-enzh-ref.zh.sgm"],
+            "zh-en": ["newstest2017-zhen-src.zh.sgm", "newstest2017-zhen-ref.en.sgm"],
+        },
+    ),
+    "wmt17/dev": FakeSGMLDataset(
+        "wmt17/dev",
+        data=["http://data.statmt.org/wmt17/translation-task/dev.tgz"],
+        md5=["9b1aa63c1cf49dccdd20b962fe313989"],
+        description="Development sets released for new languages in 2017.",
+        langpairs={
+            "en-lv": [
+                "dev/newsdev2017-enlv-src.en.sgm",
+                "dev/newsdev2017-enlv-ref.lv.sgm",
+            ],
+            "en-zh": [
+                "dev/newsdev2017-enzh-src.en.sgm",
+                "dev/newsdev2017-enzh-ref.zh.sgm",
+            ],
+            "lv-en": [
+                "dev/newsdev2017-lven-src.lv.sgm",
+                "dev/newsdev2017-lven-ref.en.sgm",
+            ],
+            "zh-en": [
+                "dev/newsdev2017-zhen-src.zh.sgm",
+                "dev/newsdev2017-zhen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt17/ms": WMTAdditionDataset(
+        "wmt17/ms",
+        data=[
+            "https://github.com/MicrosoftTranslator/Translator-HumanParityData/archive/master.zip",
+            "http://data.statmt.org/wmt17/translation-task/test-update-1.tgz",
+        ],
+        md5=["18fdaa7a3c84cf6ef688da1f6a5fa96f", "91dbfd5af99bc6891a637a68e04dfd41"],
+        description="Additional Chinese-English references from Microsoft Research.",
+        citation="@inproceedings{achieving-human-parity-on-automatic-chinese-to-english-news-translation,\n  author = {Hassan Awadalla, Hany and Aue, Anthony and Chen, Chang and Chowdhary, Vishal and Clark, Jonathan and Federmann, Christian and Huang, Xuedong and Junczys-Dowmunt, Marcin and Lewis, Will and Li, Mu and Liu, Shujie and Liu, Tie-Yan and Luo, Renqian and Menezes, Arul and Qin, Tao and Seide, Frank and Tan, Xu and Tian, Fei and Wu, Lijun and Wu, Shuangzhi and Xia, Yingce and Zhang, Dongdong and Zhang, Zhirui and Zhou, Ming},\n  title = {Achieving Human Parity on Automatic Chinese to English News Translation},\n  booktitle = {},\n  year = {2018},\n  month = {March},\n  abstract = {Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question naturally arises whether such systems can approach or achieve parity with human translations. In this paper, we first address the problem of how to define and accurately measure human parity in translation. We then describe Microsoft’s machine translation system and measure the quality of its translations on the widely used WMT 2017 news translation task from Chinese to English. We find that our latest neural machine translation system has reached a new state-of-the-art, and that the translation quality is at human parity when compared to professional human translations. We also find that it significantly exceeds the quality of crowd-sourced non-professional translations.},\n  publisher = {},\n  url = {https://www.microsoft.com/en-us/research/publication/achieving-human-parity-on-automatic-chinese-to-english-news-translation/},\n  address = {},\n  pages = {},\n  journal = {},\n  volume = {},\n  chapter = {},\n  isbn = {},\n}",
+        langpairs={
+            "zh-en": [
+                "newstest2017-zhen-src.zh.sgm",
+                "newstest2017-zhen-ref.en.sgm",
+                "Translator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-HT.txt",
+                "Translator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-PE.txt",
+            ],
+        },
+    ),
+    "wmt16": FakeSGMLDataset(
+        "wmt16",
+        data=["http://data.statmt.org/wmt16/translation-task/test.tgz"],
+        md5=["3d809cd0c2c86adb2c67034d15c4e446"],
+        description="Official evaluation data.",
+        citation="@InProceedings{bojar-EtAl:2016:WMT1,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Graham, Yvette  and  Haddow, Barry  and  Huck, Matthias  and  Jimeno Yepes, Antonio  and  Koehn, Philipp  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Neveol, Aurelie  and  Neves, Mariana  and  Popel, Martin  and  Post, Matt  and  Rubino, Raphael  and  Scarton, Carolina  and  Specia, Lucia  and  Turchi, Marco  and  Verspoor, Karin  and  Zampieri, Marcos},\n  title     = {Findings of the 2016 Conference on Machine Translation},\n  booktitle = {Proceedings of the First Conference on Machine Translation},\n  month     = {August},\n  year      = {2016},\n  address   = {Berlin, Germany},\n  publisher = {Association for Computational Linguistics},\n  pages     = {131--198},\n  url       = {http://www.aclweb.org/anthology/W/W16/W16-2301}\n}",
+        langpairs={
+            "cs-en": [
+                "test/newstest2016-csen-src.cs.sgm",
+                "test/newstest2016-csen-ref.en.sgm",
+            ],
+            "de-en": [
+                "test/newstest2016-deen-src.de.sgm",
+                "test/newstest2016-deen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test/newstest2016-encs-src.en.sgm",
+                "test/newstest2016-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "test/newstest2016-ende-src.en.sgm",
+                "test/newstest2016-ende-ref.de.sgm",
+            ],
+            "en-fi": [
+                "test/newstest2016-enfi-src.en.sgm",
+                "test/newstest2016-enfi-ref.fi.sgm",
+            ],
+            "en-ro": [
+                "test/newstest2016-enro-src.en.sgm",
+                "test/newstest2016-enro-ref.ro.sgm",
+            ],
+            "en-ru": [
+                "test/newstest2016-enru-src.en.sgm",
+                "test/newstest2016-enru-ref.ru.sgm",
+            ],
+            "en-tr": [
+                "test/newstest2016-entr-src.en.sgm",
+                "test/newstest2016-entr-ref.tr.sgm",
+            ],
+            "fi-en": [
+                "test/newstest2016-fien-src.fi.sgm",
+                "test/newstest2016-fien-ref.en.sgm",
+            ],
+            "ro-en": [
+                "test/newstest2016-roen-src.ro.sgm",
+                "test/newstest2016-roen-ref.en.sgm",
+            ],
+            "ru-en": [
+                "test/newstest2016-ruen-src.ru.sgm",
+                "test/newstest2016-ruen-ref.en.sgm",
+            ],
+            "tr-en": [
+                "test/newstest2016-tren-src.tr.sgm",
+                "test/newstest2016-tren-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt16/B": FakeSGMLDataset(
+        "wmt16/B",
+        data=["http://data.statmt.org/wmt16/translation-task/test.tgz"],
+        md5=["3d809cd0c2c86adb2c67034d15c4e446"],
+        description="Additional reference for EN-FI.",
+        langpairs={
+            "en-fi": [
+                "test/newstest2016-enfi-src.en.sgm",
+                "test/newstestB2016-enfi-ref.fi.sgm",
+            ],
+        },
+    ),
+    "wmt16/tworefs": FakeSGMLDataset(
+        "wmt16/tworefs",
+        data=["http://data.statmt.org/wmt16/translation-task/test.tgz"],
+        md5=["3d809cd0c2c86adb2c67034d15c4e446"],
+        description="EN-FI with two references.",
+        langpairs={
+            "en-fi": [
+                "test/newstest2016-enfi-src.en.sgm",
+                "test/newstest2016-enfi-ref.fi.sgm",
+                "test/newstestB2016-enfi-ref.fi.sgm",
+            ],
+        },
+    ),
+    "wmt16/dev": FakeSGMLDataset(
+        "wmt16/dev",
+        data=["http://data.statmt.org/wmt16/translation-task/dev.tgz"],
+        md5=["4a3dc2760bb077f4308cce96b06e6af6"],
+        description="Development sets released for new languages in 2016.",
+        langpairs={
+            "en-ro": [
+                "dev/newsdev2016-enro-src.en.sgm",
+                "dev/newsdev2016-enro-ref.ro.sgm",
+            ],
+            "en-tr": [
+                "dev/newsdev2016-entr-src.en.sgm",
+                "dev/newsdev2016-entr-ref.tr.sgm",
+            ],
+            "ro-en": [
+                "dev/newsdev2016-roen-src.ro.sgm",
+                "dev/newsdev2016-roen-ref.en.sgm",
+            ],
+            "tr-en": [
+                "dev/newsdev2016-tren-src.tr.sgm",
+                "dev/newsdev2016-tren-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt15": FakeSGMLDataset(
+        "wmt15",
+        data=["http://statmt.org/wmt15/test.tgz"],
+        md5=["67e3beca15e69fe3d36de149da0a96df"],
+        description="Official evaluation data.",
+        citation="@InProceedings{bojar-EtAl:2015:WMT,\n  author    = {Bojar, Ond\\v{r}ej  and  Chatterjee, Rajen  and  Federmann, Christian  and  Haddow, Barry  and  Huck, Matthias  and  Hokamp, Chris  and  Koehn, Philipp  and  Logacheva, Varvara  and  Monz, Christof  and  Negri, Matteo  and  Post, Matt  and  Scarton, Carolina  and  Specia, Lucia  and  Turchi, Marco},\n  title     = {Findings of the 2015 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},\n  month     = {September},\n  year      = {2015},\n  address   = {Lisbon, Portugal},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--46},\n  url       = {http://aclweb.org/anthology/W15-3001}\n}",
+        langpairs={
+            "en-fr": [
+                "test/newsdiscusstest2015-enfr-src.en.sgm",
+                "test/newsdiscusstest2015-enfr-ref.fr.sgm",
+            ],
+            "fr-en": [
+                "test/newsdiscusstest2015-fren-src.fr.sgm",
+                "test/newsdiscusstest2015-fren-ref.en.sgm",
+            ],
+            "cs-en": [
+                "test/newstest2015-csen-src.cs.sgm",
+                "test/newstest2015-csen-ref.en.sgm",
+            ],
+            "de-en": [
+                "test/newstest2015-deen-src.de.sgm",
+                "test/newstest2015-deen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test/newstest2015-encs-src.en.sgm",
+                "test/newstest2015-encs-ref.cs.sgm",
+            ],
+            "en-de": [
+                "test/newstest2015-ende-src.en.sgm",
+                "test/newstest2015-ende-ref.de.sgm",
+            ],
+            "en-fi": [
+                "test/newstest2015-enfi-src.en.sgm",
+                "test/newstest2015-enfi-ref.fi.sgm",
+            ],
+            "en-ru": [
+                "test/newstest2015-enru-src.en.sgm",
+                "test/newstest2015-enru-ref.ru.sgm",
+            ],
+            "fi-en": [
+                "test/newstest2015-fien-src.fi.sgm",
+                "test/newstest2015-fien-ref.en.sgm",
+            ],
+            "ru-en": [
+                "test/newstest2015-ruen-src.ru.sgm",
+                "test/newstest2015-ruen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt14": FakeSGMLDataset(
+        "wmt14",
+        data=["http://statmt.org/wmt14/test-filtered.tgz"],
+        md5=["84c597844c1542e29c2aff23aaee4310"],
+        description="Official evaluation data.",
+        citation="@InProceedings{bojar-EtAl:2014:W14-33,\n  author    = {Bojar, Ondrej  and  Buck, Christian  and  Federmann, Christian  and  Haddow, Barry  and  Koehn, Philipp  and  Leveling, Johannes  and  Monz, Christof  and  Pecina, Pavel  and  Post, Matt  and  Saint-Amand, Herve  and  Soricut, Radu  and  Specia, Lucia  and  Tamchyna, Ale\\v{s}},\n  title     = {Findings of the 2014 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2014},\n  address   = {Baltimore, Maryland, USA},\n  publisher = {Association for Computational Linguistics},\n  pages     = {12--58},\n  url       = {http://www.aclweb.org/anthology/W/W14/W14-3302}\n}",
+        langpairs={
+            "cs-en": [
+                "test/newstest2014-csen-src.cs.sgm",
+                "test/newstest2014-csen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test/newstest2014-csen-src.en.sgm",
+                "test/newstest2014-csen-ref.cs.sgm",
+            ],
+            "de-en": [
+                "test/newstest2014-deen-src.de.sgm",
+                "test/newstest2014-deen-ref.en.sgm",
+            ],
+            "en-de": [
+                "test/newstest2014-deen-src.en.sgm",
+                "test/newstest2014-deen-ref.de.sgm",
+            ],
+            "en-fr": [
+                "test/newstest2014-fren-src.en.sgm",
+                "test/newstest2014-fren-ref.fr.sgm",
+            ],
+            "fr-en": [
+                "test/newstest2014-fren-src.fr.sgm",
+                "test/newstest2014-fren-ref.en.sgm",
+            ],
+            "en-hi": [
+                "test/newstest2014-hien-src.en.sgm",
+                "test/newstest2014-hien-ref.hi.sgm",
+            ],
+            "hi-en": [
+                "test/newstest2014-hien-src.hi.sgm",
+                "test/newstest2014-hien-ref.en.sgm",
+            ],
+            "en-ru": [
+                "test/newstest2014-ruen-src.en.sgm",
+                "test/newstest2014-ruen-ref.ru.sgm",
+            ],
+            "ru-en": [
+                "test/newstest2014-ruen-src.ru.sgm",
+                "test/newstest2014-ruen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt14/full": FakeSGMLDataset(
+        "wmt14/full",
+        data=["http://statmt.org/wmt14/test-full.tgz"],
+        md5=["a8cd784e006feb32ac6f3d9ec7eb389a"],
+        description="Evaluation data released after official evaluation for further research.",
+        langpairs={
+            "cs-en": [
+                "test-full/newstest2014-csen-src.cs.sgm",
+                "test-full/newstest2014-csen-ref.en.sgm",
+            ],
+            "en-cs": [
+                "test-full/newstest2014-csen-src.en.sgm",
+                "test-full/newstest2014-csen-ref.cs.sgm",
+            ],
+            "de-en": [
+                "test-full/newstest2014-deen-src.de.sgm",
+                "test-full/newstest2014-deen-ref.en.sgm",
+            ],
+            "en-de": [
+                "test-full/newstest2014-deen-src.en.sgm",
+                "test-full/newstest2014-deen-ref.de.sgm",
+            ],
+            "en-fr": [
+                "test-full/newstest2014-fren-src.en.sgm",
+                "test-full/newstest2014-fren-ref.fr.sgm",
+            ],
+            "fr-en": [
+                "test-full/newstest2014-fren-src.fr.sgm",
+                "test-full/newstest2014-fren-ref.en.sgm",
+            ],
+            "en-hi": [
+                "test-full/newstest2014-hien-src.en.sgm",
+                "test-full/newstest2014-hien-ref.hi.sgm",
+            ],
+            "hi-en": [
+                "test-full/newstest2014-hien-src.hi.sgm",
+                "test-full/newstest2014-hien-ref.en.sgm",
+            ],
+            "en-ru": [
+                "test-full/newstest2014-ruen-src.en.sgm",
+                "test-full/newstest2014-ruen-ref.ru.sgm",
+            ],
+            "ru-en": [
+                "test-full/newstest2014-ruen-src.ru.sgm",
+                "test-full/newstest2014-ruen-ref.en.sgm",
+            ],
+        },
+    ),
+    "wmt13": FakeSGMLDataset(
+        "wmt13",
+        data=["http://statmt.org/wmt13/test.tgz"],
+        md5=["48eca5d02f637af44e85186847141f67"],
+        description="Official evaluation data.",
+        citation="@InProceedings{bojar-EtAl:2013:WMT,\n  author    = {Bojar, Ond\\v{r}ej  and  Buck, Christian  and  Callison-Burch, Chris  and  Federmann, Christian  and  Haddow, Barry  and  Koehn, Philipp  and  Monz, Christof  and  Post, Matt  and  Soricut, Radu  and  Specia, Lucia},\n  title     = {Findings of the 2013 {Workshop on Statistical Machine Translation}},\n  booktitle = {Proceedings of the Eighth Workshop on Statistical Machine Translation},\n  month     = {August},\n  year      = {2013},\n  address   = {Sofia, Bulgaria},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--44},\n  url       = {http://www.aclweb.org/anthology/W13-2201}\n}",
+        langpairs={
+            "cs-en": ["test/newstest2013-src.cs.sgm", "test/newstest2013-src.en.sgm"],
+            "en-cs": ["test/newstest2013-src.en.sgm", "test/newstest2013-src.cs.sgm"],
+            "de-en": ["test/newstest2013-src.de.sgm", "test/newstest2013-src.en.sgm"],
+            "en-de": ["test/newstest2013-src.en.sgm", "test/newstest2013-src.de.sgm"],
+            "es-en": ["test/newstest2013-src.es.sgm", "test/newstest2013-src.en.sgm"],
+            "en-es": ["test/newstest2013-src.en.sgm", "test/newstest2013-src.es.sgm"],
+            "fr-en": ["test/newstest2013-src.fr.sgm", "test/newstest2013-src.en.sgm"],
+            "en-fr": ["test/newstest2013-src.en.sgm", "test/newstest2013-src.fr.sgm"],
+            "ru-en": ["test/newstest2013-src.ru.sgm", "test/newstest2013-src.en.sgm"],
+            "en-ru": ["test/newstest2013-src.en.sgm", "test/newstest2013-src.ru.sgm"],
+        },
+    ),
+    "wmt12": FakeSGMLDataset(
+        "wmt12",
+        data=["http://statmt.org/wmt12/test.tgz"],
+        md5=["608232d34ebc4ba2ff70fead45674e47"],
+        description="Official evaluation data.",
+        citation="@InProceedings{callisonburch-EtAl:2012:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Post, Matt  and  Soricut, Radu  and  Specia, Lucia},\n  title     = {Findings of the 2012 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Seventh Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2012},\n  address   = {Montr{'e}al, Canada},\n  publisher = {Association for Computational Linguistics},\n  pages     = {10--51},\n  url       = {http://www.aclweb.org/anthology/W12-3102}\n}",
+        langpairs={
+            "cs-en": ["test/newstest2012-src.cs.sgm", "test/newstest2012-src.en.sgm"],
+            "en-cs": ["test/newstest2012-src.en.sgm", "test/newstest2012-src.cs.sgm"],
+            "de-en": ["test/newstest2012-src.de.sgm", "test/newstest2012-src.en.sgm"],
+            "en-de": ["test/newstest2012-src.en.sgm", "test/newstest2012-src.de.sgm"],
+            "es-en": ["test/newstest2012-src.es.sgm", "test/newstest2012-src.en.sgm"],
+            "en-es": ["test/newstest2012-src.en.sgm", "test/newstest2012-src.es.sgm"],
+            "fr-en": ["test/newstest2012-src.fr.sgm", "test/newstest2012-src.en.sgm"],
+            "en-fr": ["test/newstest2012-src.en.sgm", "test/newstest2012-src.fr.sgm"],
+        },
+    ),
+    "wmt11": FakeSGMLDataset(
+        "wmt11",
+        data=["http://statmt.org/wmt11/test.tgz"],
+        md5=["b0c9680adf32d394aefc2b24e3a5937e"],
+        description="Official evaluation data.",
+        citation="@InProceedings{callisonburch-EtAl:2011:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Zaidan, Omar},\n  title     = {Findings of the 2011 Workshop on Statistical Machine Translation},\n  booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation},\n  month     = {July},\n  year      = {2011},\n  address   = {Edinburgh, Scotland},\n  publisher = {Association for Computational Linguistics},\n  pages     = {22--64},\n  url       = {http://www.aclweb.org/anthology/W11-2103}\n}",
+        langpairs={
+            "cs-en": ["newstest2011-src.cs.sgm", "newstest2011-src.en.sgm"],
+            "en-cs": ["newstest2011-src.en.sgm", "newstest2011-src.cs.sgm"],
+            "de-en": ["newstest2011-src.de.sgm", "newstest2011-src.en.sgm"],
+            "en-de": ["newstest2011-src.en.sgm", "newstest2011-src.de.sgm"],
+            "fr-en": ["newstest2011-src.fr.sgm", "newstest2011-src.en.sgm"],
+            "en-fr": ["newstest2011-src.en.sgm", "newstest2011-src.fr.sgm"],
+            "es-en": ["newstest2011-src.es.sgm", "newstest2011-src.en.sgm"],
+            "en-es": ["newstest2011-src.en.sgm", "newstest2011-src.es.sgm"],
+        },
+    ),
+    "wmt10": FakeSGMLDataset(
+        "wmt10",
+        data=["http://statmt.org/wmt10/test.tgz"],
+        md5=["491cb885a355da5a23ea66e7b3024d5c"],
+        description="Official evaluation data.",
+        citation="@InProceedings{callisonburch-EtAl:2010:WMT,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Peterson, Kay  and  Przybocki, Mark  and  Zaidan, Omar},\n  title     = {Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation},\n  booktitle = {Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR},\n  month     = {July},\n  year      = {2010},\n  address   = {Uppsala, Sweden},\n  publisher = {Association for Computational Linguistics},\n  pages     = {17--53},\n  note      = {Revised August 2010},\n  url       = {http://www.aclweb.org/anthology/W10-1703}\n}",
+        langpairs={
+            "cs-en": ["test/newstest2010-src.cz.sgm", "test/newstest2010-src.en.sgm"],
+            "en-cs": ["test/newstest2010-src.en.sgm", "test/newstest2010-src.cz.sgm"],
+            "de-en": ["test/newstest2010-src.de.sgm", "test/newstest2010-src.en.sgm"],
+            "en-de": ["test/newstest2010-src.en.sgm", "test/newstest2010-src.de.sgm"],
+            "es-en": ["test/newstest2010-src.es.sgm", "test/newstest2010-src.en.sgm"],
+            "en-es": ["test/newstest2010-src.en.sgm", "test/newstest2010-src.es.sgm"],
+            "fr-en": ["test/newstest2010-src.fr.sgm", "test/newstest2010-src.en.sgm"],
+            "en-fr": ["test/newstest2010-src.en.sgm", "test/newstest2010-src.fr.sgm"],
+        },
+    ),
+    "wmt09": FakeSGMLDataset(
+        "wmt09",
+        data=["http://statmt.org/wmt09/test.tgz"],
+        md5=["da227abfbd7b666ec175b742a0d27b37"],
+        description="Official evaluation data.",
+        citation="@InProceedings{callisonburch-EtAl:2009:WMT-09,\n  author    = {Callison-Burch, Chris  and  Koehn, Philipp  and  Monz, Christof  and  Schroeder, Josh},\n  title     = {Findings of the 2009 {W}orkshop on {S}tatistical {M}achine {T}ranslation},\n  booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation},\n  month     = {March},\n  year      = {2009},\n  address   = {Athens, Greece},\n  publisher = {Association for Computational Linguistics},\n  pages     = {1--28},\n  url       = {http://www.aclweb.org/anthology/W/W09/W09-0401}\n}",
+        langpairs={
+            "cs-en": ["test/newstest2009-src.cz.sgm", "test/newstest2009-src.en.sgm"],
+            "en-cs": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.cz.sgm"],
+            "de-en": ["test/newstest2009-src.de.sgm", "test/newstest2009-src.en.sgm"],
+            "en-de": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.de.sgm"],
+            "es-en": ["test/newstest2009-src.es.sgm", "test/newstest2009-src.en.sgm"],
+            "en-es": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.es.sgm"],
+            "fr-en": ["test/newstest2009-src.fr.sgm", "test/newstest2009-src.en.sgm"],
+            "en-fr": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.fr.sgm"],
+            "hu-en": ["test/newstest2009-src.hu.sgm", "test/newstest2009-src.en.sgm"],
+            "en-hu": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.hu.sgm"],
+            "it-en": ["test/newstest2009-src.it.sgm", "test/newstest2009-src.en.sgm"],
+            "en-it": ["test/newstest2009-src.en.sgm", "test/newstest2009-src.it.sgm"],
+        },
+    ),
+    "wmt08": FakeSGMLDataset(
+        "wmt08",
+        data=["http://statmt.org/wmt08/test.tgz"],
+        md5=["0582e4e894a3342044059c894e1aea3d"],
+        description="Official evaluation data.",
+        citation="@InProceedings{callisonburch-EtAl:2008:WMT,\n  author    = {Callison-Burch, Chris  and  Fordyce, Cameron  and  Koehn, Philipp  and  Monz, Christof  and  Schroeder, Josh},\n  title     = {Further Meta-Evaluation of Machine Translation},\n  booktitle = {Proceedings of the Third Workshop on Statistical Machine Translation},\n  month     = {June},\n  year      = {2008},\n  address   = {Columbus, Ohio},\n  publisher = {Association for Computational Linguistics},\n  pages     = {70--106},\n  url       = {http://www.aclweb.org/anthology/W/W08/W08-0309}\n}",
+        langpairs={
+            "cs-en": ["test/newstest2008-src.cz.sgm", "test/newstest2008-src.en.sgm"],
+            "en-cs": ["test/newstest2008-src.en.sgm", "test/newstest2008-src.cz.sgm"],
+            "de-en": ["test/newstest2008-src.de.sgm", "test/newstest2008-src.en.sgm"],
+            "en-de": ["test/newstest2008-src.en.sgm", "test/newstest2008-src.de.sgm"],
+            "es-en": ["test/newstest2008-src.es.sgm", "test/newstest2008-src.en.sgm"],
+            "en-es": ["test/newstest2008-src.en.sgm", "test/newstest2008-src.es.sgm"],
+            "fr-en": ["test/newstest2008-src.fr.sgm", "test/newstest2008-src.en.sgm"],
+            "en-fr": ["test/newstest2008-src.en.sgm", "test/newstest2008-src.fr.sgm"],
+            "hu-en": ["test/newstest2008-src.hu.sgm", "test/newstest2008-src.en.sgm"],
+            "en-hu": ["test/newstest2008-src.en.sgm", "test/newstest2008-src.hu.sgm"],
+        },
+    ),
+    "wmt08/nc": FakeSGMLDataset(
+        "wmt08/nc",
+        data=["http://statmt.org/wmt08/test.tgz"],
+        md5=["0582e4e894a3342044059c894e1aea3d"],
+        description="Official evaluation data (news commentary).",
+        langpairs={
+            "cs-en": ["test/nc-test2008-src.cz.sgm", "test/nc-test2008-src.en.sgm"],
+            "en-cs": ["test/nc-test2008-src.en.sgm", "test/nc-test2008-src.cz.sgm"],
+        },
+    ),
+    "wmt08/europarl": FakeSGMLDataset(
+        "wmt08/europarl",
+        data=["http://statmt.org/wmt08/test.tgz"],
+        md5=["0582e4e894a3342044059c894e1aea3d"],
+        description="Official evaluation data (Europarl).",
+        langpairs={
+            "de-en": ["test/test2008-src.de.sgm", "test/test2008-src.en.sgm"],
+            "en-de": ["test/test2008-src.en.sgm", "test/test2008-src.de.sgm"],
+            "es-en": ["test/test2008-src.es.sgm", "test/test2008-src.en.sgm"],
+            "en-es": ["test/test2008-src.en.sgm", "test/test2008-src.es.sgm"],
+            "fr-en": ["test/test2008-src.fr.sgm", "test/test2008-src.en.sgm"],
+            "en-fr": ["test/test2008-src.en.sgm", "test/test2008-src.fr.sgm"],
+        },
+    ),
+    # iwslt
+    "iwslt17": IWSLTXMLDataset(
+        "iwslt17",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ar/en-ar.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ar/en/ar-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ja/en-ja.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ja/en/ja-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ko/en-ko.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ko/en/ko-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "1849bcc3b006dc0642a8843b11aa7192",
+            "79bf7a2ef02d226875f55fb076e7e473",
+            "b68e7097b179491f6c466ef41ad72b9b",
+            "e3f5b2a075a2da1a395c8b60bf1e9be1",
+            "ecdc6bc4ab4c8984e919444f3c05183a",
+            "4b5141d14b98706c081371e2f8afe0ca",
+            "d957ee79de1f33c89077d37c5a2c5b06",
+            "c213e8bb918ebf843543fe9fd2e33db2",
+            "59f6a81c707378176e9ad8bb8d811f5f",
+            "7e580af973bb389ec1d1378a1850742f",
+            "975a858783a0ebec8c57d83ddd5bd381",
+            "cc51d9b7fe1ff2af858c6a0dd80b8815",
+        ],
+        description="Official evaluation data for IWSLT.",
+        citation="@InProceedings{iwslt2017,\n  author    = {Cettolo, Mauro and Federico, Marcello and Bentivogli, Luisa and Niehues, Jan and Stüker, Sebastian and Sudoh, Katsuitho and Yoshino, Koichiro and Federmann, Christian},\n  title     = {Overview of the IWSLT 2017 Evaluation Campaign},\n  booktitle = {14th International Workshop on Spoken Language Translation},\n  month     = {December},\n  year      = {2017},\n  address   = {Tokyo, Japan},\n  pages     = {2--14},\n  url       = {http://workshop2017.iwslt.org/downloads/iwslt2017_proceeding_v2.pdf}\n}",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2017.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2017.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2017.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2017.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2017.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2017.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2017.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2017.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2017.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2017.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2017.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2017.en-zh.en.xml",
+            ],
+            "en-ar": [
+                "en-ar/IWSLT17.TED.tst2017.en-ar.en.xml",
+                "ar-en/IWSLT17.TED.tst2017.ar-en.ar.xml",
+            ],
+            "ar-en": [
+                "ar-en/IWSLT17.TED.tst2017.ar-en.ar.xml",
+                "en-ar/IWSLT17.TED.tst2017.en-ar.en.xml",
+            ],
+            "en-ja": [
+                "en-ja/IWSLT17.TED.tst2017.en-ja.en.xml",
+                "ja-en/IWSLT17.TED.tst2017.ja-en.ja.xml",
+            ],
+            "ja-en": [
+                "ja-en/IWSLT17.TED.tst2017.ja-en.ja.xml",
+                "en-ja/IWSLT17.TED.tst2017.en-ja.en.xml",
+            ],
+            "en-ko": [
+                "en-ko/IWSLT17.TED.tst2017.en-ko.en.xml",
+                "ko-en/IWSLT17.TED.tst2017.ko-en.ko.xml",
+            ],
+            "ko-en": [
+                "ko-en/IWSLT17.TED.tst2017.ko-en.ko.xml",
+                "en-ko/IWSLT17.TED.tst2017.en-ko.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2016": IWSLTXMLDataset(
+        "iwslt17/tst2016",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "1849bcc3b006dc0642a8843b11aa7192",
+            "79bf7a2ef02d226875f55fb076e7e473",
+            "b68e7097b179491f6c466ef41ad72b9b",
+            "e3f5b2a075a2da1a395c8b60bf1e9be1",
+            "975a858783a0ebec8c57d83ddd5bd381",
+            "cc51d9b7fe1ff2af858c6a0dd80b8815",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2016.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2016.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2016.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2016.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2016.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2016.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2016.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2016.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2016.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2016.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2016.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2016.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2015": IWSLTXMLDataset(
+        "iwslt17/tst2015",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2015.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2015.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2015.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2015.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2015.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2015.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2015.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2015.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2015.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2015.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2015.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2015.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2014": IWSLTXMLDataset(
+        "iwslt17/tst2014",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2014.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2014.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2014.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2014.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2014.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2014.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2014.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2014.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2014.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2014.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2014.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2014.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2013": IWSLTXMLDataset(
+        "iwslt17/tst2013",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2013.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2013.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2013.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2013.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2013.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2013.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2013.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2013.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2013.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2013.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2013.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2013.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2012": IWSLTXMLDataset(
+        "iwslt17/tst2012",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2012.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2012.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2012.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2012.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2012.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2012.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2012.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2012.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2012.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2012.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2012.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2012.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2011": IWSLTXMLDataset(
+        "iwslt17/tst2011",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2011.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2011.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2011.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2011.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2011.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2011.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2011.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2011.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2011.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2011.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2011.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2011.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/tst2010": IWSLTXMLDataset(
+        "iwslt17/tst2010",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.tst2010.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.tst2010.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.tst2010.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.tst2010.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.tst2010.en-de.en.xml",
+                "de-en/IWSLT17.TED.tst2010.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.tst2010.de-en.de.xml",
+                "en-de/IWSLT17.TED.tst2010.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.tst2010.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.tst2010.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.tst2010.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.tst2010.en-zh.en.xml",
+            ],
+        },
+    ),
+    "iwslt17/dev2010": IWSLTXMLDataset(
+        "iwslt17/dev2010",
+        data=[
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgz",
+            "https://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz",
+        ],
+        md5=[
+            "d8a32cfc002a4f12b17429cfa78050e6",
+            "ca2b94d694150d4d6c5dc64c200fa589",
+            "3cf07ebe305312b12f7f1a4d5f8f8377",
+            "19927da9de0f40348cad9c0fc61642ac",
+            "575b788dad6c5b9c5cee636f9ac1094a",
+            "1c0ae40171d52593df8a6963d3828116",
+        ],
+        description="Development data for IWSLT 2017.",
+        langpairs={
+            "en-fr": [
+                "en-fr/IWSLT17.TED.dev2010.en-fr.en.xml",
+                "fr-en/IWSLT17.TED.dev2010.fr-en.fr.xml",
+            ],
+            "fr-en": [
+                "fr-en/IWSLT17.TED.dev2010.fr-en.fr.xml",
+                "en-fr/IWSLT17.TED.dev2010.en-fr.en.xml",
+            ],
+            "en-de": [
+                "en-de/IWSLT17.TED.dev2010.en-de.en.xml",
+                "de-en/IWSLT17.TED.dev2010.de-en.de.xml",
+            ],
+            "de-en": [
+                "de-en/IWSLT17.TED.dev2010.de-en.de.xml",
+                "en-de/IWSLT17.TED.dev2010.en-de.en.xml",
+            ],
+            "en-zh": [
+                "en-zh/IWSLT17.TED.dev2010.en-zh.en.xml",
+                "zh-en/IWSLT17.TED.dev2010.zh-en.zh.xml",
+            ],
+            "zh-en": [
+                "zh-en/IWSLT17.TED.dev2010.zh-en.zh.xml",
+                "en-zh/IWSLT17.TED.dev2010.en-zh.en.xml",
+            ],
+        },
+    ),
+    # mtedx
+    "mtedx/valid": PlainTextDataset(
+        "mtedx/valid",
+        data=[
+            "https://raw.githubusercontent.com/esalesky/mtedx-eval/main/valid.tar.gz"
+        ],
+        description="mTEDx evaluation data, valid: http://openslr.org/100",
+        citation="@misc{salesky2021multilingual,\n      title={The Multilingual TEDx Corpus for Speech Recognition and Translation}, \n      author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post},\n      year={2021},\n      eprint={2102.01757},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}",
+        md5=["40618171614c50e6cbb5e5bbceee0635"],
+        langpairs={
+            "el-en": ["valid/mtedx-valid-elen.el", "valid/mtedx-valid-elen.en"],
+            "es-en": ["valid/mtedx-valid-esen.es", "valid/mtedx-valid-esen.en"],
+            "es-fr": ["valid/mtedx-valid-esfr.es", "valid/mtedx-valid-esfr.fr"],
+            "es-it": ["valid/mtedx-valid-esit.es", "valid/mtedx-valid-esit.it"],
+            "es-pt": ["valid/mtedx-valid-espt.es", "valid/mtedx-valid-espt.pt"],
+            "fr-en": ["valid/mtedx-valid-fren.fr", "valid/mtedx-valid-fren.en"],
+            "fr-es": ["valid/mtedx-valid-fres.fr", "valid/mtedx-valid-fres.es"],
+            "fr-pt": ["valid/mtedx-valid-frpt.fr", "valid/mtedx-valid-frpt.pt"],
+            "it-en": ["valid/mtedx-valid-iten.it", "valid/mtedx-valid-iten.en"],
+            "it-es": ["valid/mtedx-valid-ites.it", "valid/mtedx-valid-ites.es"],
+            "pt-en": ["valid/mtedx-valid-pten.pt", "valid/mtedx-valid-pten.en"],
+            "pt-es": ["valid/mtedx-valid-ptes.pt", "valid/mtedx-valid-ptes.es"],
+            "ru-en": ["valid/mtedx-valid-ruen.ru", "valid/mtedx-valid-ruen.en"],
+        },
+    ),
+    "mtedx/test": PlainTextDataset(
+        "mtedx/test",
+        data=["https://raw.githubusercontent.com/esalesky/mtedx-eval/main/test.tar.gz"],
+        description="mTEDx evaluation data, test: http://openslr.org/100",
+        citation="@misc{salesky2021multilingual,\n      title={The Multilingual TEDx Corpus for Speech Recognition and Translation}, \n      author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post},\n      year={2021},\n      eprint={2102.01757},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}",
+        md5=["fa4cb1548c210ec424d7d6bc9a3675a7"],
+        langpairs={
+            "el-en": ["test/mtedx-test-elen.el", "test/mtedx-test-elen.en"],
+            "es-en": ["test/mtedx-test-esen.es", "test/mtedx-test-esen.en"],
+            "es-fr": ["test/mtedx-test-esfr.es", "test/mtedx-test-esfr.fr"],
+            "es-it": ["test/mtedx-test-esit.es", "test/mtedx-test-esit.it"],
+            "es-pt": ["test/mtedx-test-espt.es", "test/mtedx-test-espt.pt"],
+            "fr-en": ["test/mtedx-test-fren.fr", "test/mtedx-test-fren.en"],
+            "fr-es": ["test/mtedx-test-fres.fr", "test/mtedx-test-fres.es"],
+            "fr-pt": ["test/mtedx-test-frpt.fr", "test/mtedx-test-frpt.pt"],
+            "it-en": ["test/mtedx-test-iten.it", "test/mtedx-test-iten.en"],
+            "it-es": ["test/mtedx-test-ites.it", "test/mtedx-test-ites.es"],
+            "pt-en": ["test/mtedx-test-pten.pt", "test/mtedx-test-pten.en"],
+            "pt-es": ["test/mtedx-test-ptes.pt", "test/mtedx-test-ptes.es"],
+            "ru-en": ["test/mtedx-test-ruen.ru", "test/mtedx-test-ruen.en"],
+        },
+    ),
+    # multi30k
+    "multi30k/2016": PlainTextDataset(
+        "multi30k/2016",
+        data=[
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz"
+        ],
+        md5=["9cf8f22d57fee2ca2af3c682dfdc525b"],
+        description="2016 flickr test set of Multi30k dataset",
+        citation='@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}',
+        langpairs={
+            "en-fr": ["test_2016_flickr.en", "test_2016_flickr.fr"],
+            "en-de": ["test_2016_flickr.en", "test_2016_flickr.de"],
+            "en-cs": ["test_2016_flickr.en", "test_2016_flickr.cs"],
+        },
+    ),
+    "multi30k/2017": PlainTextDataset(
+        "multi30k/2017",
+        data=[
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz"
+        ],
+        md5=["9cf8f22d57fee2ca2af3c682dfdc525b"],
+        description="2017 flickr test set of Multi30k dataset",
+        citation='@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}\n\n@InProceedings{elliott-etal-2017-findings,\n    title = "Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description",\n    author = {Elliott, Desmond  and Frank, Stella  and Barrault, Lo{\\"\\i}c  and Bougares, Fethi  and Specia, Lucia},\n    booktitle = "Proceedings of the Second Conference on Machine Translation",\n    month = sep,\n    year = "2017",\n    address = "Copenhagen, Denmark",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W17-4718",\n    doi = "10.18653/v1/W17-4718",\n    pages = "215--233",\n}\n',
+        langpairs={
+            "en-fr": ["test_2017_flickr.en", "test_2017_flickr.fr"],
+            "en-de": ["test_2017_flickr.en", "test_2017_flickr.de"],
+        },
+    ),
+    "multi30k/2018": PlainTextDataset(
+        "multi30k/2018",
+        data=[
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz",
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.cs.gz",
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.de.gz",
+            "https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.fr.gz",
+        ],
+        md5=[
+            "9cf8f22d57fee2ca2af3c682dfdc525b",
+            "4c6b6490e58107b2e397c5e3e1690abc",
+            "87e00327083dd69feaa029a8f7c1a047",
+            "a64563e986438ed731a6713027c36bfd",
+        ],
+        description="2018 flickr test set of Multi30k dataset. See https://competitions.codalab.org/competitions/19917 for evaluation.",
+        citation='@InProceedings{elliott-etal-2016-multi30k,\n    title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions",\n    author = "Elliott, Desmond  and Frank, Stella  and Sima{\'}an, Khalil  and Specia, Lucia",\n    booktitle = "Proceedings of the 5th Workshop on Vision and Language",\n    month = aug,\n    year = "2016",\n    address = "Berlin, Germany",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W16-3210",\n    doi = "10.18653/v1/W16-3210",\n    pages = "70--74",\n}\n\n@InProceedings{barrault-etal-2018-findings,\n    title = "Findings of the Third Shared Task on Multimodal Machine Translation",\n    author = {Barrault, Lo{\\"\\i}c  and Bougares, Fethi  and Specia, Lucia  and Lala, Chiraag  and Elliott, Desmond  and Frank, Stella},\n    booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers",\n    month = oct,\n    year = "2018",\n    address = "Belgium, Brussels",\n    publisher = "Association for Computational Linguistics",\n    url = "https://www.aclweb.org/anthology/W18-6402",\n    doi = "10.18653/v1/W18-6402",\n    pages = "304--323",\n}\n',
+        langpairs={
+            "en-fr": ["test_2018_flickr.en", "multi30k_2018.test_2018_flickr.fr.gz"],
+            "en-de": ["test_2018_flickr.en", "multi30k_2018.test_2018_flickr.de.gz"],
+            "en-cs": ["test_2018_flickr.en", "multi30k_2018.test_2018_flickr.cs.gz"],
+        },
+    ),
+    # mtnt
+    "mtnt2019": TSVDataset(
+        "mtnt2019",
+        data=["https://pmichel31415.github.io/hosting/MTNT2019.tar.gz"],
+        description="Test set for the WMT 19 robustness shared task",
+        md5=["78a672e1931f106a8549023c0e8af8f6"],
+        langpairs={
+            "en-fr": ["2:MTNT2019/en-fr.final.tsv", "3:MTNT2019/en-fr.final.tsv"],
+            "fr-en": ["2:MTNT2019/fr-en.final.tsv", "3:MTNT2019/fr-en.final.tsv"],
+            "en-ja": ["2:MTNT2019/en-ja.final.tsv", "3:MTNT2019/en-ja.final.tsv"],
+            "ja-en": ["2:MTNT2019/ja-en.final.tsv", "3:MTNT2019/ja-en.final.tsv"],
+        },
+    ),
+    "mtnt1.1/test": TSVDataset(
+        "mtnt1.1/test",
+        data=[
+            "https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz"
+        ],
+        description="Test data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/",
+        citation='@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
+        md5=["8ce1831ac584979ba8cdcd9d4be43e1d"],
+        langpairs={
+            "en-fr": ["1:MTNT/test/test.en-fr.tsv", "2:MTNT/test/test.en-fr.tsv"],
+            "fr-en": ["1:MTNT/test/test.fr-en.tsv", "2:MTNT/test/test.fr-en.tsv"],
+            "en-ja": ["1:MTNT/test/test.en-ja.tsv", "2:MTNT/test/test.en-ja.tsv"],
+            "ja-en": ["1:MTNT/test/test.ja-en.tsv", "2:MTNT/test/test.ja-en.tsv"],
+        },
+    ),
+    "mtnt1.1/valid": TSVDataset(
+        "mtnt1.1/valid",
+        data=[
+            "https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz"
+        ],
+        description="Validation data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/",
+        citation='@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
+        md5=["8ce1831ac584979ba8cdcd9d4be43e1d"],
+        langpairs={
+            "en-fr": ["1:MTNT/valid/valid.en-fr.tsv", "2:MTNT/valid/valid.en-fr.tsv"],
+            "fr-en": ["1:MTNT/valid/valid.fr-en.tsv", "2:MTNT/valid/valid.fr-en.tsv"],
+            "en-ja": ["1:MTNT/valid/valid.en-ja.tsv", "2:MTNT/valid/valid.en-ja.tsv"],
+            "ja-en": ["1:MTNT/valid/valid.ja-en.tsv", "2:MTNT/valid/valid.ja-en.tsv"],
+        },
+    ),
+    "mtnt1.1/train": TSVDataset(
+        "mtnt1.1/train",
+        data=[
+            "https://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gz"
+        ],
+        description="Validation data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/",
+        citation='@InProceedings{michel2018a:mtnt,\n    author = "Michel, Paul and Neubig, Graham",\n    title = "MTNT: A Testbed for Machine Translation of Noisy Text",\n    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",\n    year = "2018",\n    publisher = "Association for Computational Linguistics",\n    pages = "543--553",\n    location = "Brussels, Belgium",\n    url = "http://aclweb.org/anthology/D18-1050"\n}',
+        md5=["8ce1831ac584979ba8cdcd9d4be43e1d"],
+        langpairs={
+            "en-fr": ["1:MTNT/train/train.en-fr.tsv", "2:MTNT/train/train.en-fr.tsv"],
+            "fr-en": ["1:MTNT/train/train.fr-en.tsv", "2:MTNT/train/train.fr-en.tsv"],
+            "en-ja": ["1:MTNT/train/train.en-ja.tsv", "2:MTNT/train/train.en-ja.tsv"],
+            "ja-en": ["1:MTNT/train/train.ja-en.tsv", "2:MTNT/train/train.ja-en.tsv"],
+        },
+    ),
+}
diff --git a/sacrebleu/dataset/__main__.py b/sacrebleu/dataset/__main__.py
new file mode 100644
index 00000000..22954920
--- /dev/null
+++ b/sacrebleu/dataset/__main__.py
@@ -0,0 +1,43 @@
+import sys
+
+from . import DATASETS
+
+try:
+    cmd = sys.argv[1]
+except IndexError:
+    print(f"Usage: {sys.argv[0]} --check | --dump")
+    sys.exit(1)
+
+if cmd == "--check":
+    import hashlib
+    import urllib.request
+
+    url_md5 = {}
+
+    for item in DATASETS.values():
+        if item.md5 is not None:
+            assert len(item.data) == len(item.md5)
+            pairs = zip(item.data, item.md5)
+            for url, md5_hash in pairs:
+                url_md5[url] = md5_hash
+
+    for url, md5_hash in url_md5.items():
+        try:
+            print("Downloading ", url)
+            with urllib.request.urlopen(url) as f:
+                data = f.read()
+        except Exception as exc:
+            raise (exc)
+
+        if hashlib.md5(data).hexdigest() != md5_hash:
+            print("MD5 check failed for", url)
+elif cmd == "--dump":
+    import re
+
+    # Dumps a table in markdown format
+    print(f'| {"Dataset":<30} | {"Description":<115} |')
+    header = "| " + "-" * 30 + " | " + "-" * 115 + " |"
+    print(header)
+    for name, item in DATASETS.items():
+        desc = re.sub(r"(http[s]?:\/\/\S+)", r"[URL](\1)", str(item.description))
+        print(f"| {name:<30} | {desc:<115} |")
diff --git a/sacrebleu/dataset/base.py b/sacrebleu/dataset/base.py
new file mode 100644
index 00000000..ba6e65ba
--- /dev/null
+++ b/sacrebleu/dataset/base.py
@@ -0,0 +1,195 @@
+"""
+The base class for all types of datasets.
+"""
+import os
+import re
+from abc import ABCMeta, abstractmethod
+from typing import Dict, List
+
+from ..utils import SACREBLEU_DIR, download_file, smart_open
+
+
+class Dataset(metaclass=ABCMeta):
+    def __init__(
+        self,
+        name: str,
+        data: List[str] = None,
+        description: str = None,
+        citation: str = None,
+        md5: List[str] = None,
+        langpairs=Dict[str, List[str]],
+        **kwargs,
+    ):
+        """
+        Params come from the values in DATASETS.
+
+        :param name: Name of the dataset.
+        :param data: URL of the raw data of the dataset.
+        :param description: Description of the dataset.
+        :param citation: Citation for the dataset.
+        :param md5: MD5 checksum of the dataset.
+        :param langpairs: List of available language pairs.
+        """
+        self.name = name
+        self.data = data
+        self.description = description
+        self.citation = citation
+        self.md5 = md5
+        self.langpairs = langpairs
+        self.kwargs = kwargs
+
+        # Don't do any downloading or further processing now.
+        # Only do that lazily, when asked.
+
+        # where to store the dataset
+        self._outdir = os.path.join(SACREBLEU_DIR, self.name)
+        self._rawdir = os.path.join(self._outdir, "raw")
+
+    def maybe_download(self):
+        """
+        If the dataset isn't downloaded, use utils/download_file()
+        This can be implemented here in the base class. It should write
+        to ~/.sacreleu/DATASET/raw exactly as it does now.
+        """
+        os.makedirs(self._rawdir, exist_ok=True)
+
+        expected_checksums = self.md5 if self.md5 else [None] * len(self.data)
+
+        for url, expected_md5 in zip(self.data, expected_checksums):
+            tarball = os.path.join(self._rawdir, self._get_tarball_filename(url))
+
+            download_file(
+                url, tarball, extract_to=self._rawdir, expected_md5=expected_md5
+            )
+
+    @staticmethod
+    def _clean(s):
+        """
+        Removes trailing and leading spaces and collapses multiple consecutive internal spaces to a single one.
+
+        :param s: The string.
+        :return: A cleaned-up string.
+        """
+        return re.sub(r"\s+", " ", s.strip())
+
+    def _get_tarball_filename(self, url):
+        """
+        Produces a local filename for tarball.
+        :param url: The url to download.
+        :return: A name produced from the dataset identifier and the URL basename.
+        """
+        return self.name.replace("/", "_") + "." + os.path.basename(url)
+
+    def _get_txt_file_path(self, langpair, fieldname):
+        """
+        Given the language pair and fieldname, return the path to the text file.
+        The format is: ~/.sacrebleu/DATASET/DATASET.LANGPAIR.FIELDNAME
+
+        :param langpair: The language pair.
+        :param fieldname: The fieldname.
+        :return: The path to the text file.
+        """
+        # handle the special case of subsets. e.g. "wmt21/dev" > "wmt21_dev"
+        name = self.name.replace("/", "_")
+        # Colons are used to distinguish multiple references, but are not supported in Windows filenames
+        fieldname = fieldname.replace(":", "-")
+        return os.path.join(self._outdir, f"{name}.{langpair}.{fieldname}")
+
+    def _get_langpair_metadata(self, langpair):
+        """
+        Given a language pair, return the metadata for that language pair.
+        Deal with errors if the language pair is not available.
+
+        :param langpair: The language pair. e.g. "en-de"
+        :return: Dict format which is same as self.langpairs.
+        """
+        if langpair is None:
+            langpairs = self.langpairs
+        elif langpair not in self.langpairs:
+            raise Exception(f"No such language pair {self.name}/{langpair}")
+        else:
+            langpairs = {langpair: self.langpairs[langpair]}
+
+        return langpairs
+
+    @abstractmethod
+    def process_to_text(self, langpair=None) -> None:
+        """Processes raw files to plain text files.
+
+        :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed.
+        """
+        pass
+
+    def fieldnames(self, langpair) -> List[str]:
+        """
+        Return a list of all the field names. For most source, this is just
+        the source and the reference. For others, it might include the document
+        ID for each line, or the original language (origLang).
+
+        get_files() should return the same number of items as this.
+
+        :param langpair: The language pair (e.g., "de-en")
+        :return: a list of field names
+        """
+        return ["src", "ref"]
+
+    def __iter__(self, langpair):
+        """
+        Iterates over all fields (source, references, and other metadata) defined
+        by the dataset.
+        """
+        all_files = self.get_files(langpair)
+        all_fins = [smart_open(f) for f in all_files]
+
+        for item in zip(*all_fins):
+            yield item
+
+    def source(self, langpair):
+        """
+        Return an iterable over the source lines.
+        """
+        source_file = self.get_source_file(langpair)
+        with smart_open(source_file) as fin:
+            for line in fin:
+                yield line.strip()
+
+    def references(self, langpair):
+        """
+        Return an iterable over the references.
+        """
+        ref_files = self.get_reference_files(langpair)
+        ref_fins = [smart_open(f) for f in ref_files]
+
+        for item in zip(*ref_fins):
+            yield item
+
+    def get_source_file(self, langpair):
+        all_files = self.get_files(langpair)
+        all_fields = self.fieldnames(langpair)
+        index = all_fields.index("src")
+        return all_files[index]
+
+    def get_reference_files(self, langpair):
+        all_files = self.get_files(langpair)
+        all_fields = self.fieldnames(langpair)
+        ref_files = [
+            f for f, field in zip(all_files, all_fields) if field.startswith("ref")
+        ]
+        return ref_files
+
+    def get_files(self, langpair):
+        """
+        Returns the path of the source file and all reference files for
+        the provided test set / language pair.
+        Downloads the references first if they are not already local.
+
+        :param langpair: The language pair (e.g., "de-en")
+        :return: a list of the source file and all reference files
+        """
+        fields = self.fieldnames(langpair)
+        files = [self._get_txt_file_path(langpair, field) for field in fields]
+
+        for file in files:
+            if not os.path.exists(file):
+                self.process_to_text(langpair)
+        return files
diff --git a/sacrebleu/dataset/fake_sgml.py b/sacrebleu/dataset/fake_sgml.py
new file mode 100644
index 00000000..d1f63812
--- /dev/null
+++ b/sacrebleu/dataset/fake_sgml.py
@@ -0,0 +1,116 @@
+import os
+import re
+
+from ..utils import smart_open
+from .base import Dataset
+
+
+class FakeSGMLDataset(Dataset):
+    """
+    The fake SGML format used by WMT prior to 2021. Can't be properly parsed.
+    Source and reference(s) in separate files.
+    """
+
+    def _convert_format(self, input_file_path, output_filep_path):
+        """
+        Extract data from raw file and convert to raw txt format.
+        """
+        with smart_open(input_file_path) as fin, smart_open(
+            output_filep_path, "wt"
+        ) as fout:
+            for line in fin:
+                if line.startswith("<seg "):
+                    line = self._clean(re.sub(r"<seg.*?>(.*)</seg>.*?", "\\1", line))
+                    print(line, file=fout)
+
+    def _convert_meta(self, input_file_path, field, output_filep_path):
+        """
+        Extract metadata from document tags, projects across segments.
+        """
+        with smart_open(input_file_path) as fin, smart_open(
+            output_filep_path, "wt"
+        ) as fout:
+            value = ""
+            for line in fin:
+                if line.startswith("<doc "):
+                    match = re.search(rf'{field}="(.*?)"', line)
+                    if match is not None:
+                        value = match.group(1)
+
+                elif line.startswith("<seg "):
+                    # print the current value once for each field
+                    print(value, file=fout)
+
+    def process_to_text(self, langpair=None):
+        """Processes raw files to plain text files.
+
+        :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed.
+        """
+        # ensure that the dataset is downloaded
+        self.maybe_download()
+        langpairs = self._get_langpair_metadata(langpair)
+
+        for langpair in langpairs:
+            fieldnames = self.fieldnames(langpair)
+            origin_files = [
+                os.path.join(self._rawdir, path) for path in langpairs[langpair]
+            ]
+
+            # Add the source file three more times for docid, genre, origlang
+            origin_files += [
+                os.path.join(self._rawdir, langpairs[langpair][0]) for _ in range(3)
+            ]
+
+            for field, origin_file in zip(fieldnames, origin_files):
+
+                origin_file = os.path.join(self._rawdir, origin_file)
+                output_file = self._get_txt_file_path(langpair, field)
+
+                if field.startswith("src") or field.startswith("ref"):
+                    self._convert_format(origin_file, output_file)
+                else:
+                    # document metadata keys
+                    self._convert_meta(origin_file, field, output_file)
+
+    def fieldnames(self, langpair):
+        """
+        Return a list of all the field names. For most source, this is just
+        the source and the reference. For others, it might include the document
+        ID for each line, or the original language (origLang).
+
+        get_files() should return the same number of items as this.
+        """
+        meta = self._get_langpair_metadata(langpair)
+        length = len(meta[langpair])
+
+        assert (
+            length >= 2
+        ), f"Each language pair in {self.name} must have at least 2 fields."
+
+        fields = ["src"]
+
+        if length == 2:
+            fields.append("ref")
+        else:
+            for i, _ in enumerate(meta[langpair][1:]):
+                fields.append(f"ref:{i}")
+
+        if not self.name.startswith("wmt08"):
+            fields += ["docid", "genre", "origlang"]
+
+        return fields
+
+
+class WMTAdditionDataset(FakeSGMLDataset):
+    """
+    Handle special case of WMT Google addition dataset.
+    """
+
+    def _convert_format(self, input_file_path, output_filep_path):
+        if input_file_path.endswith(".sgm"):
+            return super()._convert_format(input_file_path, output_filep_path)
+        else:
+            with smart_open(input_file_path) as fin:
+                with smart_open(output_filep_path, "wt") as fout:
+                    for line in fin:
+                        print(line.rstrip(), file=fout)
diff --git a/sacrebleu/dataset/iwslt_xml.py b/sacrebleu/dataset/iwslt_xml.py
new file mode 100644
index 00000000..4381271d
--- /dev/null
+++ b/sacrebleu/dataset/iwslt_xml.py
@@ -0,0 +1,8 @@
+from .fake_sgml import FakeSGMLDataset
+
+
+class IWSLTXMLDataset(FakeSGMLDataset):
+    """IWSLT dataset format. Can be parsed with the lxml parser."""
+
+    # Same as FakeSGMLDataset. Nothing to do here.
+    pass
diff --git a/sacrebleu/dataset/plain_text.py b/sacrebleu/dataset/plain_text.py
new file mode 100644
index 00000000..7f7a93db
--- /dev/null
+++ b/sacrebleu/dataset/plain_text.py
@@ -0,0 +1,36 @@
+import os
+
+from ..utils import smart_open
+from .base import Dataset
+
+
+class PlainTextDataset(Dataset):
+    """
+    The plain text format. Data is separated into source and reference files.
+    Each line of the two files is aligned.
+    """
+
+    def process_to_text(self, langpair=None):
+        """Processes raw files to plain text files.
+
+        :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed.
+        """
+        # ensure that the dataset is downloaded
+        self.maybe_download()
+        langpairs = self._get_langpair_metadata(langpair)
+
+        for langpair in langpairs:
+            fieldnames = self.fieldnames(langpair)
+            origin_files = [
+                os.path.join(self._rawdir, path) for path in langpairs[langpair]
+            ]
+
+            for field, origin_file in zip(fieldnames, origin_files):
+
+                origin_file = os.path.join(self._rawdir, origin_file)
+                output_file = self._get_txt_file_path(langpair, field)
+
+                with smart_open(origin_file) as fin:
+                    with smart_open(output_file, "wt") as fout:
+                        for line in fin:
+                            print(line.rstrip(), file=fout)
diff --git a/sacrebleu/dataset/tsv.py b/sacrebleu/dataset/tsv.py
new file mode 100644
index 00000000..011bf20d
--- /dev/null
+++ b/sacrebleu/dataset/tsv.py
@@ -0,0 +1,61 @@
+import os
+
+from ..utils import smart_open
+from .base import Dataset
+
+
+class TSVDataset(Dataset):
+    """
+    The format used by the MTNT datasets. Data is in a single TSV file.
+    """
+
+    @staticmethod
+    def _split_index_and_filename(meta, field):
+        """
+        Splits the index and filename from a metadata string.
+
+        e.g. meta="3:en-de.tsv", filed=[Any value] -> (3, "en-de.tsv")
+             "en-de.tsv", filed="src" -> (1, "en-de.tsv")
+             "en-de.tsv", filed="tgt" -> (2, "en-de.tsv")
+        """
+        arr = meta.split(":")
+        if len(arr) == 2:
+            try:
+                index = int(arr[0])
+            except ValueError:
+                raise Exception(f"Invalid meta for TSVDataset: {meta}")
+            return index, arr[1]
+
+        else:
+            index = 0 if field == "src" else 1
+            return index, meta
+
+    def process_to_text(self, langpair=None):
+        """Processes raw files to plain text files.
+
+        :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed.
+        """
+        # ensure that the dataset is downloaded
+        self.maybe_download()
+        langpairs = self._get_langpair_metadata(langpair)
+
+        for langpair in langpairs:
+            fieldnames = self.fieldnames(langpair)
+            origin_files = [
+                os.path.join(self._rawdir, path) for path in langpairs[langpair]
+            ]
+
+            for field, origin_file, meta in zip(
+                fieldnames, origin_files, langpairs[langpair]
+            ):
+                index, origin_file = self._split_index_and_filename(meta, field)
+
+                origin_file = os.path.join(self._rawdir, origin_file)
+                output_file = self._get_txt_file_path(langpair, field)
+
+                with smart_open(origin_file) as fin:
+                    with smart_open(output_file, "wt") as fout:
+                        for line in fin:
+                            # be careful with empty source or reference lines
+                            # MTNT2019/ja-en.final.tsv:632 `'1033\t718\t\t\n'`
+                            print(line.rstrip("\n").split("\t")[index], file=fout)
diff --git a/sacrebleu/dataset/wmt_xml.py b/sacrebleu/dataset/wmt_xml.py
new file mode 100644
index 00000000..d701742f
--- /dev/null
+++ b/sacrebleu/dataset/wmt_xml.py
@@ -0,0 +1,151 @@
+import os
+
+import lxml.etree as ET
+
+from ..utils import smart_open
+from .base import Dataset
+
+
+class WMTXMLDataset(Dataset):
+    """
+    The 2021+ WMT dataset format. Everything is contained in a single file.
+    Can be parsed with the lxml parser.
+    """
+
+    @staticmethod
+    def _unwrap_wmt21_or_later(raw_file, allowed_refs=[]):
+        """
+        Unwraps the XML file from wmt21 or later.
+        This script is adapted from https://github.com/wmt-conference/wmt-format-tools
+
+        :param raw_file: The raw xml file to unwrap.
+        :return: Dictionary which contains the following fields:
+            - `src`: The source sentences.
+            - `docid`: ID indicating which document the sentences belong to.
+            - `origlang`: The original language of the document.
+            - `ref:{translator}`: The references produced by each translator.
+            - `ref`: An alias for the references from the first translator.
+        """
+        tree = ET.parse(raw_file)
+        # Find and check the documents (src, ref, hyp)
+        src_langs, ref_langs, translators = set(), set(), set()
+        for src_doc in tree.getroot().findall(".//src"):
+            src_langs.add(src_doc.get("lang"))
+
+        for ref_doc in tree.getroot().findall(".//ref"):
+            ref_langs.add(ref_doc.get("lang"))
+            translator = ref_doc.get("translator")
+            if len(allowed_refs) == 0 or translator in allowed_refs:
+                translators.add(translator)
+
+        assert (
+            len(src_langs) == 1
+        ), f"Multiple source languages found in the file: {raw_file}"
+        assert (
+            len(ref_langs) == 1
+        ), f"Multiple reference languages found in the file: {raw_file}"
+        src = []
+        docids = []
+        orig_langs = []
+
+        def get_field_by_translator(translator):
+            if not translator:
+                return "ref"
+            else:
+                return f"ref:{translator}"
+
+        refs = {get_field_by_translator(translator): [] for translator in translators}
+
+        src_sent_count, doc_count = 0, 0
+        for doc in tree.getroot().findall(".//doc"):
+            docid = doc.attrib["id"]
+            origlang = doc.attrib["origlang"]
+
+            # Skip the testsuite
+            if "testsuite" in doc.attrib:
+                continue
+
+            doc_count += 1
+            src_sents = {
+                int(seg.get("id")): seg.text for seg in doc.findall(".//src//seg")
+            }
+
+            def get_sents(doc):
+                return {
+                    int(seg.get("id")): seg.text if seg.text else ""
+                    for seg in doc.findall(f".//seg")
+                }
+
+            ref_docs = doc.findall(".//ref")
+
+            trans_to_ref = {
+                ref_doc.get("translator"): get_sents(ref_doc) for ref_doc in ref_docs
+            }
+
+            for seg_id in sorted(src_sents.keys()):
+                # no ref translation is avaliable for this segment
+                if not any([value.get(seg_id, "") for value in trans_to_ref.values()]):
+                    continue
+                for translator in translators:
+                    refs[get_field_by_translator(translator)].append(
+                        trans_to_ref.get(translator, {translator: {}}).get(seg_id, "")
+                    )
+                src.append(src_sents[seg_id])
+                docids.append(docid)
+                orig_langs.append(origlang)
+                src_sent_count += 1
+
+        # For backward compatibility, if "ref" is not in the fields,
+        # add reference sentences from the first translator as "ref" field
+        if "ref" not in refs:
+            refs["ref"] = refs[min(refs.keys())]
+
+        return {"src": src, **refs, "docid": docids, "origlang": orig_langs,}
+
+    def process_to_text(self, langpair=None):
+        """Processes raw files to plain text files.
+
+        :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed.
+        """
+        # ensure that the dataset is downloaded
+        self.maybe_download()
+        langpairs = self._get_langpair_metadata(langpair)
+
+        for langpair, files in langpairs.items():
+            rawfile = os.path.join(
+                self._rawdir, files[0]
+            )  # all source and reference data in one file for wmt21 and later
+
+            with smart_open(rawfile) as fin:
+                fields = self._unwrap_wmt21_or_later(fin, allowed_refs=self.kwargs.get("refs", []))
+
+            for fieldname in fields:
+                textfile = self._get_txt_file_path(langpair, fieldname)
+
+                # skip if the file already exists
+                if os.path.exists(textfile) and os.path.getsize(textfile) > 0:
+                    continue
+
+                with smart_open(textfile, "w") as fout:
+                    for line in fields[fieldname]:
+                        print(self._clean(line), file=fout)
+
+    def fieldnames(self, langpair):
+        """
+        Return a list of all the field names. For most source, this is just
+        the source and the reference. For others, it might include the document
+        ID for each line, or the original language (origLang).
+
+        get_files() should return the same number of items as this.
+
+        :param langpair: The language pair (e.g., "de-en")
+        :return: a list of field names
+        """
+        self.maybe_download()
+        meta = self._get_langpair_metadata(langpair)[langpair]
+        rawfile = os.path.join(self._rawdir, meta[0])
+
+        with smart_open(rawfile) as fin:
+            fields = self._unwrap_wmt21_or_later(fin, allowed_refs=self.kwargs.get("refs", []))
+
+        return list(fields.keys())
diff --git a/sacrebleu/metrics/base.py b/sacrebleu/metrics/base.py
index e49d1c22..93fb1081 100644
--- a/sacrebleu/metrics/base.py
+++ b/sacrebleu/metrics/base.py
@@ -220,7 +220,7 @@ def _check_sentence_score_args(self, hyp: str, refs: Sequence[str]):
             err_msg = 'The argument `hyp` should be a string.'
         elif isinstance(refs, str) or not isinstance(refs, Sequence):
             err_msg = 'The argument `refs` should be a sequence of strings.'
-        elif not isinstance(refs[0], str):
+        elif not isinstance(refs[0], str) and refs[0] is not None:
             err_msg = 'Each element of `refs` should be a string.'
 
         if err_msg:
@@ -251,7 +251,7 @@ def _check_corpus_score_args(self, hyps: Sequence[str],
                 err_msg = "`refs` should be a sequence of sequence of strings."
             elif not isinstance(refs[0], Sequence):
                 err_msg = "Each element of `refs` should be a sequence of strings."
-            elif not isinstance(refs[0][0], str):
+            elif not isinstance(refs[0][0], str) and refs[0][0] is not None:
                 err_msg = "`refs` should be a sequence of sequence of strings."
 
         if err_msg:
@@ -325,12 +325,8 @@ def _cache_references(self, references: Sequence[Sequence[str]]) -> List[Any]:
         num_refs = set()
 
         for refs in zip(*references):
-            # remove undefined / empty references
-            # i.e. we have fewer references for this particular sentence
-            lines = [x for x in refs if x is not None and x != ""]
-
-            if len(lines) == 0:
-                raise RuntimeError("Empty or `None` reference sentence found.")
+            # Remove undefined references
+            lines = [x for x in refs if x is not None]
 
             # Keep track of reference counts to allow variable reference
             # info in the signature
diff --git a/sacrebleu/metrics/bleu.py b/sacrebleu/metrics/bleu.py
index e9fae14e..f1668811 100644
--- a/sacrebleu/metrics/bleu.py
+++ b/sacrebleu/metrics/bleu.py
@@ -16,12 +16,13 @@
 MAX_NGRAM_ORDER = 4
 
 _TOKENIZERS = {
-    'none': 'tokenizer_base.BaseTokenizer',
+    'none': 'tokenizer_none.NoneTokenizer',
     'zh': 'tokenizer_zh.TokenizerZh',
     '13a': 'tokenizer_13a.Tokenizer13a',
     'intl': 'tokenizer_intl.TokenizerV14International',
     'char': 'tokenizer_char.TokenizerChar',
     'ja-mecab': 'tokenizer_ja_mecab.TokenizerJaMecab',
+    'ko-mecab': 'tokenizer_ko_mecab.TokenizerKoMecab',
     'spm': 'tokenizer_spm.TokenizerSPM',
 }
 
@@ -133,7 +134,7 @@ class BLEU(Metric):
         'exp': None,    # No value is required
     }
 
-    TOKENIZERS = ['none', 'zh', '13a', 'char', 'intl', 'ja-mecab', 'spm']
+    TOKENIZERS = ['none', 'zh', '13a', 'char', 'intl', 'ja-mecab', 'ko-mecab', 'spm']
 
     # mteval-v13a.pl tokenizer unless Chinese or Japanese is provided
     TOKENIZER_DEFAULT = '13a'
@@ -143,13 +144,14 @@ class BLEU(Metric):
     _TOKENIZER_MAP = {
         'zh': 'zh',
         'ja': 'ja-mecab',
+        'ko': 'ko-mecab',
     }
 
     _SIGNATURE_TYPE = BLEUSignature
 
     def __init__(self, lowercase: bool = False,
                  force: bool = False,
-                 tokenize: Optional[str] = '13a',
+                 tokenize: Optional[str] = None,
                  smooth_method: str = 'exp',
                  smooth_value: Optional[float] = None,
                  max_ngram_order: int = MAX_NGRAM_ORDER,
@@ -171,21 +173,27 @@ def __init__(self, lowercase: bool = False,
         assert self.smooth_method in self.SMOOTH_DEFAULTS.keys(), \
             "Unknown smooth_method {self.smooth_method!r}"
 
-        # Default tokenizer logic
+        # If the tokenizer wasn't specified, choose it according to the
+        # following logic. We use 'v13a' except for ZH and JA. Note that
+        # this logic can only be applied when sacrebleu knows the target
+        # language, which is only the case for builtin datasets.
         if tokenize is None:
             best_tokenizer = self.TOKENIZER_DEFAULT
 
-            # Set `zh` or `ja-mecab` if target language is provided
+            # Set `zh` or `ja-mecab` or `ko-mecab` if target language is provided
             if self.trg_lang in self._TOKENIZER_MAP:
                 best_tokenizer = self._TOKENIZER_MAP[self.trg_lang]
         else:
             best_tokenizer = tokenize
             if self.trg_lang == 'zh' and best_tokenizer != 'zh':
                 sacrelogger.warning(
-                    "You should use the 'zh' tokenizer for Chinese.")
+                    "Consider using the 'zh' or 'spm' tokenizer for Chinese.")
             if self.trg_lang == 'ja' and best_tokenizer != 'ja-mecab':
                 sacrelogger.warning(
-                    "You should use the 'ja-mecab' tokenizer for Japanese.")
+                    "Consider using the 'ja-mecab' or 'spm' tokenizer for Japanese.")
+            if self.trg_lang == 'ko' and best_tokenizer != 'ko-mecab':
+                sacrelogger.warning(
+                    "Consider using the 'ko-mecab' or 'spm' tokenizer for Korean.")
 
         # Create the tokenizer
         self.tokenizer = _get_tokenizer(best_tokenizer)()
diff --git a/sacrebleu/sacrebleu.py b/sacrebleu/sacrebleu.py
index f52911e4..3664ae3b 100755
--- a/sacrebleu/sacrebleu.py
+++ b/sacrebleu/sacrebleu.py
@@ -80,8 +80,11 @@ def parse_args():
                             help='Use a subset of sentences whose document annotation matches a given regex (see SUBSETS in the source code).')
     arg_parser.add_argument('--download', type=str, default=None,
                             help='Download a test set and quit.')
-    arg_parser.add_argument('--echo', choices=['src', 'ref', 'both'], type=str, default=None,
-                            help='Output the source (src), reference (ref), or both (both, pasted) to STDOUT and quit.')
+    arg_parser.add_argument('--echo', nargs="*", type=str, default=None,
+                            help='Output the source (src), reference (ref), or other available field (docid, ref:A, ref:1 for example) to STDOUT and quit. '
+                                 'You can get available fields with options `--list` and `-t`' 'For example: `sacrebleu -t wmt21 --list`. '
+                                 'If multiple fields are given, they are outputted with tsv format in the order they are given.'
+                                 'You can also use `--echo all` to output all available fields.')
 
     # I/O related arguments
     # Multiple input files can be provided for significance testing for example
@@ -117,7 +120,8 @@ def parse_args():
                                 f"add-k: {METRICS['BLEU'].SMOOTH_DEFAULTS['add-k']})")
     bleu_args.add_argument('--tokenize', '-tok', choices=METRICS['BLEU'].TOKENIZERS, default=None,
                            dest='bleu_tokenize',
-                           help='Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, `ja-mecab` for Japanese and `13a` (mteval) otherwise.')
+                           help='Tokenization method to use for BLEU. If not provided, defaults to `zh` for Chinese, '
+                                '`ja-mecab` for Japanese, `ko-mecab` for Korean and `13a` (mteval) otherwise.')
     bleu_args.add_argument('--lowercase', '-lc', dest='bleu_lowercase', action='store_true', default=False,
                            help='If True, enables case-insensitivity. (Default: %(default)s)')
     bleu_args.add_argument('--force', default=False, action='store_true',
@@ -238,11 +242,14 @@ def main():
 
     if args.list:
         if args.test_set:
-            print(' '.join(get_langpairs_for_testset(args.test_set)))
+            langpairs = get_langpairs_for_testset(args.test_set)
+            for pair in langpairs:
+                fields = DATASETS[args.test_set].fieldnames(pair)
+                print(f'{pair}: {", ".join(fields)}')
         else:
             print('The available test sets are:')
             for testset in sorted(get_available_testsets()):
-                desc = DATASETS[testset].get('description', '').strip()
+                desc = DATASETS[testset].description.strip()
                 print(f'{testset:<30}: {desc}')
         sys.exit(0)
 
@@ -258,7 +265,7 @@ def main():
             if 'citation' not in DATASETS[test_set]:
                 sacrelogger.error(f'No citation found for {test_set}')
             else:
-                print(DATASETS[test_set]['citation'])
+                print(DATASETS[test_set].citation)
         sys.exit(0)
 
     if args.num_refs != 1 and (args.test_set is not None or len(args.refs) > 1):
diff --git a/sacrebleu/tokenizers/tokenizer_base.py b/sacrebleu/tokenizers/tokenizer_base.py
index e2a52e8b..faf3de4c 100644
--- a/sacrebleu/tokenizers/tokenizer_base.py
+++ b/sacrebleu/tokenizers/tokenizer_base.py
@@ -7,7 +7,7 @@ def signature(self):
 
         :return: signature string
         """
-        return 'none'
+        raise NotImplementedError()
 
     def __call__(self, line):
         """
@@ -16,4 +16,4 @@ def __call__(self, line):
         :param line: a segment to tokenize
         :return: the tokenized line
         """
-        return line
+        raise NotImplementedError()
diff --git a/sacrebleu/tokenizers/tokenizer_ko_mecab.py b/sacrebleu/tokenizers/tokenizer_ko_mecab.py
new file mode 100644
index 00000000..7d880428
--- /dev/null
+++ b/sacrebleu/tokenizers/tokenizer_ko_mecab.py
@@ -0,0 +1,52 @@
+from functools import lru_cache
+
+try:
+    import mecab_ko as MeCab
+    import mecab_ko_dic
+except ImportError:
+    # Don't fail until the tokenizer is actually used
+    MeCab = None
+
+from .tokenizer_base import BaseTokenizer
+
+FAIL_MESSAGE = """
+Korean tokenization requires extra dependencies, but you do not have them installed.
+Please install them like so.
+
+    pip install sacrebleu[ko]
+"""
+
+
+class TokenizerKoMecab(BaseTokenizer):
+    def __init__(self):
+        if MeCab is None:
+            raise RuntimeError(FAIL_MESSAGE)
+        self.tagger = MeCab.Tagger(mecab_ko_dic.MECAB_ARGS + " -Owakati")
+
+        # make sure the dictionary is mecab-ko-dic
+        d = self.tagger.dictionary_info()
+        assert d.size == 811795, \
+            "Please make sure to use the mecab-ko-dic for MeCab-ko"
+        # This asserts that no user dictionary has been loaded
+        assert d.next is None
+
+    @lru_cache(maxsize=2**16)
+    def __call__(self, line):
+        """
+        Tokenizes an Korean input line using MeCab-ko morphological analyzer.
+
+        :param line: a segment to tokenize
+        :return: the tokenized line
+        """
+        line = line.strip()
+        sentence = self.tagger.parse(line).strip()
+        return sentence
+
+    def signature(self):
+        """
+        Returns the MeCab-ko parameters.
+
+        :return: signature string
+        """
+        signature = self.tagger.version() + "-KO"
+        return 'ko-mecab-' + signature
diff --git a/sacrebleu/tokenizers/tokenizer_none.py b/sacrebleu/tokenizers/tokenizer_none.py
new file mode 100644
index 00000000..a204c000
--- /dev/null
+++ b/sacrebleu/tokenizers/tokenizer_none.py
@@ -0,0 +1,10 @@
+from .tokenizer_base import BaseTokenizer
+
+class NoneTokenizer(BaseTokenizer):
+    """Don't apply any tokenization. Not recommended!."""
+
+    def signature(self):
+        return 'none'
+
+    def __call__(self, line):
+        return line
diff --git a/sacrebleu/utils.py b/sacrebleu/utils.py
index 07692900..f8108b34 100644
--- a/sacrebleu/utils.py
+++ b/sacrebleu/utils.py
@@ -13,7 +13,6 @@
 from tabulate import tabulate
 import colorama
 
-from .dataset import DATASETS, SUBSETS, DOMAINS, COUNTRIES
 
 # Where to store downloaded test sets.
 # Define the environment variable $SACREBLEU, or use the default of ~/.sacrebleu.
@@ -274,63 +273,35 @@ def args_to_dict(args, prefix: str, strip_prefix: bool = False):
     return d
 
 
-def process_to_text(rawfile, txtfile, field: int = None):
-    """Processes raw files to plain text files. Can handle SGML, XML, TSV files, and plain text.
-    Called after downloading datasets.
-
-    :param rawfile: the input file (possibly SGML)
-    :param txtfile: the plaintext file
-    :param field: For TSV files, which field to extract.
-    """
-    def _clean(s):
-        """
-        Removes trailing and leading spaces and collapses multiple consecutive internal spaces to a single one.
-
-        :param s: The string.
-        :return: A cleaned-up string.
-        """
-        return re.sub(r'\s+', ' ', s.strip())
-
-    if not os.path.exists(txtfile) or os.path.getsize(txtfile) == 0:
-        sacrelogger.info(f"Processing {rawfile} to {txtfile}")
-        if rawfile.endswith('.sgm') or rawfile.endswith('.sgml'):
-            with smart_open(rawfile) as fin, smart_open(txtfile, 'wt') as fout:
-                for line in fin:
-                    if line.startswith('<seg '):
-                        print(_clean(re.sub(r'<seg.*?>(.*)</seg>.*?', '\\1', line)), file=fout)
-        # IWSLT
-        elif rawfile.endswith('.xml'):
-            with smart_open(rawfile) as fin, smart_open(txtfile, 'wt') as fout:
-                for line in fin:
-                    if line.startswith('<seg '):
-                        print(_clean(re.sub(r'<seg.*?>(.*)</seg>.*?', '\\1', line)), file=fout)
-        # MTNT
-        elif rawfile.endswith('.tsv'):
-            with smart_open(rawfile) as fin, smart_open(txtfile, 'wt') as fout:
-                for line in fin:
-                    print(line.rstrip().split('\t')[field], file=fout)
-        # PLAIN TEXT
-        else:
-            with smart_open(rawfile) as fin, smart_open(txtfile, 'wt') as fout:
-                for line in fin:
-                    print(line.rstrip(), file=fout)
-
-
-def print_test_set(test_set, langpair, side, origlang=None, subset=None):
+def print_test_set(test_set, langpair, fields, origlang=None, subset=None):
     """Prints to STDOUT the specified side of the specified test set.
 
     :param test_set: the test set to print
     :param langpair: the language pair
-    :param side: 'src' for source, 'ref' for reference
+    :param fields: the fields to print
     :param origlang: print only sentences with a given original language (2-char ISO639-1 code), "non-" prefix means negation
     :param subset: print only sentences whose document annotation matches a given regex
     """
-    if side == 'src':
-        files = [get_source_file(test_set, langpair)]
-    elif side == 'ref':
-        files = get_reference_files(test_set, langpair)
-    elif side == "both":
-        files = [get_source_file(test_set, langpair)] + get_reference_files(test_set, langpair)
+    if test_set not in DATASETS:
+        raise Exception(f"No such test set {test_set}")
+
+    fieldnames = DATASETS[test_set].fieldnames(langpair)
+    all_files = DATASETS[test_set].get_files(langpair)
+
+    if "all" in fields and len(fields) != 1:
+        sacrelogger.error("Cannot use --echo all with other fields")
+        sys.exit(1)
+    elif "all" in fields:
+        fields = fieldnames
+
+    files = []
+    for field in fields:
+        if field not in fieldnames:
+            sacrelogger.error(f"No such field {field} in test set {test_set} for language pair {langpair}.")
+            sacrelogger.error(f"available fields for {test_set}/{langpair}: {', '.join(fieldnames)}")
+            sys.exit(1)
+        index = fieldnames.index(field)
+        files.append(all_files[index])
 
     streams = [smart_open(file) for file in files]
     streams = filter_subset(streams, test_set, langpair, origlang, subset)
@@ -347,7 +318,10 @@ def get_source_file(test_set: str, langpair: str) -> str:
     :param langpair: The language pair (e.g., "de-en")
     :return: the path to the requested source file
     """
-    return get_files(test_set, langpair)[0]
+    if test_set not in DATASETS:
+        raise Exception(f"No such test set {test_set}")
+
+    return DATASETS[test_set].get_source_file(langpair)
 
 
 def get_reference_files(test_set: str, langpair: str) -> List[str]:
@@ -359,7 +333,9 @@ def get_reference_files(test_set: str, langpair: str) -> List[str]:
     :param langpair: The language pair (e.g., "de-en")
     :return: a list of one or more reference file paths
     """
-    return get_files(test_set, langpair)[1:]
+    if test_set not in DATASETS:
+        raise Exception(f"No such test set {test_set}")
+    return DATASETS[test_set].get_reference_files(langpair)
 
 
 def get_files(test_set, langpair) -> List[str]:
@@ -375,28 +351,7 @@ def get_files(test_set, langpair) -> List[str]:
 
     if test_set not in DATASETS:
         raise Exception(f"No such test set {test_set}")
-    if langpair not in DATASETS[test_set]:
-        raise Exception(f"No such language pair {test_set}/{langpair}")
-
-    cachedir = os.path.join(SACREBLEU_DIR, test_set)
-    source, target = langpair.split("-")
-
-    source_path = os.path.join(cachedir, f"{langpair}.{source}")
-
-    num_refs = len(DATASETS[test_set][langpair]) - 1
-    if num_refs == 1:
-        reference_paths = [os.path.join(cachedir, f"{langpair}.{target}")]
-    else:
-        reference_paths = [os.path.join(cachedir, f"{langpair}.{target}.{num}") for num in range(num_refs)]
-
-    all_files = [source_path] + reference_paths
-
-    for fname in all_files:
-        if not os.path.exists(fname):
-            download_test_set(test_set, langpair)
-            break
-
-    return all_files
+    return DATASETS[test_set].get_files(langpair)
 
 
 def extract_tarball(filepath, destdir):
@@ -411,21 +366,13 @@ def extract_tarball(filepath, destdir):
             zipfile.extractall(path=destdir)
 
 
-def check_md5sum(dest_path, expected_md5):
+def get_md5sum(dest_path):
     # Check md5sum
-    if expected_md5 is not None:
-        md5 = hashlib.md5()
-        with open(dest_path, 'rb') as infile:
-            for line in infile:
-                md5.update(line)
-            cur_md5 = md5.hexdigest()
-        if cur_md5 != expected_md5:
-            sacrelogger.error(f'Fatal: MD5 sum of downloaded file was incorrect (got {cur_md5}, expected {expected_md5}).')
-            sacrelogger.error(f'Please manually delete {tarball!r} and rerun the command.')
-            sacrelogger.error('If the problem persists, the tarball may have changed, in which case, please contact the SacreBLEU maintainer.')
-            sys.exit(1)
-        else:
-            sacrelogger.info(f'Checksum passed: {cur_md5}')
+    md5 = hashlib.md5()
+    with open(dest_path, 'rb') as infile:
+        for line in infile:
+            md5.update(line)
+    return md5.hexdigest()
 
 
 def download_file(source_path, dest_path, extract_to=None, expected_md5=None):
@@ -445,22 +392,31 @@ def download_file(source_path, dest_path, extract_to=None, expected_md5=None):
     outdir = os.path.dirname(dest_path)
     os.makedirs(outdir, exist_ok=True)
 
-    lockfile = os.path.join(outdir, f'{os.path.basename(dest_path)}.lock')
-    with portalocker.Lock(lockfile, 'w', timeout=60):
-        if os.path.exists(dest_path):
-            check_md5sum(dest_path, expected_md5)
-        else:
+    # Make sure to open in mode "a"
+    lockfile = f"{dest_path}.lock"
+    with portalocker.Lock(lockfile, timeout=60):
+
+        if not os.path.exists(dest_path) or os.path.getsize(dest_path) == 0:
+
             sacrelogger.info(f"Downloading {source_path} to {dest_path}")
+            md5 = hashlib.md5()
+
             try:
                 with urllib.request.urlopen(source_path) as f, open(dest_path, 'wb') as out:
                     out.write(f.read())
             except ssl.SSLError:
-                sacrelogger.warning('An SSL error was encountered in downloading the files. If you\'re on a Mac, '
+                sacrelogger.error('An SSL error was encountered in downloading the files. If you\'re on a Mac, '
                                     'you may need to run the "Install Certificates.command" file located in the '
                                     '"Python 3" folder, often found under /Applications')
                 sys.exit(1)
 
-            check_md5sum(dest_path, expected_md5)
+            if expected_md5 is not None:
+                cur_md5 = get_md5sum(dest_path)
+                if cur_md5 != expected_md5:
+                    sacrelogger.error(f'Fatal: MD5 sum of downloaded file was incorrect (got {cur_md5}, expected {expected_md5}).')
+                    sacrelogger.error(f'Please manually delete {dest_path!r} and rerun the command.')
+                    sacrelogger.error(f'If the problem persists, the tarball may have changed, in which case, please contact the SacreBLEU maintainer.')
+                    sys.exit(1)
 
             # Extract the tarball
             if extract_to is not None:
@@ -476,53 +432,16 @@ def download_test_set(test_set, langpair=None):
     """
     if test_set not in DATASETS:
         raise Exception(f"No such test set {test_set}")
-
-    outdir = os.path.join(SACREBLEU_DIR, test_set)
-    os.makedirs(outdir, exist_ok=True)
-
-    expected_checksums = DATASETS[test_set].get('md5', [None] * len(DATASETS[test_set]))
-    for dataset, expected_md5 in zip(DATASETS[test_set]['data'], expected_checksums):
-        tarball = os.path.join(outdir, os.path.basename(dataset))
-        rawdir = os.path.join(outdir, 'raw')
-
-        download_file(dataset, tarball, extract_to=rawdir, expected_md5=expected_md5)
-
-    file_paths = []
-
-    # Process the files into plain text
-    languages = get_langpairs_for_testset(test_set) if langpair is None else [langpair]
-    for pair in languages:
-        src, tgt = pair.split('-')
-        rawfile = DATASETS[test_set][pair][0]
-        field = None  # used for TSV files
-        if rawfile.endswith('.tsv'):
-            field, rawfile = rawfile.split(':', maxsplit=1)
-            field = int(field)
-        rawpath = os.path.join(rawdir, rawfile)
-        outpath = os.path.join(outdir, f'{pair}.{src}')
-        process_to_text(rawpath, outpath, field=field)
-        file_paths.append(outpath)
-
-        refs = DATASETS[test_set][pair][1:]
-        for i, ref in enumerate(refs):
-            field = None
-            if ref.endswith('.tsv'):
-                field, ref = ref.split(':', maxsplit=1)
-                field = int(field)
-            rawpath = os.path.join(rawdir, ref)
-            if len(refs) >= 2:
-                outpath = os.path.join(outdir, f'{pair}.{tgt}.{i}')
-            else:
-                outpath = os.path.join(outdir, f'{pair}.{tgt}')
-            process_to_text(rawpath, outpath, field=field)
-            file_paths.append(outpath)
-
+    dataset = DATASETS[test_set]
+    file_paths = dataset.get_files(langpair)
     return file_paths
 
 
 def get_langpairs_for_testset(testset: str) -> List[str]:
     """Return a list of language pairs for a given test set."""
-    return list(filter(lambda x: re.match(r'\w\w\-\w\w', x), DATASETS.get(testset, {}).keys()))
+    if testset not in DATASETS:
+        return []
+    return list(DATASETS[testset].langpairs.keys())
 
 
 def get_available_testsets() -> List[str]:
@@ -537,7 +456,8 @@ def get_available_origlangs(test_sets, langpair) -> List[str]:
 
     origlangs = set()
     for test_set in test_sets.split(','):
-        rawfile = os.path.join(SACREBLEU_DIR, test_set, 'raw', DATASETS[test_set][langpair][0])
+        dataset = DATASETS[test_set]
+        rawfile = os.path.join(SACREBLEU_DIR, test_set, 'raw', dataset.langpairs[langpair][0])
         if rawfile.endswith('.sgm'):
             with smart_open(rawfile) as fin:
                 for line in fin:
@@ -560,7 +480,8 @@ def filter_subset(systems, test_sets, langpair, origlang, subset=None):
     indices_to_keep = []
 
     for test_set in test_sets.split(','):
-        rawfile = os.path.join(SACREBLEU_DIR, test_set, 'raw', DATASETS[test_set][langpair][0])
+        dataset = DATASETS[test_set]
+        rawfile = os.path.join(SACREBLEU_DIR, test_set, 'raw', dataset.langpairs[langpair][0])
         if not rawfile.endswith('.sgm'):
             raise Exception(f'--origlang and --subset supports only *.sgm files, not {rawfile!r}')
         if subset is not None:
@@ -631,3 +552,6 @@ def print_subset_results(metrics, full_system, full_refs, args):
         key = Color.format(f'{key:<{max_left_width}}', 'yellow')
         for n_system, score in scores:
             print(f'{key}: sentences={n_system:<6} {score.name:<{max_metric_width}} = {score.score:.{w}f}')
+
+# import at the end to avoid circular import
+from .dataset import DATASETS, SUBSETS, DOMAINS, COUNTRIES
diff --git a/setup.py b/setup.py
index 6c440ec4..40e5bb8b 100755
--- a/setup.py
+++ b/setup.py
@@ -145,13 +145,14 @@ def get_long_description():
     # your project is installed. For an analysis of "install_requires" vs pip's
     # requirements files see:
     # https://packaging.python.org/en/latest/requirements.html
-    install_requires=['portalocker', 'regex', 'tabulate>=0.8.9', 'numpy>=1.17', 'colorama'],
+    install_requires=['portalocker', 'regex', 'tabulate>=0.8.9', 'numpy>=1.17', 'colorama', 'lxml'],
 
     # List additional groups of dependencies here (e.g. development
     # dependencies). You can install these using the following syntax,
     # for example:
     # $ pip install -e .[dev,test]
-    extras_require={'ja': ['mecab-python3==1.0.5', 'ipadic>=1.0,<2.0']},
+    extras_require={'ja': ['mecab-python3==1.0.5', 'ipadic>=1.0,<2.0'],
+                    'ko': ['mecab-ko==1.0.0', 'mecab-ko-dic>=1.0,<2.0']},
 
     # To provide executable scripts, use entry points in preference to the
     # "scripts" keyword. Entry points provide cross-platform support and allow


Test patch:
diff --git a/test.sh b/test.sh
index 81d2b494..ee1657f6 100755
--- a/test.sh
+++ b/test.sh
@@ -35,12 +35,15 @@ trap "rm -f .tmp* data/.tmp*" EXIT INT TERM
 
 # For Travis CI to work on Windows/Mac OS X
 if [[ "$OSTYPE" == "linux-gnu"* ]]; then
-  CMD="python -m sacrebleu"
+  PYTHON="python"
+  CMD="$PYTHON -m sacrebleu"
 elif [[ "$OSTYPE" == "msys" ]]; then
-  CMD="python -m sacrebleu"
+  PYTHON="python"
+  CMD="$PYTHON -m sacrebleu"
 elif [[ "$OSTYPE" == "darwin"* ]]; then
   # OS X ships python -> python2 by default, be explicit
-  CMD="python3 -m sacrebleu"
+  PYTHON=python3
+  CMD="$PYTHON -m sacrebleu"
 fi
 
 export SACREBLEU=$(pwd)/.sacrebleu
@@ -54,6 +57,7 @@ SKIP_INITIAL=${SKIP_INITIAL:-}
 SKIP_CHRF=${SKIP_CHRF:-}
 SKIP_TER=${SKIP_TER:-}
 SKIP_MECAB=${SKIP_MECAB:-}
+SKIP_MECAB_KO=${SKIP_MECAB_KO:-}
 SKIP_MTEVAL13=${SKIP_MTEVAL13:-}
 SKIP_MTEVAL14=${SKIP_MTEVAL14:-}
 
@@ -100,6 +104,14 @@ if [[ ! -d en-ja-translation-example-master ]]; then
    echo "Downloading and unpacking English-Japanese test data..."
    wget -q https://github.com/MorinoseiMorizo/en-ja-translation-example/archive/master.zip
    unzip master.zip
+   rm master.zip
+fi
+
+if [[ ! -d en-ko-translation-example-master ]]; then
+   echo "Downloading and unpacking English-Korean test data..."
+   wget -q https://github.com/NoUnique/en-ko-translation-example/archive/master.zip
+   unzip master.zip
+   rm master.zip
 fi
 
 if [ -z $SKIP_INITIAL ]; then
@@ -128,15 +140,16 @@ if [ -z $SKIP_INITIAL ]; then
   # Test echoing of source, reference, and both
   # Replace \r\n with \n for Windows compatibility
   ${CMD} -t wmt17/ms -l zh-en --echo src | tr -d '\015' > .tmp.echo
-  diff .tmp.echo $SACREBLEU/wmt17/ms/zh-en.zh
+  diff .tmp.echo $SACREBLEU/wmt17/ms/wmt17_ms.zh-en.src
   if [[ $? -ne 0 ]]; then
       echo "Source echo failed."
       exit 1
   fi
-  ${CMD} -t wmt17/ms -l zh-en --echo ref | tr -d '\015' | cut -f3 > .tmp.echo
-  diff .tmp.echo $SACREBLEU/wmt17/ms/zh-en.en.2
+  ${CMD} -t wmt17/ms -l zh-en --echo ref:2 | tr -d '\015' > .tmp.echo
+  diff .tmp.echo $SACREBLEU/wmt17/ms/wmt17_ms.zh-en.ref-2 > /dev/null
   if [[ $? -ne 0 ]]; then
       echo "Source echo failed."
+      pwd
       exit 1
   fi
 
@@ -273,7 +286,7 @@ if [ -z $SKIP_CHRF ]; then
           # Test chrF
           score=$(cat $txt | ${CMD} -w 4 -t wmt17 -l $source-$target -b --metrics chrf --chrf-eps-smoothing)
           expected_score=${CHRF[$name]}
-          echo "import sys; sys.exit(1 if abs(${score}-${expected_score}) > 1e-6 else 0)" | python
+          echo "import sys; sys.exit(1 if abs(${score}-${expected_score}) > 1e-6 else 0)" | $PYTHON
 
           if [[ $? -eq 1 ]]; then
               echo "FAILED chrF test $pair/$sys (wanted $expected_score got $score)"
@@ -286,7 +299,7 @@ if [ -z $SKIP_CHRF ]; then
           score=$(cat $txt | ${CMD} -w 4 -t wmt17 -l $source-$target -b --metrics chrf --chrf-word-order 2 --chrf-eps-smoothing)
           expected_score=${CHRFPP[$name]}
 
-          echo "import sys; sys.exit(1 if abs(${score}-${expected_score}) > 1e-6 else 0)" | python
+          echo "import sys; sys.exit(1 if abs(${score}-${expected_score}) > 1e-6 else 0)" | $PYTHON
 
           if [[ $? -eq 1 ]]; then
               echo "FAILED chrF++ test $pair/$sys (wanted $expected_score got $score)"
@@ -459,6 +472,7 @@ declare -A MTEVAL=( ["newstest2017.PJATK.4760.cs-en.sgm"]=23.15
                     ["newstest2017.uedin-nmt.5112.zh-en.sgm"]=25.7
                     ["newstest2017.xmunmt.5160.zh-en.sgm"]=26.0
                     ["kyoto-test"]=14.48
+                    ["flores101.devtest.en-ko"]=33.18
                   )
 
 if [ -z $SKIP_MTEVAL13 ]; then
@@ -480,7 +494,7 @@ if [ -z $SKIP_MTEVAL13 ]; then
           # mteval=$(echo "print($bleu1 * 100)" | python)
           score=$(cat $txt | ${CMD} -w 2 -t wmt17 -l $source-$target -b)
 
-          echo "import sys; sys.exit(1 if abs($score-${MTEVAL[$name]}) > 0.01 else 0)" | python
+          echo "import sys; sys.exit(1 if abs($score-${MTEVAL[$name]}) > 0.01 else 0)" | $PYTHON
 
           if [[ $? -eq 1 ]]; then
               echo "FAILED test $pair/$sys (wanted ${MTEVAL[$name]} got $score)"
@@ -536,7 +550,7 @@ if [ -z $SKIP_MTEVAL14 ]; then
 
           score=$(cat $txt | ${CMD} -w 2 -t wmt17 -l $source-$target -b --tokenize intl)
 
-          echo "import sys; sys.exit(1 if abs($score-${MTEVAL14[$name]}) > 0.01 else 0)" | python
+          echo "import sys; sys.exit(1 if abs($score-${MTEVAL14[$name]}) > 0.01 else 0)" | $PYTHON
 
           if [[ $? -eq 1 ]]; then
               echo "FAILED test $pair/$sys (wanted ${MTEVAL14[$name]} got $score)"
@@ -659,7 +673,7 @@ if [ -z $SKIP_TER ]; then
 
           expected_score="${TER[$name]}"
 
-          echo "import sys; sys.exit(1 if abs(0.01 * ${score}-${expected_score}) > 0.01 else 0)" | python
+          echo "import sys; sys.exit(1 if abs(0.01 * ${score}-${expected_score}) > 0.01 else 0)" | $PYTHON
 
           if [[ $? -eq 1 ]]; then
               echo "FAILED test $pair/$sys (wanted $expected_score got $score)"
@@ -691,6 +705,37 @@ if [[ -z $SKIP_MECAB ]]; then
         ref=$(dirname $txt)/$(basename $txt .hyp.$target).ref.$target
         score=$(cat $txt | ${CMD} -w 2 -l $source-$target -b $ref)
 
+        echo "import sys; sys.exit(1 if abs($score-${MTEVAL[$name]}) > 0.01 else 0)" | $PYTHON
+
+        if [[ $? -eq 1 ]]; then
+            echo "FAILED test $pair/$sys (wanted ${MTEVAL[$name]} got $score)"
+            exit 1
+        fi
+        echo "Passed $source-$target $sys mteval-v13a.pl: ${MTEVAL[$name]} sacreBLEU: $score"
+
+        let i++
+    done
+fi
+
+#############
+# Mecab-ko tests
+#############
+if [[ -z $SKIP_MECAB_KO ]]; then
+  echo "-----------------------"
+  echo "Testing Mecab-ko tokenizer"
+  echo "-----------------------"
+    pair="en-ko"
+    source=$(echo $pair | cut -d- -f1)
+    target=$(echo $pair | cut -d- -f2)
+    for txt in en-ko-translation-example-master/*.hyp.$target; do
+        name=$(basename $txt .hyp.$target)
+
+        if [[ ! -z $limit_test && $limit_test != $name ]]; then continue; fi
+
+        sys=$(basename $txt .hyp.$target)
+        ref=$(dirname $txt)/$(basename $txt .hyp.$target).ref.$target
+        score=$(cat $txt | ${CMD} -w 2 -l $source-$target -b $ref)
+
         echo "import sys; sys.exit(1 if abs($score-${MTEVAL[$name]}) > 0.01 else 0)" | python
 
         if [[ $? -eq 1 ]]; then
diff --git a/test/test_api.py b/test/test_api.py
index d9bb30dc..d571e072 100644
--- a/test/test_api.py
+++ b/test/test_api.py
@@ -61,7 +61,7 @@ def test_api_get_langpairs_for_testset():
     for testset in DATASETS.keys():
         available = get_langpairs_for_testset(testset)
         assert type(available) is list
-        for langpair in DATASETS[testset].keys():
+        for langpair in DATASETS[testset].langpairs.keys():
             # skip non-language keys
             if "-" not in langpair:
                 assert langpair not in available
diff --git a/test/test_dataset.py b/test/test_dataset.py
new file mode 100644
index 00000000..d2dae073
--- /dev/null
+++ b/test/test_dataset.py
@@ -0,0 +1,78 @@
+import os
+import shutil
+import random
+
+import sacrebleu.dataset as dataset
+from sacrebleu.utils import smart_open
+
+
+def test_maybe_download():
+    """
+    Test the maybe_download function in Dataset class.
+
+    Check a few random datasets for downloading and correct file placement.
+    """
+    # ensure all file have been downloaded
+    selected_datasets = random.choices(list(dataset.DATASETS.values()), k=10)
+    for ds in selected_datasets:
+        shutil.rmtree(ds._rawdir, ignore_errors=True)
+        ds.maybe_download()
+
+        all_files = os.listdir(ds._rawdir)
+        for url in ds.data:
+            filename = ds._get_tarball_filename(url)
+            assert filename in all_files
+            filepath = os.path.join(ds._rawdir, filename)
+            assert os.path.getsize(filepath) > 0
+
+
+def test_process_to_text():
+    """
+    Test the function `process_to_text` in Dataset class.
+
+    Ensure each field of specified language pair have the same length.
+    """
+    selected_datasets = random.choices(list(dataset.DATASETS.values()), k=10)
+    for ds in selected_datasets:
+        if os.path.exists(ds._outdir):
+            for filename in os.listdir(ds._outdir):
+                filepath = os.path.join(ds._outdir, filename)
+                if os.path.isfile(filepath):
+                    os.remove(filepath)
+
+        ds.process_to_text()
+
+        for pair in ds.langpairs:
+            all_files = ds.get_files(pair)
+
+            # count the number of lines in each file
+            num_lines = [sum(1 for _ in smart_open(f)) for f in all_files]
+
+            # ensure no empty file
+            assert num_lines[0] > 0
+
+            # assert each file has the same length
+            assert all(x == num_lines[0] for x in num_lines)
+
+
+def test_get_files_and_fieldnames():
+    """
+    Test the functions `get_files` and `fieldnames` in Dataset class.
+
+    Ensure the length of the returned list is correct.
+    `get_files()` should return the same number of items as `fieldnames()`.
+    """
+    for ds in dataset.DATASETS.values():
+        for pair in ds.langpairs:
+            assert len(ds.get_files(pair)) == len(ds.fieldnames(pair))
+
+
+def test_source_and_references():
+    """
+    Test the functions `source` and `references` in Dataset class.
+
+    Ensure the length of source and references are equal.
+    """
+    for ds in dataset.DATASETS.values():
+        for pair in ds.langpairs:
+            assert len(list(ds.source(pair))) == len(list(ds.references(pair)))