Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper asserts the importance of translation technologies in crisis situations and describes the TICO-19 benchmark for validating and testing translation systems for low-resourced languages. Also, other additional resources such as translation memories and in-domain monolingual data are presented. The paper concludes with initial benchmarked results using publicly available pre-trained models.

I think the work is well-aligned with the workshop's objective. I would like to applaud the authors for their efforts to construct such a benchmark with the under-served languages in mind.

I do have a few questions:

1. What does the 95% rate correspond to? Does it correspond to the direct assessment scheme employed in WMT?
Were there any domain-specific evaluation guidelines?

2. If I understood the paper correctly, the additional resources in Section 5 are not used at all to train/fine-tune the "Our OPUS" and "Our TED" models? Is this correct?
review_title: Great initiative that meets the purpose of the workshop

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
This work described a translation system TICO-19 based on multiple sources including English PubMed and Wikipedia. English documents were translated into 35 languages with terminologies provided by Facebook and Google, where the OPUS-MT system was employed.

**Comments**
1. In the Quality Assurance subsection, It is not clear how the first editing is performed? How do you solve the disagreements? Can you provide examples?

2. In the second round QA process, how the 95% rate was achieved? How do you evaluate?

3. The MT is pre-trained on OPUS parallel data. It is not clear how do you use the resources in section 5 to adjust the pre-trained system.

4. Can you please clarify the structure of the pipeline system? After collecting the data and specifying the target language, how do Facebook, Google terminology datasets being used in the MT system?
review_title: TICO-19: the Translation Initiative for COvid-19 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors describe the TICO dataset, a development and testing dataset for multi-lingual medical translation. They describe in great detail the creation of this dataset, and the QA measures they undertook. I am particularly a fan of Table 8 which includes specific measures and details about which dataset underwent which parts of their QA process, although I note there appears to be an extra empty aggregate row.

The authors provide a great overview of the varying languages, grouping them for relative importance in the translation effort, and making a qualitative assessment of the resources available for each of them.

Overall I think that this is a high quality development and testing set. I would be happy to see this paper accepted, although I think there are some easily tackleable areas for improvement. My concerns are below:

It's not clear to me whether or the not the authors intend on releasing their systems (licensing of the data?). As is, I would assume not, which would be unfortunate but not a reason to stop publication. 

I disagree with the relative downweighting of the importance of the parallel data in the paper - the authors should document the sizes of the data they found, even if small.  Licensing concerns are relevant for others using the data, but a proper description of it, including sizes and genres, belongs in this paper.

I am concerned that the relative importance of the PubMed data in the evaluation datasets, by a small number of documents providing a large number of sentences, will primarily reward systems for a somewhat random game of vocabulary whack-a-mole instead of rewarding the ability to produce a high quality translation over a diverse set of inputs. While adjusting this is likely impossible at this point due to expense or annotation effort, I would like to see an explicit treatment of this potential issue.

Wherever possible I would request that the authors create wayback or archived versions of the links in the paper and to prefer full links over shortened ones instead of relying on bit.ly's promise to be around forever. This will help ensure that their work remains accessible from a long term perspective.
review_title: Great resource