Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes three contributions: (1) a new multilingual parallel corpus that covers 1,477 languages, with text from the Bible -- for the region of interest in this paper (Cameroon) this means 22 languages instead of 18 with JW300; (2) a method for determining similarity of languages based on LM scores; (3) a series of experiments that evaluate the utility of adding a third language to a pair of interest based on either the similarity metric proposed by (2) or based on historical data, for use in a multilingual MT system. The authors show large improvements by adding a third language based on historical language similarity (always outperforms random) or their language-model-derived similarity (often outperforms random).

Working on MT for African languages is important, and I’d definitely like to see more of it. Unfortunately, I think this paper suffers substantially from insufficient replication details. Even if those details were present, I don’t think it is necessarily sufficiently novel for ICLR.

Regarding replication, the paper relies on three technologies: language modeling for the language similarity calculation, CLM and MLM pretraining, and multilingual MT. Crucial details are missing for all three of these. For the language modeling and for CLM and MLM pretraining, it isn’t clear what the monolingual data source is. Without being specified, it’s implied that it’s just one side of the parallel corpora - but why would pretraining help in that setting if it’s not adding data? In general, the data should be discussed with more precision. It would be really useful to talk about sizes of datasets (if the different languages have differently sized datasets). 
For multilingual MT, it isn’t clear what flavor of multilingual MT is being used: is it a tagged model? Where are the tags (source versus target versus embedding)? I understand that the authors have provided code, but the paper should stand alone, and I wasn’t able to easily answer these questions after spending 15 minutes with the code.

Regarding novelty, I think the paper would need to flesh out either contributions (2) or (3) to really find a home at a venue like ICLR. (2) is an interesting idea, but right now we have one extrinsic test of the idea - I think a fully fleshed out version would need to compare to baselines and also provide intrinsic evaluations that compare to other “ground truth” definitions of similarity, such as the historical definition used in this paper. (3) would require looking into why we’re seeing such huge jumps just from adding another language. Are there disparities in dataset sizes that aren’t mentioned in the paper? If adding another language at most doubles the amount of parallel data, I don’t see why that should reflect a boost from 12.6 to 28.8 BLEU. There might be something really interesting going on here, but it would need to be explored much more carefully.

Smaller points:

When citing Eberhard, I would include “M.” with the first name, not the last name.

The proposed similarity metric is based on an equation normalizing a perplexity difference according to a base perplexity. But I don’t think this can be treated as a similarity, as lower is better in perplexity. So if I have language A that receives a perplexity of 10 according to LM0, and another language B that receives a perplexity of 3 (and thus much higher average probability per token), and LM0 assigns its own language a perplexity of 2, then we’ll get scores of (10-2)/2 = 4 for A and (3-2)/2 = 0.5 for B, making A look more similar. I think you probably intended to use average probability or average log probability here, or have the equation calculate a difference.
review_title: Good work on an understudied area, but light on details for replication.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers translation between African languages. Overall, I think this is a great effort, I think it's great that the authors are tackling this important problem.

However, the field of multilingual machine translation is a very well-researched field, and it seems that the authors have developed their methodology largely independent of the literature in this field. In fact, there are already existing well-researched methods on many of the topics presented in this paper. To give just a few examples:

* *Pre-training for low-resource translation:* Liu, Yinhan, et al. "Multilingual denoising pre-training for neural machine translation." arXiv preprint arXiv:2001.08210 (2020).
* *Leveraging linguistic similarities:* Lin, Yu-Hsiang, et al. "Choosing transfer languages for cross-lingual learning." arXiv preprint arXiv:1905.12688 (2019).
* *Translation between low-resource language pairs:* Chen, Yun, et al. "A teacher-student framework for zero-resource neural machine translation." arXiv preprint arXiv:1705.00753 (2017).

I would suggest that the authors read these papers, and maybe other papers that cite them. Also, perhaps read papers on the ACL Anthology (https://www.aclweb.org/anthology/) from prominent conferences such as ACL, EMNLP, NAACL that contain the keywords "multilingual" and "translation" to get a better idea of the state of the art in the field. There are lots of methods that people have developed, and I think that they could be effectively applied to the very important problem at hand here!
review_title: Laudable effort! But little awareness of previous work

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper motivates clearly the need for research in machine translation of underresourced (and thus underresearched) African languages, and proposes ways to aid the training of MT systems using data from related languages. The main contribution is that two ways (and a random baseline) to select a langauge to add to the training of an MT model are evaluated (one based on linguists' clustering, one based on probabilities assigned through a language model of the target langauge) on pairs drawn from a set of three langauges, choosing new langauges from a set of 13.

#### Strengths/what I loved:
- The paper performs a very clearly motivated and well set-up experiment with appropriate analysis.
- I loved to see a focus on datasets for underresourced langauges and the paper's repeated focus on whether or not previous work was reproducible and how this paper tries to be as reproducible as possible (even if that is not possible with Bible data for legal reasons).
- I always like to see attention to detail and explanation as i the "Due to the multi-task nature..."-paragraph on page 7!
- The observation that adding weakly related languages *hurt* was very interesting to see, thank you for including it!

#### Criticism/weaknesses:
- While the overall setup (choose a language to add using either linguistic knowledge or automatic metrics) is solid and interesting, the setup and contributions promised on pg. 2 make it sound especially like you are proposing a way to infer linguistic relations between languages ("a method for aggregating languages based on [a number of features]").
- The related work section is rather unfocused: it might help to make clear at the beginning of each paragraph how all the work cited in that paragraph relates to the present paper (e.g., "There are a number of papers on translation of African langauges, covering English to Setswana (Abbott & Martinus (2018); Transformers on the Autshumato dataset (Groenewald & Fourie (2009) reaching BLEU 33.53), English to Afrikaans..." or something along these lines. The division between the first and the second paragraph isn't quite clear to me and I am not sure what the third paragraph is on: transfer in MT?
- Not only training, but unfortunately also validation is performed only on Bible data and only on a cluster of 3 languages. The first is an issue because while I understand that Bibles are the easiest data to come by and that for training one takes what one can get, any results that stay within this Bible domain are unlikely to transfer well to real-world settings, making me doubt the practical use of the paper as-is. The latter exacerbates the issue, because essentially as a paper motivated by poor MT performance or nonexistence on many African languages it fails to convince me that it is indeed solving that task and not one of Bible translation, and as a paper that talks about linguistic features in general, the extremely narrow focus on a cluster of literally only 3 languages seems unnecessarily restrictive if we are already settling for Bible data. (Note that I think it is fair to say that these limitations were necessary because individual experiments are too costly to run more than this very barebones proof-of-concept, but that claim should be made and substantiated.)
- Experiments are not tested for statistical significance or otherwise qualified through a sense of the variance that's inherent in these results. This is especially unfortunate for the "Random" baseline, which given the small set of candidates will produce rather unreliable estimates when only tested on 1 sample (please let me know if I misunderstood!)

#### Questions:
- Point 2 of the Motivation section claims that language barriers aid the spread of misinformation... that's not quite clear to me: shouldn't any barrier impede information flow and in fact easy machine translation help the spread? Unless it is correcting counter-information that you want to advocate for...
- Page 5 says you used an English tokenizer for all languages. Just to clarify: this is tokenizing before applying BPE, right? Did you try to go without tokenization at all, i.e., using BPE as a standalone tokenizer?
- What is the point of the last two paragraphs of section 5, i.e., talking about clustering? From my understanding you don't use that for your experiments, as there you only choose the 1 nearest neighbor---and if you did want to build larger clusters, the greedy approach sketched in here would be a needlessly restricted choice: there are a number of better agglomerative clustering algorithms out there!
- The paper does cite the Masakhane project preprint from March 2020 (though with a broken arXiv identifier in the references!), but I would be very curious to see how it relates to the newer October 2020 preprint that contains a lot more content. (Of course, I know that this paper is far too recent to expect or even ask you to refer to it in the preprint! I would just personally be curious and think it might be a nice thing to add for a camera-ready.)


#### Typos and other small things:
- Probably due to my unfamiliarity, I wasn't 100% clear on what you meant by ``vernacular'' throughout, it might be nice to give a quick definition.
- Some small grammar errors throughout, for example "as follow" instead of "as follows" in a few places.
- Citations should be in parentheses when they are not constituents in a sentence and bare when they are (as in the beginning of the second paragraph on page 3). The citation of Lample and Conneau on page 2 somehow ended up with double parentheses...
- I would have liked to see references for section 2, but I suppose that is not really the scope or point of the papers. I also guess that the third point implies that colonial languages are learned instead of vernaculars? That might get expanded a little, too.
- The JW300 dataset should be properly cited (Agić and Vulić, 2019: https://www.aclweb.org/anthology/P19-1310/ ).
- The description of MLM on page 4 says "we keep them unchanged 10%"---this sounds like this is a deviation you are introducing.
- The entire first half of page 6 could be condensed into a much smaller and more concise paragraph, namely a sentence like the one that then starts with "At a paragraph level", or to more clearly separate motivation from actual definition (which is nice and simple)
- I would separate the top and bottom half of Table 1 more clearly, or better yet, make them separate tables altogether, given how they don't even both have the same set of columns.
- I fear the example given in the first paragraph of page 7 is a poor example: I see no reason a priori why learning a different unrelated language should *hurt* other languages I know. Then again, as a non-native speaker of English, my own native langauge certainly has taken the back seat in my head... either way it might be good to elaborate a little on this idea or just leave it out.

Looking forward to discuss!
review_title: Small contribution on transfer in MT for underresearched African languages

final_score: rating: 3: Clear rejection
review_content: This paper explores the problem of machine translation for African languages. The authors show that trilingual models outperform bilingual models for a set of languages spoken in Cameroon, and explore two strategies to find the best auxiliary language for a given translation pair: a manual one based on their linguistic relationship, and an automatic one based on language modelling.

I agree with the authors that African languages have been underrepresented in NLP research in general and machine translation in particular. In that regard, I think that their motivation is truly admirable, and I would like to encourage them to keep working on this direction.

Unfortunately, I think that the work itself does not meet the standards of the conference in its current form. While the focus on African languages is new and interesting, I find that the paper has little substance beyond that. According to the authors the paper makes 3 main contributions, which I feel are too narrow or otherwise questionable as discussed next:

1) "Our first contribution is the creation and public release of a new dataset." The dataset in question is a parallel version of the Bible. There have already been some efforts to extract parallel corpora from the Bible, which are not even discussed in the paper. For instance, Mayer & Cysouw (2014) [1] report covering 830 languages, which are likely to include the ones explored in the paper. Moreover, the authors extracted the corpus from an existing website with Bible versions in >1000 languages, so this was just a scraping effort. In connection to that, I am not sure if the authors considered potential copyright restrictions when releasing their corpus.

2) "Our second contribution is the proposal of a new metric to evaluate similarity between languages". The proposed method measures the similarity between two languages, L1 and L2, by applying a language model trained on L1 to L2. This looks like a rather simplistic approach and, in the lack of any other baseline, it is hard to get a sense of how good it is. Also, I am not familiar with this topic, but there is certainly some work on automatically identifying similar languages. For instance, [2] shows that the degree of overlap in the BPE vocabulary of different languages is already a good indicator of the their linguistic proximity.

3) "Our third contribution is the set of empirical evidences that doing multi-task learning on multiple similar languages generally improves the performances on individual pairs." There were already in-depth studies showing that multilingual training is helpful for low-resource machine translation, see for instance [3]. Unfortunately, this is not even cited in the paper.

In connection to the last point, the authors limit themselves to trilingual models and the main focus of their work is on identifying the most suitable auxiliary language. An obvious baseline that is missing in the paper is to train a multilingual model on the combination of all languages.

In addition, the models used by the author are usually trained at a large scale, and might not work well with very small datasets like the Bible. Even a phrase-based statistical machine translation system might work better in this setup, and I overall miss better baselines.

[1] http://www.lrec-conf.org/proceedings/lrec2014/pdf/220_Paper.pdf
[2] https://aiide.org/ojs/index.php/AAAI/article/view/4677
[3] https://arxiv.org/abs/1907.05019
review_title: Review

Meta-review: This paper introduces a new multilingual parallel Bible dataset for African languages, a new method for determining similarities between languages, and a collection of experiments to evaluate methods for choosing an additional language based on (a) similarity and (b) language history to include in a multilingual MT system. Results show that strategic inclusion of an additional language can substantially improve BLEU. Reviewers universally agree that progress on MT for African languages is a very important goal. However, reviewers pointed to several major concerns with the current draft: (1) lack of sufficient detail for replicating experiments, (2) missing analysis to interpret why experimental gains are so large, and (3) missing discussion and comparison with already existing methods in multilingual MT (e.g. multilingual training for low-resource languages). I agree with reviewers that the paper is not ready for acceptance in its current form, but encourage re-submission, possibly at an NLP conference. 

Decision: Reject