SASDP: Selective Asynchronous Sharded Data Parallel Training for Massive Neural Machine Translation
Authors: Fei Yuan, Yinquan Lu, Lingpeng Kong, Lei Li, Jingjing Xu


Reviewer paper summaries: This paper aims to train a large multi-lingual machine translation on massive language pairs. First, this work curates a 435-language dataset and evenly divides them by grouping the languages. Then,  an asynchronous training scheme is proposed. In the scheme, a language-specific encoder is always kept in each local computation node, but such an encoder is not part of the final product. A shared encoder-decoder is also trained, and they are updated in an asynchronous fashion. Empirically, the authors claim that the proposed method is 16.2 times faster than an M2m model, and is 2.2 higher in BLUE compared with an NLLB model.
Summary of strengths: 1. Empirically, this work curates a 435-language dataset and proposes an asynchronous training scheme, which yields good results in terms of both translation quality and training cost.
2. The authors show efforts to include various multi-lingual baselines, which are quite expensive to run.
Summary of weaknesses: 1. The claim of being 16.2 times faster is not fair. The number of parameters in the two systems is not the same.
2. There are two major components of this work that may affect the translation quality: 1) a local but not used language-specific encoder, 2) a non-overlap data distribution scheme. However, there is a lack of direct ablation studies. 
    - The Single-FT could be a baseline for studying the language-specific encoder, but it is not clear if it is obtained in a controlled setting; i.e., they should have the same data shards and the same asynchronous training.
    - For the second component, it is unclear if isolating the languages is helpful. A baseline, which simply evenly distributes the data, should be considered (just like the "Random" method in Table 4, but "Random" is not compared with the main method).
3. The subsection starting from L535 studies the importance of having balanced shards. However, the process of having language groups decided by similarity is unnecessary. For example, the group of languages can be simply randomly selected, and they are not likely to be balanced. 
4. Given the intuition described in L040, the grouping of the languages should be based on their linguistic features (e.g., the "Similarity" approach in Table 4) rather than their amount in the training set (what this paper did). Understandably, if the "Similarity" approach is directly applied, the shared will not be balanced. However, this can be easily solved by having more clusters (larger than 8) and then evenly fitting the clusters into 8 bins.
5. In Table 1, I don't think counting the SASDP model as having 5.6B parameters is appropriate. 5.6B is derived by counting the non-shared encoder 8 times (presumably because they are different) and adding the number of parameters for the shared encoder-decoder. This math is problematic for two reasons: 1) the non-shared parameters are included in the final product; 2) for each note, the computation considers one decoder and two encoders, not eight encoders. 
6. The PULL operation, as described in Section 3.4, requires pulling 1.6 x 8 B parameters. However, it is not considered in the 1.6B-parameter communication in L290, where the number of 1.6B is used to claim the save of 71% communication cost.
Comments, suggestions and typos: **Writing Suggestions**:
1. L61-L63: the sentence is too hard to understand. 
2. In Eqns. (1) and (2), the second $θ_{mix-enc}$ for $F_i$ should be $θ_{mix-dec}$

**Questions**:
1. Is the training of Single-FT exactly the same as SASDP, except for not having the local language-specific encoders? This includes the same data loading scheme, same training configration, and same inference setting?
2. In Table 3, what are the numbers? Are they BLEU scores? If so, does S-T means the BLEU on source and target? Since some of the languages may share the same tokens, and some of the samples may have code-switch, I am not sure if S-T BLEU is a meaningful metric for "source copy."
Overall assessment scores: 2.5