Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Paper presents a curvature aware bi-level optimization algorithm applied to multilingual/multi-task machine translation. The proposed algorithm is designed to encourage the loss is reduced along the directions of low-curvature (with the implicit inductive bias of flat minimas being better at generalization). Akin to "Sharpness-Aware Minimization (Foret et al. 2020)" but in a multi-task environment. Experiments on three different datasets tests the effectiveness of the proposed method (called Curvature Aware Task Scaling - $CATS$). Results show clear trend that the proposed method is highly effective at improving the quality of low-resource languages. Further analysis provided to understand the tradeoffs, sensitivity and dynamics of the algorithm. 

- Making use of curvature information during the optimization of multi-task models for translation is novel and expected to be impactful in the long run (while the paper is not demonstrating convincing empirical results, it is showing the potential of using higher order signals during optimization, which I found to be an important contribution).
main_review: At high level paper proposes to learn to weight/scale the gradients of different languages/tasks in a bi-level optimization setup. The signal used for scaling the gradients is coming from the curvature (flatness) of the loss landscape, where the inductive bias is to guide the optimization towards flatter regions of the loss landscape. 

Authors found that optimization tension or the tug-of-war between high and low resource languages is due to the observation that they compete to update the loss landscape during early stage of training, with high resource ones dominating the optimization trajectory during the rest of training, and connect this to the existing methods (like upsampling low-resource languages). 

Paper provides a new perspective on the problem of interference in multilingual models which I found novel.

**Questions**
- What is the number of inner optimization steps `m`
- Figure 1(right) is not clear, how do the colored arrows relate to the (left)? Perhaps add some additional comments to the caption and the main text. 
- Figure 3 (left) the range of similarities are quite narrow, $\pm 0.01$, I'm a bit reluctant to make claims about gradient similarities (in terms of direction) given the range of the Figure, color-coding is misleading. But I agree that the available signal is somewhat supporting the argument "gradients are more similar when training with balanced data".  
- What is the inner optimization step rule, `Adam`, `SGD` or else?
- What is the value of the epsilon in $Eq.4$ ?
- In Section 3 you mention that you *only* report one-to-many results, but in $Fig.3,5$ you mention many-to-one. I couldn't get the motivation, could you please clarify?
- Section 4.2.1 refers to $Fig.5$ but i guess it should be citing to $Fig.3$ right?
- What is the method you used to compute the top-eigenvalues of the Hessian? $Fig.2$
- You have a section detailing the use of layer-norm and its impact on the gradient alignment. What is the architecture you are using in the first place? Pre-norm Transformer or post-norm?
- When you use $CATS\alpha$ how do we choose the sampling ratios? Also in the appendix you mention that you mix K=4 (TED&WMT) and K=8 (OPUS100), what is the rationale behind these, and did you measure the sensitivity of $CATS\alpha$ for K. 
- If we use T=5 and $CATS\alpha$ would that yield us the best adaptive schedule? Could you please elaborate on this?
- When you are plotting the NLL ($Fig. 4,5$), are you using $Eq.4$ ? What is the value we are seeing in the plots? (Asking because there is almost $1.0$ diff between different methods vs $CATS\alpha$)
- In WMT experiments, only German is added to the mix and is considered to be high-resource language. While it is the highest resource language in the mix (28M examples), it is still dwarfed by languages in the WMT (Czech, Chinese, French, Spanish ...) in terms of the amounts of data. I'm curious why you chose to add only 1 high resource language in the mix? Considering the gap between temperature based sampling and $CATS\alpha$ in $Table.3$ I suspect the ranking would change if you were to add more high resource languages. Given that the community is mixing high and low resource languages in a more uniform way (ie. by not including only one high resource language but as many as available), do you think the choice of languages is introducing a selection bias and skewing the results? I'm very much okay with the framing and main contribution as boosting the quality for low-resource languages btw. 
- Experimental results does not support your argument on "without hurting high resources (line 18, line 64)". The only realistic high-resource setting in all your experiments (TED, WMT, OPUS) is the WMT English->German direction, and with the experimental results you present in $Table 3$ (T=1 26.8 BLEU vs $CATS\alpha$ 25.9 BLEU) we cannot support the argument (also in $Table.4$ if we "assume" language pairs that have 1M training examples being "high-resource", final results are contradicting with the argument 22.1 vs 19.9 BLEU). Please revise accordingly (I think low-resource boost with this principled approach is still a valuable contribution, but we should be cautious about not overclaiming). 
- Baselines: I was puzzled by the motivation behind the selected baselines (Table 2). GradNorm is a method that has never been applied to the domain of interest here. If you would like to compare your method with a solid baseline exploiting the gradient directionality during optimization, it would be better to compare it against "Gradient Vaccine". Similarly, MultiDDS is making use of the validation set information, which is making it hard to compare since its inductive bias is quite different. I think the closest approach to your proposed method is Foret et al. 2020 "Sharpness-Aware Minimization for Efficiently Improving Generalization", which is also not applied to multilingual translation (but along the lines of using GradNorm as a baselines). I'd really like to see a section discussing what are we trying to measure by the selected "baselines".


**About the background and related work:**
-  2nd paragraph of your intro: [1] and [2] are not multilingual translation papers. While they are related work, they do not support the argument you are trying to make in the paragraph (which grounds the motivation). Please either remove the unrelated papers from the motivation (which will help the reader to locate the relevant information) or change the framing of the paragraph to include NLP in general. 
- Again in the 2nd paragraph, papers [10, 24, 46] are not providing the evidence you are using to motivate the manuscript. The benefits on some dimensions start to diminish, but some others keep improving with more scale and data (eg. the quality on (extremely) low-resource languages start to diverge and regress - but this only supports the argument on scaling model, ). Rather than motivating it along the line of deficiency of one particular branch in the multilingual NMT (data, scale, # languages), you can still motivate your work since optimization is a canonical branch encapsulating all. In short, I don't think you need to motivate your work which improves the optimization angle by showing the deficiency of other branches, but rather by showing the lack of compelling approaches that address optimization in particular (which the 3rd paragraph does nicely).
- Please credit [34] for the section 3.2 where both the motivational figure, equations and the narrative are very much alike with aligned flow, and perhaps point the reader to [34] for more details. 
- On Interference, or "the curse of multilinguality", paper [8] rebrands the interference problem, which was studied and introduced by [2] with "capacity bottleneck". It is not clear that the problem is exclusively related to the "multilinguality", as it could be observed under several other scenarios like multi-domain, multi-view, multi-class problems (in fact the capacity bottleneck is introduced in the seminal paper by Caruana 
 on multi-task learning where the abstraction is over tasks, not languages). I believe this will make the framing of the paper more generic and applicable beyond "multilingual" translation domain. If you don't want to abstract it further, please either be more specific under multilingual translation context, or apply proper credit assignment if you would like to highlight when and under which context it was introduced. 
- One thing that made me frown a few times while reading the paper was the way paper uses the previous work (citation rationale). Not all the citations are really supporting the argument the authors are trying to make, they sometimes appear to be remotely related or not related at all (we can of course bundle all of them under NLP applications if we wanted to). Please do another pass with this in mind and clean the citations. I hope the authors will improve this point. 

**Typos**
- line 190: "pubic" --> public
- Sub-captions of Figure 5 are hard to decouple between (a) and (b). Could you please use some more spacing?
- line 264: temperate -> temperature

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work looks at the optimization of multilingual NMT models. Typically, the data across different language pairs is highly skewed and prior work either upsamples low-resource languages such that it's uniform or uses a temperature (T=5) to upsample lowRes and downsample HiRes. This work proposes a new optimization algorithm called CATS which encourages solutions with low local curvature. The proposed technique has been shown to be effective when compared to 3 temperature-based baselines (T=1,5,infinity) and two dynamic methods multiDDS and GradNorm. Experiments are performed on 3 public datasets: TED, WMT and OPUS-100. Robustness of the technique to overparameterization and to large batches has been studied.
main_review: Originality:
- The paper approaches the optimization problem from a novel standpoint and establishes that low local curvature affects optimization. Then it proposes an algorithm that jointly also learns weights \alpha which regularizes local curvature of the loss landscape.

Clarity:
- For the most part, the paper is clearly written. However, there are a lot of minor mistakes. The figure numbers are mixed up and the paper could have been more polished.

Positives:
- The paper proposes a novel optimization algorithm called CATS which alleviates the lowres/highres interference problem in multilingual NMT.
- Results on 3 standard benchmarks against many strong baselines show that the proposed approach leads to improves across lores, midres and hires language-pairs.

Negatives:
- The paper should have had a dedicated section on the complexity of the proposed approach. This has largely been ignored. How much extra time does the additional optimization add to training? How stable is the proposed approach? 
- There are a lot of presentation mistakes which could have been easily avoided and a bit more care in organizing figures and tables better could have helped the reader.

Minor comments:
- L74: I think the idea of upsampling low-resource languages was already proposed in [20].
- L125: $N$ should be \mathcal{N}?
- L249-251: "Contrary to popular belief..." can you please cite prior work that states that language similarity has this effect?
- L241 Figure 5b shows... Do you mean Figure 3? 
- Figure 4 I believe the legend should say T=infinity and not T=1.
- L279 "empirical an" --> "empirical and"
- Section 4, why was the T=infinity baseline not evaluated for WMT and OPUS-100 corpora?
- L367 "principle optimization.." --> "principled.."?

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of robustness optimization in multilingual machine translation models. In response to this problem, the author empirically found that in the loss optimization of multilingual machine translation models under unbalanced data scenarios, "sharpness" of local curvature in the loss landscape causes interference among languages, which caused instability in training. Inspired by the robust optimization work of Multi-task Learning, the paper proposes the Curvature Aware Task Scaling (CATS) optimization algorithm, which alleviates the phenomenon that different translation directions competitive update the loss landscape during the early stage of training in multi-language translation training, while high resource ones dominating the optimization trajectory during the rest of training.
main_review: Strength:
1. The proposed CATS optimization method has achieved better results than the multi-objective optimization methods: GradNorm and MultiDDS.
2. The paper has done a very good analysis and visualization of the "sharpness" of local curvature in the loss landscape for the multilingual machine translation model in the unbalanced data scenario, making the research problem very intuitively presented.
3. This work also conducts experimental exploration on more practical scenarios: overparameterization and large batch size, which makes the proposed method more practicable. 

Weakness:
1. The unbalanced data scenario has not been properly explored by experiments. Under what circumstances can it be counted as an unbalanced data scenario, and what is the data ratio? Therefore, the experiments should not pay more attention to one given setting like TED, WMT, etc., but should construct unbalanced scenarios of different ratios by sampling data in one setting like WMT to verify this important issue.
2. There is a lack of a reasonable ablation study on the upsampling parameter T, so we cannot confirm whether the oversampling overfit phenomenon will occur, and to what extent will the upsampling reach.
3. Some baselines are missing in the experimental comparison, such as 1) giving different weights to the loss of unbalanced translation pairs so that in the later stages of training, there will be no situation where rich-resource pairs dominate the training loss; 2) the use of low-resource language pairs further finetune the multilingual model and use the method like R3F to maintain the generalization ability of the model.
4. In some low-resource language translations from 1.2->2.0, although the improvement of 0.8 can be claimed, it is insignificant in a practical sense.

Missing References:
1. Aghajanyan, Armen, et al. "Better Fine-Tuning by Reducing Representational Collapse." International Conference on Learning Representations. 2020.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper theoretically motivates and then implements a curvature-based, online, per-language learning rate adjustment to improve the performance of low-resource languages in multilingual machine translation (MT) models. Experiments in three scenarios varying in multilinguality and data size show that the technique works consistently, outperforming a widely-adopted temperature-based sampling baseline. Through a number of analysis and ablation experiments, they are able to provide insight into how and why the optimization technique works.
main_review: This paper takes a step toward solving an important problem (balancing languages in multilingual MT) using a well-motivated technique influenced by recent advances in the optimization literature. It has a very good balance of clear motivation, theoretical background, empirical results and analysis. The method may be too complex to see wide adoption without a clever or clear open source implementation (the baseline, after all, is very very easy to build), but the ideas and the connections to the optimization literature are likely to be influential regardless. The paper definitely has more than enough technical content to warrant a conference paper, and is bordering on a journal paper. I learned a lot and earmarked many other papers from the substantial bibliography to follow up on.

In terms of weaknesses, the paper is very dense. In particular, the analysis sections require a lot of effort to understand what was done and why. I spent much more time than I would have liked staring at legends, captions (sometimes multiple captions per figure!), and cross-referencing with the text to try to figure out what’s going on. Both halves of Figure 5 are particularly confusing. However, this time was generally worth the effort.

Small concerns:

The paper is littered with small disfluencies (particularly confusion of singular and plural nouns and corresponding agreement), but I found it did not hinder my understanding.

I found the definition in Section 3 of p_{|D_n|} confusing, but its intended meaning quickly became clear after seeing its use in the (familiar) temperature sampling baseline.

In the first paragraph of section 4.2.1 when Figure 5b is referenced, I’m 95% sure Figure 5a is what was intended.

Also in 4.2.1, I found the phrase “data volume” ambiguous and confusing. I guess you mean the total amount of training data for all languages? As opposed to the amount of data for any one language?

In general, the relationship between CATS (the proposed method) and layer norm is not clear in the main text. Figure 5b and the “Training without layer normalization” paragraph, together with the “Ablation layer normalization” paragraph seem to suggest that layer normalization is always switched off for CATS throughout all of the paper’s main results, but this is never explicitly stated in the main text. Likewise, in Figure 5b, it isn’t clear what the baseline system is (is it CATS or T=5 or T=1?). Presumably, the w/o layer norm line is just the baseline (whatever it is) without layer norm.
limitations_and_societal_impact: This is handled nicely in the appropriate section.
review_title: null

Meta-review: This paper studies the problem of training multilingual machine translation model in presence of imbalanced data, which is common in multilingual parallel corpus. The method is based on empirical observation that in multilingual MT training, "sharpness" of local curvature in the loss landscape causes interference among languages and instability. The paper then propose CATS, a curvature-based, online, per-language learning rate adjustment to improve the performance of low-resource languages in multilingual machine translation (MT) models.
Experiments in three scenarios  varying in multilinguality and data size (WMT-10, TED, OPUS-100) show that the technique works consistently, outperforming a widely-adopted temperature-based sampling baseline. Through a number of analysis and ablation experiments, they are able to provide insight into how and why the optimization technique works.
Experiments can be improved with more analysis (e.g. upsampling T). The paper also lacks comparison with some recent strong baselines for multilingual machine translation. The authors may consider moving some results and analysis from appendix to the main content. The overall writing, including figures, tables and captions, could be improved as suggested by all reviewers. 



Decision: Accept (Poster)