Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an optimization method for training multi-task learning models by increasing the task alignments. Their key assumption is that standard MTL can result in catastrophic forgetting of pretrained knowledge and lead to local task optimum in the finetuning stages. To avoid this, the paper proposed a method to jointly optimize the task losses and gradient alignments between tasks. Their key contributions are:
- The paper empirically examined the question of whether worse alignments of task gradients can lead to negative transfer, though extensive experiments on multilingual tasks.
- Based on such observation, the paper proposes a simple and effective optimization method for MTL that jointly optimizes the task losses and gradient alignments between tasks.
main_review: Strengths:
- The hypothesis is very interesting: is the negative transfer actually caused by the misalignments of task gradients? The paper provides some empirical evidences showing that such correlation (may or may not in causality relationship though) did exist, through a large number of synthetic and multilingual experiments. 
- The proposed sequential reptile algorithm is simple and effective. 
- The experiments are pretty extensive and show many interesting insights, which also verifies the effectivenesses of the proposed method. I think the following insights can be important:

a). Both reptile and sequential reptile show better generalizability across different tasks of the final optimized model.

b). By implicitly adding the task gradient alignment, the resulted finetuned MTL model show less forgetting of the pretrained knowledge, and higher cosine similarities across tasks.

c) The task gradient alignment does have correlation with the MTL task model performance, though it's still not sure if they are in causality relationship.

Weaknesses:
- The paper's findings would be much more generalizable if the experiments are also performed in the MTL setting of a single language, i.e., in a standard MTL setting.
- There are still a few issues in the experiments:

a) In the synthetic experiment, instead of only showing the loss landscape of a single initialization point, it would be more convincing if such patterns hold for random initialization.

b) In Table 1&2, the average performance is calculated as the mean performance across different languages. Such average doesn't consider the differences of the number of train/eval examples for different languages. Typically, we observed most MTL methods hurt for high-resource languages while improve on low-resource language. Simply averaging across different languages lead to a discrepancy between MTL model training and results reporting, as the loss is optimized as the sum of the example losses in training time. Therefore, simply averaging the performance across languages can be a misleading performance report as the actual empirical effectiveness per example may differ. But I guess this issue exists in most MTL papers. 

c) In Table 5 & 6, it would be good to include the single-task learning (STL) performance on zero-shot cross-lingual experiments. 

d) More experiments on some hyper-parameters could lead to better understanding of the algorithms, especially on inner and outer learning rates and inner gradient steps. 

summary_of_the_review: Overall, I think the paper's merits outweigh its drawbacks. It provides some important questions and findings to multi-task learning. Therefore I would recommend accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Sequential Reptile, a meta-learning method to perform inter-task gradient alignment to alleviate negative transfer and catastrophic forgetting. Empirically, experiments on both multi task learning and zero-shot cross-lingual transfer settings for QA and NER tasks demonstrate the effectiveness of the proposed method.
main_review: Strength

1.The paper is well-organized and clearly presented.

2.The proposed Sequential Reptile is very simple and extensive experiments show that Sequential Reptile can significantly outperform baseline methods.

Weaknesses

1.Negative transfer and catastrophic forgetting are the common issues in the transfer learning field, which are not specific for multilingual learning. Besides, the proposed Sequential Reptile method is also general (simply modify the task sampling strategy to a sequential way during the inner optimization), which also has not special designs for the multilingual setting. I am curious why the authors choose this setting instead of more general transfer learning settings to verify the effectiveness of the proposed method.

2.The authors claim that finetune a model with MTL objective will gradually enforce the model to memorize task-specific knowledge. Did the 
authors compare with some transfer learning methods like multi-task adversarial training to learn task-invariant representations?

3.The experiments only focus on two tasks, i.e., QA and NER. It would be better to verify on more multilingual tasks, e.g., XGLUE and Xtreme.

4.Are the improvements statistically significant?

summary_of_the_review: I carefully review this paper. I would vote for marginally below the acceptance threshold and make the final decision after seeing the rebuttal from authors .
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new training method to improve gradients alignment between languages (tasks) in multilingual finetuning of pretrained language models. The method is based on an existing algorithm, Reptile, which was originally developed for meta learning. The authors adapted it for multi-task learning and verified its effectiveness in multilingual finetuning.
main_review: Strength: 
- This paper tackles a problem which is less understood nor solved. The experiments results demonstrate the empirical significance of the work.
- The method is simple and the authors provide insights into the problem and the methods, i.e. the adaptation of Reptile.
- The experiments design are reasonable, specifically the zero-shot crosslingual transfer experiments are interesting.
- The overall writing is clear and the content flows well.

Weakness:
- One major drawback of the paper is that it mixes two problems, i.e. catastrophic forgetting (of knowledge in pretrained model) vs. negative transfer between tasks (languages) during finetuning. It seems the authors focus on solving the second problem, but they also hypothesize that solving the second problem would improve the first problem as a by-product. For example, the paper made several claims such as "As a result of such negative transfer, we see from Fig. 5a and Fig. 5b that the baselines suffer from catastrophic forgetting.". However, such hypothesis (that the former caused the latter) was never verified. 
- Another limitation is the incompleteness in experimentation. For example, the authors use BERT base, which is a relative small model and draw conclusions such as "It implies that the baselines suffer from negative transfer while ours is relatively less vulnerable.". Several work has shown that negative transfer is related to model capacity, e.g. capacity bottleneck has been well studied in MTL literature and specifically in multilingual translation [1]. Therefore, I'd like to see experiments with larger models to verify that this method is still as effective in mitigating negative transfer as was observed in small models.
- Related to the above point, it's unclear whether some of the worse performance from baselines are due to specific experiment setup which puts those approaches at disadvantage. For example, the authors shows that "We see that all the baselines butours highly degrade the performance on high resource languages" but the experiment was conducted in a manner where such observation may be expected, i.e. the authors adjust the sampling distribution to pt∝(Nt)^1/5, which upsamples low resource while downssamples high resource, that is, it intrinsically hurts high resource languages' performance.


The technical details in several places are not clear or lack of justification:
1. In Eq. 1, the MTL loss is defined as the sum of single task losses. This definition is quite limited since minimizing this loss may not be the objective of multi-task learning. A more general MTL objective has been studied as a Pareto front of task losses [2].
2. This work points out the importance of optimization setting which leads to different learning trajectories. However, the experiments were done with a specific set of hyperparameter, e.g. the authors chose 8 tasks (languages) in inner loop and sampled them with a specific distribution, the batch sizes were chosen differently for different tasks (e.g. QA vs. NLI). However, the author did not provide empirical justification that how senstive is the method to the choice of those hyperparameters.
3. In Fig 5.c, Why RecAdam has the largest L2 distance given that it has explicit objective to regularize this metric?
4. The clustering preprocessing in zero-shot crosslingual transfer experiments are very unclear. Why do you need to do the clustering to form tasks instead of using language as tasks as is in other experiments?
5. Fig 7.c does not exactly capture the training overhead from the proposed method. Could you provide a wall clock based comparison (which is more precise) as you mentioned in the text?
6. Related work misses relevant work on the optimization methods of multilingual training, e.g. [3] and [4].

[1] Lepikhin, Dmitry, et al. "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding." International Conference on Learning Representations. 2020.
[2] Lin, Xi, et al. "Pareto multi-task learning." Advances in neural information processing systems 32 (2019): 12060-12070.
[3] Wang, Xinyi, Yulia Tsvetkov, and Graham Neubig. "Balancing Training for Multilingual Neural Machine Translation." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.
[4] Li, Xian, and Hongyu Gong. "Robust Optimization for Multilingual Translation with Imbalanced Data." arXiv preprint arXiv:2104.07639 (2021).
summary_of_the_review: This paper proposes a simple method to improve multilingual finetuning of pretrained language models. The proposed solution and its effectiveness indicated by the experiments results has potential for enough technical significance. However, the current version of the paper has room for improvement in terms of experiment completeness in order to draw conclusion on the practical value of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for multi-task learning (MTL), namely sequential reptile (SR). The proposed method aims to align task gradients and mitigate negative interference among tasks during the finetuning process of multilingual BERT. Specifically, different from a regular Reptile which performs the inner loop on a single (sampled) task, the modified version does so on a batch of tasks, sampled according to some prior distributions. The paper argues that this simple modification can promote gradient alignment naturally. Empirically, the proposed SR method outperforms all baselines considered significantly, while using similar or even less computational resources.
main_review: Pros:
1. The experimental results are strong and very promising. On all settings considered in this paper, the proposed method significantly outperforms all methods. If this can generalize to other MTL problems, the method has a good potential for the MTL community.
2. The paper provides plenty of experimental evidence to demonstrate the effectiveness of the method. These results also reveal its behavior in the multilingual setting that can be valuable for future research (although I do find some of them to be a bit hard to understand, please see below).

Cons:
1. The novelty of the method is somewhat limited. The proposed modification is just a simple change to the original reptile. In addition, the resulting SR algorithm is actually very similar to a trivial MTL algorithm such that its inner loop is essentially a naive MTL algorithm without within-batch mixing, while the outer loop can be seen as a form of regularization. So indeed the method is a combination of naive MTL with regularization, as suggested in Eq (1). This, however, seems to deviate from the original design principle of Reptile.
2. The intuition behind the method and the source of improvement is not clear to me. This is particularly important given my first point that the method is quite similar to existing methods at the first glance yet shows superiority. There are several points that need better clarification: (1) why are you considering Reptile specifically in the first place? what is the intuition of applying a few-shot learning method here? (2) the intuition behind SR is 'to consider gradient alignment across tasks as well'. While it does sound natural to do so, I find it hard to understand the overall framework with this modification. The original reptile (as well as MAML) aims to find a good initialization for all tasks and therefore considers the bi-level optimization setup which contains task-specific inner loop and task-universal outer loop. Here, this small modification of SR actually fundamentally changes this intuition and set task-universal goals for both inner and outer loops. This does not make sense to me and I wish to get more intuition on this point. (3) The source of improvement is said to be 'our method can effectively filter out language specific knowledge when solving the downstream tasks, which prevents negative transfer and helps retain linguistic knowledge acquired from the pretraining.' But why is that case? The results have shown that the resulting model is closer to the pretrained mBERT. But there is nothing specifically designed for SR to do so. The outer loop can be a potential cause yet regular reptile also has this step (while being much less similar in terms of L2 distance). I think it will be helpful if more analysis can be included to show the source of improvement.

Minor issues/questions/suggestions:
1. It seems to me that adding regularization towards the original mBERT is helpful for the performance. Can we add it for other baselines considered (or perhaps other related techniques introduced in [1,2,3])? Would that improve their performance? 
2. How about using within-batch mixing for the proposed method? How would that compare against the current version?

[1] Noise Stability Regularization for Improving BERT Fine-tuning. Hua et al., 2021.
[2] Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models. Leet et al., ICLR 2020.
[3] SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. Jiang et al., ACL 2020.
summary_of_the_review: Overall, this paper presents a simple method with strong empirical improvements, yet the intuition and source of improvement can be somewhat confusing.
review_title: null

Meta-review: This paper presents a gradient alignment approach to alleviate negative transfer and catastrophic forgetting for multitask and cross lingual learning. Experiments on many domains and datasets demonstrate the efficacy of the proposed approach.

All reviewers agree that the simplicity of the proposed method is a strength of the paper and the experiments are promising. They have suggestions to improve the experiments section, which I believe the authors have addressed in their rebuttal by adding GLUE, image classification, and statistical significance tests, among others. 

I recommend accepting this paper for ICLR.
Decision: Accept (Poster)