Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an investigation of how to fine-tune a model with multilingual data effectively with a theoretical and empirical evidence. They proposed a method to avoid catastrophic forgetting by using interior-point solver to optimize the objective function based on the theoretical work. The authors claim that the method is able to reach the Pareto stationary -- no more changes occurred by optimizing multiple objectives by applying optimization. They show the superiority of the approaches in Named Entity Recognition, Question Answering and Natural Language Inference, and several languages.

strengths_and_weaknesses: Strengths:
- Interesting work with a good theoretical foundation
- The task is very useful for learning multilingual models while optimizing the model to reach closer to the upper bound

Weaknesses:
- The paper is not easy to follow. The paper method is unclear until I carefully read the paper several times. I would say the paper would be easier to follow if they put more details on the Algorithm and the introduction. For example, what method does the paper use to optimize the weights, not just pointing to the equation. And, it would be great to add a short description of the method in the introduction so that the first readers can understand the proposed approach.
- The method is not well-motivated with a clear hypothesis. Therefore, I cannot find the rationale why the LF-MLF is useful in the abstract and introduction.

Typo:
- In conclusion, Learning => learning 
questions: - Did you run any significant tests or compute the std of all results (for all runs)? The scores for LF-MLF are similar to Uniform-MLF for some tasks and settings.
- I am curious about the running time of the optimization. Does this method efficient in practice?
limitations: There is no specific limitation section provided in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for maintaining multilinguality under fine-tuning when a set of languages is used for fine-tuning. The core idea is to maintain crosslingual generalization by finding updates that are deviating not too much from the original pretrained model and its loss, and those that many languages benefit from. The proposed loss is and its optimization are theoretically motivated and defined, and then evaluated on downstream tasks of the XTREME benchmark with XLM RoBERTa, where a few high-resource languages are chosen for fine-tuning, and the remaining languages are used for zero-shot evaluation.   

[NOTE: presentation and overall score updated after author rebuttal]
strengths_and_weaknesses: Strengths:
1. The problem setting of zero-shot generalization *after* fine-tuning is novel and offers impactful applications.
2. The experimental results are largely convincing.
3. The detailed impact studies (Section 5.3 and following) are insightful and thorough.

Weaknesses: [NOTE: all well addressed in revised version]
1. The paper misrepresents multilingual fine-tuning ("multilingual fine-tuning, which has not been explored before" l.34; "the fine-tuning scenario where multiple source languages are involved in fine-tuning, namely multilingual fine-tuning, remains unexplored." l.59). It has been explored before, most prominently in the paper for the XTREME benchmark (https://arxiv.org/abs/2003.11080) that this very paper is also evaluating on. "translate-train" and "in-language multi-task" (from XTREME) yielded gains over monolingual fine-tuning, and should be discussed, in particular in relation to the Uniform MLF baseline presented in this work. Another work that builds on multilingual fine-tuning is for example https://arxiv.org/abs/2205.02022 (NAACL 2022). What hasn't been studied in this setting afaik is the problem of forgetting for some languages, fine-tuning was either done on all languages of interest or only on English.
2. The abstract and introduction are not well representing the focus of the remaining sections, as it does not introduce and motivate the forgetting problem, that the proposed method is mainly developed for and the title is derived from. 
3. A comparison to all-language fine-tuning should be included as an upper bound, since training data is available for at least the NER task. 
4. Significance tests should be performed, since some differences are sometimes small (and averages across multiple languages) and might not be strong enough to conclude superiority of one model over the other. At least standard deviation could be reported as well (since it's 3 runs).
questions:  [NOTE: all well addressed in revised version]
1. Please discuss [26] and [27] in Related Work as well, as baselines are based on them, and the differences should be discussed.
2. How are mini-batches represented in the loss definition? And how are they composed (mixed across languages or only one language? This might make a different for optimization. 
3. How are the "heads" defined in Section 3.2? Why can one not use the pretraining head after having fine-tuned the rest of the parameters to get an estimate of L_p?
4. Figure 2 is not readable in black and white / grayscale.


limitations: Limitations are not addressed, except for that in some scenarios the proposed method does not outperform all other methods. Please discuss in which scenarios the method might not work, or where the assumption that being close to the original pretraining model/objective is not sufficient/desirable for downstream crosslingual generalization.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a way of learning from multilingual annotated data. They focus on the zero-shot transfer to low-resource languages for tasks like NER, QA and NLI.  The authors claim that for this setting they need to optimize the model for two new objectives: one that minimizes the forgetting of the low-resource languages from pretraining, and one that ensures that when fine-tuning on the different multi-lingual annotated datasets that the descent direction between them is shared or is in common. They formulate this as a constrained optimization problem and call their method Less-forgetting Multi-lingual Fine-tuning (LF-MLF). They show consistent but not very large improved performance on NER, QA and NLI and look at where labelled data from more languages helps - and unsurprisingly it does. 

strengths_and_weaknesses: Strengths 

They show consistent improvements averaged over up to 45 zero-shot language directions over three different tasks which is quite extensive. 

Weaknesses

It is not clear from the paper how their approach is different from the baseline methods. The authors should explain the most relevant baselines and describe how they are different: Project Conflicting Gradients method and Gradient Vaccine.  This is a major flaw in the paper. 

They have limited the usefulness of the methods to a very narrow, specific use-case: multilingual and zero-shot. The less-forgetting direction could in theory be useful for monolingual fine-tuning as well as multilingual fine-tuning. The combined descent could be useful for transfer learning with labelled data too - not just zero-shot. Why did the authors not mention this or do any experiments along these lines? 

It is probably incorrect to state line 75: "MTL methods are not suitable for multi-lingual fine-tuning" as the authors use such methods as baselines in the paper, even though they do not describe them and how they are different to the LF-MLF. 
 
The authors do not discuss the size of labelled data for each of the multilingual cases or discuss how this affects the results. Eg. in Table 2 en, de, fr, ru etc. as performance on fine-tuning just on Russian gives better performance that just training on English which does not make sense. 

The amount of improvement is not great. The simple uniform multilingual fine tuning method is the second best method and LF-MLF is ahead of it by only 0.5-1 point. 

questions: For Figure 3 and 4 please make sure that you specify if the x or y axis is the source or target.

limitations: This work does not discuss its limitations which is a limitation.
review_title: null

Meta-review: metareview: The paper proposes a method for finetuning multi-lingual pre-trained language in multiple languages simultaneously.  The task is formalized as a  constrained optimization problem and the upper bound of the forgetting is given in theory.  A method is developed for multi-lingual fine-tuning to minimize the upper bound.  Experiments are conducted in multiple downstream tasks and the model is fine-tuned in a few high-resource languages and the performance is improved in low resource languages as zero-shot settings.  The authors responded the reviewers' concerns and the reviewers agree the responses addressed their concerns.  The paper is recommended to be accepted, and I ask the authors to carefully prepare the final camera-ready version based on the reviewers' feedback.
confidence: Less certain
Decision: Accept