Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the relationship between multitask learning and robustness to neuron failures. The paper mathematically shows that  multitasking models are more robust to noises, and diverse tasks lead to higher robustness than similar tasks. The paper also presents experiments that demonstrate higher robustness in multitasking models (both linguistic and non-linguistic). 

strengths_and_weaknesses: I think the strengths of this paper includes:
- clearly written
- interesting motivation bringing cognitive science and bilingualism into the space
- interesting problem 
- experiments are conducted on several datasets and tasks. 
- the results could inspire other researchers 
- thorough appendix
The weaknesses are:
- the connection between the human and what the paper does sounds interesting, but I cannot fully understand it.
- I'm having trouble in understanding any practical implications and insights.
questions: - What would be practical implications and insights?
limitations: The authors addresses limitations and ethical considerations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces the hypothesis that neural networks are more resilient to structural noise when they have been trained to solve multiple tasks, rather than a single one. The hypothesis finds inspiration in a linguistic phenomenon under which cognitive functions of multilingual speakers are more resilient to dementia and cognitive decline. The paper proposes a formal explanation for this phenomenon based on an analysis of a linear case in which the task is that of approximating a number $T$ of target vectors $c_i$ using linear combinations of an orthonormal weight matrix $W^T$. In this case, they show that increasing the number of approximated tasks effectively decreases both the expected MSE of the solutions and the Frobenius norm of the linear coefficients (thus inducing L2 regularization on them), thus making a connection between structural robustness and L2 regularization. Then, the authors experiment both with linear representation layers on image recognition datasets and on language modelling, observing that models trained on more than 1 task tend to be more robust to different kinds of perturbations.
strengths_and_weaknesses: **Strengths**
1. This paper presents an intriguing hypothesis on the properties of optimal solutions to multi-task learning setups.
2. It presents an insightful theoretical analysis of the linear case that could possibly be extended in the future to account for more realistic scenarios.
3. Backs the theoretical claim with multiple experiments on different datasets.
4. The paper is reasonably clear, but there are several points that would need clarification (see Weaknesses below).

**Weaknesses**
1. Some parts of the experimental paradigm look a bit arbitrary. For instance, the authors use IMDB as an evaluation set. Why that is not clear. Do the results replicate, say, on validation sets of English Wikipedia? A second experiment uses the LAMBADA dataset as a secondary task. LAMBADA is *not* a language modeling dataset. It's just an evaluation dataset containing only a few selected passages, so it is not clear why it is a good choice for these experiments. It is also not clear what is the connection between the attention weights and the predicted effect on L2 regularization.  
2. There seem to be missing references that come up on a quick google search, such as this one: https://arxiv.org/pdf/2004.11072.pdf
3. The paper introduces experiments, whose results are only presented in the appendix. This is not appropriate. Either these experiments should be only briefly mentioned in the main body, or the full results should be discussed in it. Only introducing them seems like a way of working around the page limit.
4. There seems to be a strong jump between the predictions in the linear case to the non-linear one. The latter ones are only explored in the Transformer LM experiments. It would have been good to exploit the simpler image recognition experiments to also explore whether the results followed the predictions in non-linear networks trained with CE.
questions: 1. It is not clear to me whether the experiments were adequately controlled. Were the "monotask" and "multitask" models trained for a comparable number of iterations on a comparable number of data points?
2. $T$ as in number of tasks and $T$ in $W^T$ are completely different $T$s, right? The second stands for transpose? If so, this is a tad confusing. I would strongly recommend using a different symbol for the transpose (see https://tex.stackexchange.com/questions/30619/what-is-the-best-symbol-for-vector-matrix-transpose) or using a different letter for the number of tasks.
3. The image recognition experiments are performed on the linear case. What do you think would change if using a CE loss or corrupting a non-linear network as in the LM experiments? 
limitations: The authors have acknowledged that the studied networks are not state-of-the-art, but that is also not relevant to the question they want to study. They also make clear that beyond the loose inspiration on the cognitive phenomenon of "Bilingual Cognitive Reserve", no extrapolations should be made without significant work. To stay true to that acknowledged limitation, the authors could revise the title fragment "A Neural Model for Bilingual Cognitive Reserve", which for the uncareful reader could warrant such extrapolations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper is inspired by cognitive research studies where bilingualism increases brain robustness and demonstrates whether artificial neural networks are more robust when trained on multiple languages or tasks. The theoretical justification was provided by analyzing linear multitask representation learning across three types of corruption: Additive Gaussian noise to the weights, random weight pruning, and magnitude-based weight pruning. The experimental studies on both vision tasks using CNNs, and language tasks using monolingual and bilingual GPT-2 models establish that multitasking leads to higher robustness for diverse task vectors.
strengths_and_weaknesses: Strengths:
* Interpretation of both monolingual and bilingual language models and how these models connect to structural robustness is interesting.
* The paper formed a theoretical basis based on earlier cognitive bilingual studies and concluded that robustness comes from diverse tasks.  
* Another strength of the paper is its vast number of experiments.
 

Weaknesses:

* The authors, inspired by cognitive science research, indicate that bilingualism increases brain robustness by reducing the rate of cognitive decline due to aging.
* However, the current work lacks a discussion on some known results from neuroscience?
* In particular, I would have liked to see a discussion on how the brain learns the language structure of individuals who speak a single language vs. bilingual and neural networks.
* It is unclear which linguistic properties are missing because of adding noise to neural network models?
* Can the authors associate the linear mapping between these single or multi-task network models with mono or bilingual brain datasets? 
* Why do authors add noise to the weights of language models? Can we infer anything from the human brain? A clear justification is needed.

questions: * What is the difference in the brain's language network for monolingual vs. bilingual people?
* What are linguistic properties missing in mono vs. bilingual people with aging or neural degeneration?
* What are linguistic properties active in mono vs. bilingual people despite aging or neural degeneration?
* Similarly, all the above questions need to be addressed for Neural networks as well.
* Why do authors add noise to the weights of language models? Can we infer anything from the human brain? A clear justification is needed

Minor Comments:
Typos:
* Conclusion: For linear representation learning we obtained a characterization (, is missing before we)
* No citations for GPT-2 model when first time was introduced in the introduction.
* Missing citations for datasets and many acronyms


limitations: * The authors presented the problem clearly; however, the scope and applications of the problem are missing.
* Authors could check the weaknesses section and add more limitations if they have not been addressed in this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper argues that multi-task training functions as a regularizer in neural network optimization, yielding models which are more robust to different types of structural failures (perturbations or dropouts of weights). They demonstrate this formally by analyzing a representation learning task in which a model learns to predict multiple downstream task representations, and show a connection to L2-regularization under assumptions of the structural failure / noise process and orthogonality of task target representations. They next test the idea empirically in a broad set of experiments in multiple modalities (vision and language), and show that multi-task learning leads to slower performance decay (relative to single/fewer tasks) as the degree of various weight perturbations increases.
strengths_and_weaknesses: I think the paper presents an interesting idea, and the theory and experiments do a good job at backing up the claims as far as I can see.

- The paper combines a clear theoretical presentation with several stages of theoretical experiments. But the theory addresses the properties of an ideal (SVD) solution rather than investigating actual optimization methods. This gap would be less problematic if the authors included L2 regularization as a baseline / comparison case in the experimental section (see Question 1 below).
- The intuitive presentation is quite clear and worth the space in the paper, I think.
- The link to bilingualism is interesting but I think not entirely necessary. While the path leading the authors to their discovery is interesting, there doesn't seem to be a necessary conceptual link in either direction: the theory or experiments aren't necessarily about bilingualism (certainly not -- there are experiments on vision tasks!), and the authors themselves say that these results shouldn't be taken to imply anything strong for the cognitive neuroscience of language/bilingualism.
questions: 1. The theory section of the paper reaches for an interesting connection between multitask training and L2 regularization, and this is later channeled in the weight norm table (Table 1) of the experimental section. But why not make this connection stronger in the experimental section, by using L2 regularization as a baseline / comparison case? I would expect to find that strong L2 leads to better performance, but that L2 models would still decay more quickly than multi-task trained models. The remaining gap in "cognitive reserve" between L2 and multi-task models would be explanatory work for future papers :)
2. The tasks in figure 3 don't seem to show a crossover point like the later language tasks (in which models trained on more tasks perform poorly in low-noise regimes, but perform better than fewer-task models in high-noise regimes). Why not?
3. Minor comments
	1. Figure 4 (right) makes it hard to see the intended trend (change in performance as a function of epochs). Can we see a slice of the current x-axis with epoch on the x-axis? Or plot pruning probability as color and epoch as x-axis?
	2. Figure 5 caption doesn't match text description. Is Task 2 bilingual training (according to caption) or training on English LAMBADA (according to main text description)?
	3. Not sure, but it might be better to plot perplexity on a log scale in figure 1, 4 so that we can better see the differences between the two models.
limitations: Yes.
review_title: null

Meta-review: metareview: This paper has received positive reviews. There was some initial concern by one reviewer about 
* Missing discussion of neuroscience research
* Why noise is added to model weights

But these issues were clarified in the author response and the reviewer has updated their evaluation. 

There was additional discussion on the connection to neuroscience research, with one reviewer suggesting not to push the authors to make stronger links than they see fit. The limitations in Appendix E are appropriate and can even feature more prominently in the main body. 

In any case, these weaknesses are relatively minor compared to the paper's strengths:
* Interesting motivation and connections between bilingual models and people, and how deficits impact them. 
* insightful theoretical analysis 
* Experiments with multiple settings 

I therefore recommend acceptance. 

And in any case, I encourage the authors to take into account all comments by the reviewers when preparing their revision, especially the following:
* Comments and questions by Reviewer WfPh, especially about the experimental paradigm. 
* The comment and discussion by Reviewer 1N4w on the theoretical analysis and L2 regularization as a comparison. 




confidence: Certain
Decision: Accept