Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve
Authors: Giannis Daras
Negin Raoof
Zoi Gkalitsiou
Alex Dimakis
Venue: NeurIPS 2022 Accept (2022)
Keywords: multitask learning
robustness
linear representation learning

Abstract: We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise. Our study is motivated by research in cognitive science showing that symptoms of dementia and cognitive decline appear later in bilingual speakers compared to monolingual patients with similar brain damage, a phenomenon called bilingual cognitive reserve. Our language model experiments replicate this phenomenon on bilingual GPT-2 and other models.
We provide a theoretical justification of this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. We open-source our code and models in the following URL: https://github.com/giannisdaras/multilingual\_robustness.

Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve
Giannis Daras*
Negin Raoof
The University of Texas at Austin giannisdaras utexas.edu
The University of Texas at Austin eeginraoof@gmmil..com
Zoi Gkalitsiou
Alexandros G. Dimakis
The University of Texas at Austin zoi. boi.gkalitsiou @ wwwstinutexas.co. edu
The University of Texas at Austin
dimakis austin. utexas.edu
Abstract
We find surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide theoretical justification of this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads higher robustness for diverse task vectors. We open source our code and models in the following URL: http://ggthhh..coogggaaaaaaamatt robustness
Introduction
Converging evidence from cognitive science research indicates that bilingualism increases brain robustness by reducing the rate of cognitive decline due to aging [I] 2 and delaying the onset of symptoms of dementia [3] 4. appears that individuals who speak more than one language on regular basis are able to maintain typical cognitive functioning despite neural degeneration. This mismatch between cognitive functioning and brain pathology is called Cognitive Reserve (5), and its underlying mechanisms are poorly understood and are active topic of investigation.
Inspired by this research, we study whether artificial neural networks are more robust when trained on multiple languages or multiple tasks. Our experiments demonstrate that training on multiple tasks indeed increases structural robustness. We train monolingual and bilingual GPT-2 models with the same architecture and dataset sizes. Initially, monolingual GPT-2 6 models are slightly outperforming the bilingual ones, but when we introduce structural noise (by randomly deleting neurons or adding noise to the weights) bilingual models degrade more gracefully and eventually outperform the monolingual models in the high-noise regime. For some amount of noise, bilingual models start outperforingg the monolingual ones demonstrating cross-over in performance due to their increased robustness. We observe this phenomenon numerous models across three different types of corruption: additive Gaussian noise to the weights, random weight pruning and magnitude- based weight pruning 77).
Our Contributions: We provide a theoretical justification of this phenomenon by mathematically analyzing linear multitask representation learning g 97 Our analysis shows that introducing more
equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Monolingual
Bilingual
600
Average Parplxity
500
400
Cross-over point
0-
0.00
0.05
0.10
0.15
0.20
0.30
0.40
Random Weight Pruning Probability
Figure 1: Performance monolingual and bilingual GPT-2 models with the same architecture and training dataset size. We show the performance as we randomly crase weights. The x-axis indicates the probability of crasing an attention weight parameter (setting to it zero). The y-axis indicates the average perplexity over 20 runs with 95% confidence intervals. The bilingual model initially shows slightly worse performance, but weights deleted, the monolingual model declines faster and performs worse the highly damaged regime. This indicates that the bilingual GPT-2 model is more robust to neuron weight erasures. We show similar results for several models and types of errors in our experimental section.
C2
C1
Figure 2: Let c1, C2, be the standard basis of R² For two tasks, the best one dimensional approximation to C1, 02 is è [1/2, 1/2, 0]" but best one dimensional approximation to three tasks C1 C22 C3 [1/3, 1/3, 1/3]". original task vectors cı C22 og orthogonal i.c. diverse, since this creates regularization.
is creating l2 regularization since || ||ć1|1.. It is important that the
diverse tasks creates & regularization in the linear task heads. Further, we formally connect the Euclidean norm the learned representations to structural robustness under errors in the network weights. Our main theorem establishes that multitasking leads to higher robustness to additive noise for linear representations when the task vectors are selected as random and independent Gaussian vectors. Our results also establish that when the tasks are significantly overlapping, multitasking does lead to higher robustness and hence task diversity is necessary.
We experimentally observe that multitasking increases structural robustness for numerous networks and multiple problems including MNIST, CIFARIO 10, Newsgroup20, GPT models and finetuned GPT models on GLUE tasks. We train networks under exactly comparable dataset and architecture conditions and show that models become more robust structural failures they are trained with more tasks. We experiment with three different types of structural failures and show robustmess increases for all of them. We also experimentally observe that the addition of diverse tasks seems to regularize the model weights, as we predict in our theoretical analysis.
2 Theoretical Analysis
Building intuition. We start with a small numerical example to build intuition. Given feature vector E d we compute a dimensional linear representation Wa using matrix W e kxd We choose

---

W such that we best approximate a set of ground truth task vectors, fci, C2.... cr), that lie in R'. The learned approximation is é₁ W 7s Essentially, we use linear combinations of the columns of WT to approximate the task vectors. For simplicity, we assume that the columns of W are unit norm. We study the case where T, otherwise there are infinite solutions.
Assume we work in d = 3 dimensions with = 3 total tasks, = [1, 01,0, .2 = [0. 1,0 c3 [o, 0, 1J" Set our learned representation dimension to be k 1 dimensional. When T = 2, using only the first two tasks C C2. an optimal solution is W = v2 [1, 1, 0j. The corresponding linear head
1 = 72 and the approximate vectors are éς = W 71 = [0.5, 0.5, 0J" = 02. is now the scalar 71 = v2
Therefore the best one dimensional subspace to jointly approximate c1. is the span of W = v2 (1, 0]. Now we introduce one more task and find the one dimensional subspace that best approximates C1. C2, That becomes W y3 1. with linear heads 7 /3 = 2/ 3/3-
The approximate vectors now are & (W'*> = [1/3, 1/3/" E = 2g Notice that || = 1/3 for 3 tasks but ||]]| = 1/2 for two tasks. The point that for more tasks, the vector that jointly approximates all task vectors becomes shorter. Equivalently, the E2 norm of the linear task heads decreases from 7/ v2 to 7 v3 as the tasks increased from two to three showing
how multitasking creates regularization. A graphical representation of this example is given in Figure It is important that the task vectors c; are orthogonal, increasing the effective dimensionality of the problem. The intuition is that diverse tasks increase the effective dimension, making the best approximation vector shorter.
2
Our main theoretical result is that this phenomenon is quite general and makes multitasking lead to structural robustness. We connect the norm of the approximated task vectors with robustness to weight perturbations and show that for Gaussian, independent task vectors the average norm shrinks more tasks are added. This is intuitive since high dimensional Gaussian vectors netr-orthogonal. Surprisingly, we empirically show that task vectors for numerous problems also exhibit this behavior.
Analysis. We consider neural network fo : Rd - R$ and collection of tasks (Ti,.., Tr). We are trying to learn 0, ?/ E RR* to solve the following optimization problem:
argmin
Fl fo(x).
The neural network So can be simple as single matrix W Rd R5 For linear networks, we consider the following dataset generation process: for task T. we sample a Gaussian æ and we generate its label y by taking the inner-product with task vector G, i.c. y cf x for task T. Given infinite samples and MSE loss, the optimization problem of ( is equivalent to the following problem.
Definition 2.1 (Optimization Problem). Let k < T d. We define the Factorized Best Rank-k approximation of a matrix C' E _dxT as the optimization problem:
W r argmin || - C
We are interested in the case when the dimensionality of the representation k is smaller than the number of tasks T, otherwise the best Rank-k k approximation of is not unique.
The following Proposition states that in the considered setting, Problem can be solved with SVD. Proposition 2.2. For any matrix C B with distinct singular values, any solution of 2. satisfies:
W* p*
U.
where UEVT the SVD of and Ex is the same as E, except than the last T k diagonal entries that are zeroed out.
The fact that the Singular Value decompasition computes the best rank-k approximation to matrix can be found in several textbooks e.g. Golub and Van Loan [10. Blum et al. [.]
This proposition establishes that I* = UT and 1 Ei VT is valid solution of 2 Onwards, we will be calling this the SVD Solution.
Definition 2.3. We define the SVD solution of (2), to be:
WSVD - I'vD V
(4)
3

---

We note that if any multitask learning algorithm is used to obtain W ,T* one can run Gram-Schmide Schmidt to make W orthonormal and hence obtain the factorization we use. It is important that W stays normalized and all scaling is pushed to I' since to measure robustness to weight shifts, we are going to add noise to W only, and higher W scaling is equivalent lower effective noise.
We study how the performance is affected when the representation network, fe. is corrupted. Definition 2.4. For any sample x, the Mean Squared Error (MSE) for task is defined be the expected error between the model prediction under noise and the true value y. Namely,
MSE Bo, [(7% fo. - u)
where Jo. is the model that emerges after corrupting fo-
This measures how well the model approximates the ground truth under the presence of noise and under the constraint of a joint representation for multiple tasks.
The simplest corruption process to study adding noise to the representation matrix, i.e.
We = W N, Nij ~ (0 o²), i.i.d
(6)
Then, we denote the mean squared error for the task with MSE and the average mean squared error across the T tasks with MSE".>" We are now ready to introduce our results.
Theorem 2.5 (Mean Squared Error for Additive Noise). Let C RaxT be a matrix with distinct singular values 01 > 02 > dT. Let W, T be the SVD solution of ② Under the Additive Noise Model defined in we have that:
Average MSE without noise
MSE T
MSE T,0
2i-1 (2(()2
T
Average MSE under noise"
Noise Variance
As shown, the noisy MSE decomposes into the sum of the noiseless MSE plus the noise variance times a function that depends on the number of tasks:
Σ 1.
R(T)
is important to emphasize that more tasks added, the matrix C' changes, but the interlacing theorem allows us to connect the singular values of smaller submatrices, as discussed in the Appendix. R(T) the robustness slope: if a model with T tasks has smaller slope, it will eventually outperform a model with, say T - tasks and larger slope, for sufficiently large noise. This is true even if the noiseless performance for the T 1-task model is better, indicating a cross-over MSE. Therefore the key is understanding when the sum of the top & singular values of C' scales sublinearly in T. This not true for tasks that are aligned, but we can show it holds for independent Gaussian task vectors. We believe it holds for more general families of diverse task vectors and our experiments verify it also holds for numerous real task vectors learned from text and vision datasets.
Connection with la regularization. For the SVD solution (see Definition 4 the sum of the top-k singular values squared is the squared Frobenius norm of r. Indeed, we have that |vD || ||E, 11.. Since Σ₁ is diagonal matrix, each row of V T is rescaling of the corresponding row of V.. Rows of T have norm 1, hence the i-th row of Σş VT will have norm o; The Frobenius norm squared is just the sum of the squared norms of the rows. Hence, we get that
[TvD
Σ (C))
(9)
Using this simple observation, can get the following alternative expression of Theorem 2.5 Corollary 2.6. Let e g be a matrix with distinct singular values. Let W,T I be the SVD solution of (2) Under the Additive Noise Model defined in (6). we have that:
||T12 0²
MSR. To
T.O + MSE
(10)
T

---

provides two important insights: i) the normalization with the number of tasks that appears 7 is justified since the Frobenius norm of I grows with the number of task. ii) if we can prove that the slope (defined in Equation (8) is dropping, then we are effectively proving that multitasking gives l2 regularization as we showed in the toy introductory example. This also holds for the case of Gaussian, i.i.d. task vectors, as shown in the following theorem.
Corollary 2.6
Theorem 2.7. Let RRxxT be a rundom matrix with Gaussian, i..d. entries of variance 1/d and d ). Let Ci, Ci+1 be the matrices formed by selecting the first t, (t + 1) columns of C. Then, there is a noise level othrres such that with probability > 1 - exp (-22 (s2(vo (va) ). the SVD solutions (see of 2 (for Ct, Ce+1 respectively), under the noise corruption model, satisfy:
MSE MSE < *SE 1.2
Vo 2 Othres.
(11)
Remark 2.8. In words, this result shows that adding new tasks gives provably increased robustness to high noise corruption in the weights, when the task vectors are Gaussian.
Remark 2.9. Observe that the MSE under noise drops for every single new task added. The assumption
d= )) ) can be relaxed to d = 22(t)), and we get increased robustness for the first t added tasks. Nevertheless, for most applications d (2(7)) is a realistic assumption: Even for smallest dataset MNIST d = 728, and we experiment with up to 10 tasks.
3 Experimental Evaluation
We divide the experimental section in two parts. In the first part, we add noise to the final linear repre- sentation layer of various networks and verify that our theoretical analysis agrees with experimentally observed multitasking robustness on real datasets (MNIST, CIFARIO. NewsGroup20). In the second part, we show that multitasking leads to robustness to general weight corruptions in any layer of a complex transformer. Specifically, we show that multilingual Language Models are more robust to weight shifts (across all the layers) compared to monolingual trained under the same setting. This is the first evidence of increased Cognitive Reserve in bilingual artificial neural networks.
Experiments with Linear Representation Layers. We perform experiments on three datasets (MNIST, CIFARIO, Newsgroup20) and two modalities (Vision and Language). The datasets normally involve one classification task each. We create multiple binary tasks by distinguishing between pairs of labels. For example, in CIFAR1O, one task might be distinguish between dogs and cats and another between airplanes and cars. We assign a value in fo, 1] to each sample for each task to transform them to regression tasks (to match our theory). For example, if task i is to distinguish between dogs and cats, value 0 corresponds to dog and value 1 to cat.
The second issue is learning the task vectors from training data. For MNIST, we can simply learn linear layer C with columns (ci...., CT) such that: of x s y for each task. For more complex datasets like CIFAR Newsgroup20, linear networks have lower performance and hence it is less interesting to examine their robustness. Instead, we first use another network to extract representations go(x) and then learn a linear layer acting on the encodings such that of (x R y. For CIFAR used a pre-trained Resnet50 as the encoder while for NewsGroup. a pre-trained BERT 12). We would like point out that our theory is still valid for this case this is equivalent to the linear layer C receiving inputs from a learned representation as opposed to the features directly. As the number of tasks increase, reduce the number of training examples per task. We do this to make sure that the total training dataset size stays the same as the number of tasks increase.
Figure 3 shows how the average MSE behaves as noise increases for different number of tasks. Note that even though all models begin from roughly the same performance in the noiseless setting, the multitask models are much more robust to the corruption of their weights consistently among all the datasets and modalities. This is aligned with our theoretical analysis which predicts that the robustness slope (defined in Equation
(8)) decreases with the number of tasks. We calculate robustness slopes for learned task vectors for real datasets and plot their decay in the Appendix, where we further include all the details of how these models were trained.
Experiments with I Language Models. Our objective is to compare robustness to neural weight perturbations in monolingual and bilingual language models. We use the following perturbation
5

---

Kantake
Figure 3: MSE of model (versus optimal task vector) as function of noise added to the weights. From left to right: MNIST, CIFARIO, NewsGroup20. As shown for all these datasets, adding tasks increases the robustness of the model to noise in the weights.
models: 1) Random deletion of weight parameters: we zero-out p percent of the attention layer weights, 2) Magnitude pruning: we model attention weights by the magnitude and delete the smallest p percent of weights [7] 3) Random normal noise: we add zero-mean random Gaussian noise with standard deviation o² the attention weights.
On the selection of the linguistic pair, we selected Greek, a highly inflected language with very different morphology, syntax and phonology compared to English. It also uses a different script since Greek characters were not Romanized. This minimizes transfer between languages, something we wanted to avoid. In the Appendix, we present additional experiments for other Romance languages.
The dataset for the bilingual model is a concatenation of articles from English and Greek Wikipedia. To avoid the computational cost of training for new language, we start from the pre-trained GPT-2 (small) ( and we use the Language Model Recycling Technique, introduced in [13. GPT-2 small transformer-basee architecture for causal language modeling, with 12 attention blocks and 124M parameters. The tokenizer uses Byte Pair Encoding and has a vocabulary of 50, 257 tokens. For the bilingual model, we generate the architecture. We keep the vocabulary size the same, as changing the vocabulary size can affect the scale of the perplexity score for these models. Note that Wikipedia documents were not in the original training of GPT-2, but our monolingual baseline was subsequently finetuned on English Wikipedia. Details on all our training hyperparameters
tokenizer, vocabulary and embedding layer without changing
included in the Appendix.
We measure the quality of generated text using perplexity. Our bilingual model achieves 89 perplexity on randomly picked subset of the OSCAR [14] dataset and 76 perplexity on the English IMDB dataset [15]. Monolingual GPT-2 model achieves 36 perplexity on the IMDB dataset. the Appendix we include generated text for both the models. Although the perplexity of the bilingual model does not match the pre-trained GPT-2, the generated text of reasonable quality text in both languages.
Text Generation. Our first experiment is to compare the performance of both models under various parameter perturbations. First, try deleting random portion p (p from 0% to 40%) of attention layers' weight to observe and compare the trend of decay in text generation quality between the two models. We evaluate both models the IMDB dataset. As the graph in Figure shows, the monolingual model starts with text predictions closer to the source text, resulting in lower perplexity without noise. However, matches the performance of the monolingual one and eventually outperforms that.
shows,
delete more significant portion of weights, the bilingual model
Next, we try magnitude-based based pruning of portion of weights, p. to observe and compare the trend of decay in text generation quality between the two models. We sort the attention layer weights by the magnitude and set p percent of weights with the lowest magnitude to zero. Again, IMDB dataset to evaluate models. The graphs in Figure 4 show that as the training process continues, the model achieves a lower perplexity. Moreover, pruning additional weights has a less substantial impact on the model's performance. This graph shows that training the pre-trained GPT-2 model for few epochs bilingual dataset significantly improves robustness to weight perturbations.
the
In another experiment, we observe how the maximum singular value of the weight matrices changes throughout training process. We track the maximum singular value of attention layer weights. We use a pretrained GPT-2 model baseline, and train this model for 16k iterations on English text data from Wikipedia. Resuming from this checkpoint, we train two models: 1) We continue training model

---

Maculingull
Epech
2000
Bilingual
Epoch
1200
Epoch
Avonge Poplesity
Avrerage Pembestry
1590
1000
Epoch
Epoch
3000
Epoch 6
600
Ppoch
Epech 8
Cross-over point
20
2010
0515
0,225
0u00
0.995
0040
0.50
0.35
Buld
0.45
6.5
Guureian Noise Standand Deviation
Magtinate baned Weight Proming Probshility
Figure 4: Robustness to magnitude-baset weight pruning and additive Gaussian noise. When plotting perplexity under additive Gaussian noisc, x-axis indicates the standard deviation of noisc added to weights. Y-axis indicates the average perplexity over 20 runs with 95% confidence intervals. The second plot shows perplexity as we delete more weights based on magnitude, for the bilingual model each epoch. X-axis indicates probability of deleting sorted attention weight parameters. After only one epoch, the model shows higher sensitivity to weight perturbations. However, after eight epochs of training, it becomee more robust.
1 on task I (English Wikipedia dataset) for 16k more iterations. 2) We train a second model on a different English dataset, the LAMBADA dataset [16), for 16k more iterations. Figure[Jindiaate the results of this experiment by plotting maximum singular values of the first attention layer. As the Figure shows, training model on new dataset (task 2) results in faster decay of the maximum singular value.
Task
Task
Atemiion Weugh Mox Steeatee
Trikin
Figure 5: We show the effect of monolingual and bilingual training the maximum singular value of attention weights. The red line shows the maximum singular value for a monolingual model trained on English Wikipedia for 32k iterations. The green line shows the maximum singular value if in the 16K iteration we switch to bilingual training. As shown, bilingual training leads to faster decay in the maximum singular value.
Text Classification. We conduct another set of experiments to observe the robustness of fine-tuned monolingual and bilingual GPT-2 models for text classification. this section, we fine-tune both the monolingual and the bilingual GPT-2 models (previously trained) for downstream classification
23 24 25] 26 27 28 to compare the robustness of models to weight perturbations. The two perturbation methods tested in this section random weight deletion and random Gaussian noise added to attention weights. For each task, we fine-tune both models for ten epochs. When applying random pruning, the accuracy of each model is evaluated after deleting P percent of model weights, p ranging from 0% to 45%. When perturbing model weights by adding noise, try various Gaussian noise distributions with standard deviations ranging from 0 to 0.09. Experiment results can be found in the Appendix section.
tasks using the GLUE benchmark [7, T8 19 20 21
22
Random Pruning. We compare the classification accuracy between the fine-tuned model from the
monolingual pre-trained network and the fine-tuned model using the bilingual network. Each element in attention parameters is pruned with probability p. where p ranges from .0 to .45. We evaluate the classification accuracy for the following GLUE tasks: CoLA, QQP, SST2, MRPC, QNLI, and RTE.

---

Task
Fine-tuned using
Monolingual ckpt
Bilingual ckpt
SST2
70567.87
60663.121
QQP
70608. 195
60649.586
MRPC
70398.953
60590.769
RTE
70508 968
60590.765
CoLA
70519.781
60600.933
Table 1:
We compute the sum of the squares of the weights of an attention layer for monolingual and bilingual models. The latter have smaller magnitudes, indicating that multitasking induces weight regularization.
Pruning Probability
QQP
SST2
COLA
MRPC
RTE
m.
m
m.
m.
m.
0.00
0.876
0.843
0.908
0.862
0.437
0.218
0.828
0.774
0.646
0.595
0,05
0.873
0.842
0.909
0.866
0,425
0.203
0,804
0.769
0,640
0.589
0.10
0.867
0.833
0.899
0.868
0.403
0.204
0.730
0.744
0.603
0.575
0.15
0.848
0.819
0.871
0.866
0.366
0.185
0.619
0.730
0.600
0.562
0.20
0.804
0.786
0.836
0.859
0.326
0.179
0.416
0.663
0.561
0.553
0.25
0.711
0.732
0.806
0.847
0.267
0.159
0.377
0.653
0.543
0.546
0.30
0.656
0.678
0.760
0.828
0.216
0.137
0.320
0.504
0.537
0.536
0.35
0.638
0.674
0.714
0.815
0.153
0.092
0.317
0.420
0.522
0.494
0.40
0.632
0.655
0.683
0.793
0.097
0.058
0.316
0.328
0.521
0.488
0.45
0.632
0.636
0.651
0.773
0.060
0.042
0.316
0.328
0.525
0.485
Table 2: Performance under a range random pruning probabilities for various GLUE tasks. Columns labeled with "m" determine classification accuracy monolingual models and columns labeled as "b" determine accuracy of bilingual. CoLA is evaluated using Matthew's 's Correlation and other tasks are evaluated by accuracy.
We expect the accuracy of both models to decay we prune more considerable number of parameters. The monolingual model shows a faster decay in almost all tasks. For some tasks such as SST2, QQP, and MRPC, we observe that the bilingual model starts with lower accuracy, and its performance exceeds the monolingual model as we prune 5% to s 25% parameters, A detailed set of results in Table show models' average prediction accuracy the GLUE benchmark.
Random Noise. We also experiment with adding Gaussian noise to the weights. We vary the noise standard deviation from ,0 to 0.09. We evaluate the classification accuracy for the same tasks, When no noise is added to model parameters, the monolingual model performs slightly better for tasks like QQP and SST2. As increase the noise, the accuracies of both models drop with almost identical rates. However, both graphs illustrate cross-over point after which the bilingual model outperforms the monolingual. The bilingual model achieves significantly higher accuracy in the MRPC task when the standard deviation is greater than & 0.03. For CoLA and RTE, the monolingual model maintains maintains higher performance regardless of the noise level. A detailed set of results in the Appendix section shows models' average prediction accuracy on the GLUE benchmark.
Related Work
Cognitive Reserve and Bilingualism. Our work is inspired by Cognitive Science and evidence of Cognitive Reserve in bilinguals. One implication of our theory is that multitasking leads to smaller weights on average. This could be related studies performed in healthy older adults that indicate that despite overall less gray matter volume and poorer white matter integrity (i.e., poorer structural brain connectivity), older healthy bilinguals perform equally well or outperform monolinguals in several cognitive tasks 2
We would like to emphasize that our research is solely on artificial networks which have huge differ- ences to biological neurons. No definite extrapolations should be made to Cognitive Neuroscience without further work. Nonetheless, we show that there is simple mathematical abstraction that seems to align with the significantly more complex phenomena observed bilingual cognitive reserve.
Multitask I Learning. The most closely related work is by Mao et al. 29 which shows that multitask learning increases adversarial robustness. The intuition behind their proof that, with task diversity, the gradient of the loss with respect to the wrong label is small as orthogonal tasks make gradients
8

---

E3)
285
028-
689
673
0.0
628-
Monolingual B Biingual
Monolingua
005
Bilingual
00
01
04
02
64
Titerion Protohiity
Deletton Protshilty
03
MartraxCoom
02-
Classicates
Monulingual
Bilingual
Bilingull
Deleteen Frobubidity
Eidetionn Probubiitt
Figure 6: Performance comparison in GLUE tasks: QQP, SST2, CoLA, and MRPC under random erasures. QQP: Monolingual drops lower than the bilingual model after 25% of the parameters deleted. SST2: Monolingual drops with a faster rate, falling behind the bilingual after deleting 15% of the parameters. CoLA: Both models reach * MCC (random prediction) with * 45% of parameters pruned. MRPC: The accuracy of the monolingual degrades at faster rate as pruning probability increases higher than & 10%.
that cancel out. Wu et al. (30] establishes connection between robustness to weight perturbations and adversarial attacks. Our work is related but different since it directly establishes connection between structural robustness and multitasking and shows a cross-over in performance across various domains and tasks. Our theoretical analysis is also completely different compared to prior works. More information on multitask learning can be found in Mao et al. [29] and Ghamizi et al. [3
Many studies network compression and the Lottery Ticket Hypothesis related our Magnitude Pruning experiments. LeCun et al. (32), Han et al. 77) find that selectively pruned networks can be trained from randomly initialized weights match the performance of the original network. Frankle and Carbin (33 introduces the hypothesis that randomly initialized neural networks contain a very sparse sub-network that, if initialized correctly, can achieve the accuracy of the original model. Chen et al. [34] studies this in continual learning and examines various pruning methods.
Conclusions
We demonstrated connection between multitask learning and robustness structural failures for artificial neural networks. For linear representation learning we obtained a characterization of robustness through the spectrum of the task matrix. We showed that robustness comes from diverse tasks which imply a bounded spectral norm for C. One limitation of our theoretical work is that we did not analyze learning algorithms but directly used the SVD solution. It would be interesting gradient descent introduces further regularization or other effects, especially in the non-linear case.
Experimentally, we observed increased robustness for both linguistic and non- non-linguistic tasks. More complex settings like multi-lingual models, cross-language transfer and their interactions remain to be explored. Finally, it remains open if bilingualism and cognitive reserve in humans can indeed be connected to our framework. would fascinating if neuroimaging techniques can measure any form of anatomical functional regularization that bilingualism could be creating in humans.

---

Acknowledgments
This research has been supported by NSF Grants CCF 1763702, AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the Archie Straiton Endowed Faculty Fellowship.
References
[1] John AE Anderson, John G Grundy, Cheryl I. Grady, Fergus IM Craik, and Ellen Bialystok. Bilingualism contributes to reserve and working memory efficiency: Evidence from structural and functional neuroimaging. Neuropsycoologia, 163:108071 2021.
[2] Brian T Gold, Nathan F Johnson, and David K Powell. Lifelong bilingualism contributes to cognitive reserve against white matter integrity declines aging. Neurapsychologia, 5 51(13): 2841-2846, 2013.
[3] Ellen Bialystok, Fergus IM Craik, and Morris Freedman. Bilingualism as protection against the onset of symptoms of dementia. Neuropsychologia, 45(2):459-464, 2007.
[4] Fergus IM Craik, Ellen Bialystok, and Morris Freedman. Delaying the onset of alzheimer discase: Bilingualism form of cognitive reserve. Neurology, 15(19):1226-172,, 2010.
[5] Daniel Barulli and Yaakov Stern. Efficiency, capacity, compensation, maintenance, plasticity: emerging concepts in cognitive reserve. Trends in cognitive sciences, 17(10):502-500 2013.
[6] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[7] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 1135- 1 1143, Cambridge, MA, USA, 2015. MIT Press.
[8] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes: The benefit of multitask representation learning. J. Mach. Learn. Res., 17(1):2553-2844 jan 2016. ISSN 1532-4435.
[9] Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021. URL
[10] Gene H Golub and Charles F Van Loan. Matrix computations. edition, 1996.
[11] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science, Cambridge University Press, 2020.
[12] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[13] Wietse de Vries and Malvina Nissim. As good as new. how successfully recycle english gpt-2 to make models for other languages. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021. doi: 18.15//3/v1 1/2223///222211 1/2021 1iw....gnnnnnnl...777 URL http'
//dx. www.oo......oooooooooooooooooooo 1ww.31886110000 f findings-ach ac1.74
[14] Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipelines for processing huge corpora on medium low resource infrastructures. Proceedings of Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 16, Mannheim, 2019. Leibniz-Institut für Deutsche Sprache. doi: 10. 14618/ids-putl 9021. URL http:ttt/bbbrrreo/////
de/urn//rnn :nn de bez mh39-90215
10

---

[15] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts, Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URI.
[16] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context, 2016.
[17] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. Glue: multi-task benchmark and analysis platform for natural language under- standing. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP,, 2018. doi: 10.18653/11/118-35 18-5446. URL. http
7/dx. //dx.doi...
[18] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1643 2013.
[19] Daniel Cer. Mona Diab, Eneko Agirre, Inigo Sopez-Gazpio, and Lucia Specia. Semeval- 2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) 2017. doi: 10.18653/v1/s11-20000
http:tp:://dd doi org/ 100..1////vvv1771
[20] Shankar Iyer,
Nikhil Dandekar, and
Kornel
Csernai.
First
quora
dataset release: Question pairs, 2017.
URL attps://data.quee quora. com/
First- Pirst-Quora Dataset-Releane Ree.eaasse-uession-Paa
[21] Hector Levesque, Emestt Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[22] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
[23] Roy Bar-Haim Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop recognising textual entailment, volume 6, pages 6-4, Venice, 2006.
[24] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recog- nizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pages 1-9. Association for Computational Linguistics, 2007.
[25] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[26] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
[27] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 11625-641, 2019.
[28] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, 2016. doi: 10. 1653/v1//d16 1264. URI. http::/dxx....../ doi
/0. 1tg/100018653//1/66-12222

---

[29] Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray. Shuran Song, Junfeng Yang.
and Carl Vondrick. Multitask learning strengthens adversarial robustness, Lecture Notes in Computer Science, page 158-174, 2020. ISSN 161 1-3349. doi: 10. 10.1077/978-3-030-583336 10. URI, htt..tttp///xx.do:// http://xxxddi.orr./../ org/ 1007 5000//88----000-5------5--00-0
[30] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-2966 2020.
[31] Salah Ghamizi, Maxime Cordy, Mike Papadakis, and Yves Le Traon. Adversarial robustness in multi-task learning: Promises and illusions, 2021.
[32] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1990.
[33] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019.
[34] Tianlong Chen, Zhenyu Zhang. Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference Learning Representations. 2021.
[35] Michel Ledoux. The concentration of measure phenomenon. Number 89. American Mathemati- cal Soc., 2001.
[36] Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. Proceedings of the 15th International Symposium of Information Science, pages
March 2017. doi: 10.528 1/2e2/0//ooooo44 4120316.
[37] Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pages 610-623, 2021.
[38] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL httt:: www.:newriicc.ooo cwtpppppcccppppp//pppppppp hi
9015-pytorch-aa an- inn-immrratie-sseee
library
pdf
[40] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-an natural language processing, 2019.
[41] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
httt:://githe com/ google/jax
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
12

---

(b) Did you describe the limitations of your work? [Yes] See Section S of the main paper and Section the Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section E of the Appendix.
(d) Have you read the ethies review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of theoretical results? [Yes] The formal statement is in the main paper.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section A (Proofs) of the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See Section in the Appendix. The code open-sourced in the following url: https://github.com
roobsstnss
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section of the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We included error bars wherever the cost of running multiple times was not prohibitive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section of the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Section F of the Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We open-source the code and the models in the following url: httpgghhubb..oogggaaiisssrr///uu
rooustness
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "s2NiViaG7DD", "reviewer": "Reviewer_QnuC", "summary": "The paper is inspired by cognitive research studies where bilingualism increases brain robustness and demonstrates whether artificial neural networks are more robust when trained on multiple languages or tasks. The theoretical justification was provided by analyzing linear multitask representation learning across three types of corruption: Additive Gaussian noise to the weights, random weight pruning, and magnitude-based weight pruning. The experimental studies on both vision tasks using CNNs, and language tasks using monolingual and bilingual GPT-2 models establish that multitasking leads to higher robustness for diverse task vectors.", "questions": "* What is the difference in the brain's language network for monolingual vs. bilingual people?\n* What are linguistic properties missing in mono vs. bilingual people with aging or neural degeneration?\n* What are linguistic properties active in mono vs. bilingual people despite aging or neural degeneration?\n* Similarly, all the above questions need to be addressed for Neural networks as well.\n* Why do authors add noise to the weights of language models? Can we infer anything from the human brain? A clear justification is needed\n\nMinor Comments:\nTypos:\n* Conclusion: For linear representation learning we obtained a characterization (, is missing before we)\n* No citations for GPT-2 model when first time was introduced in the introduction.\n* Missing citations for datasets and many acronyms\n\n", "limitations": "* The authors presented the problem clearly; however, the scope and applications of the problem are missing.\n* Authors could check the weaknesses section and add more limitations if they have not been addressed in this paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Interpretation of both monolingual and bilingual language models and how these models connect to structural robustness is interesting.\n* The paper formed a theoretical basis based on earlier cognitive bilingual studies and concluded that robustness comes from diverse tasks.  \n* Another strength of the paper is its vast number of experiments.\n \n\nWeaknesses:\n\n* The authors, inspired by cognitive science research, indicate that bilingualism increases brain robustness by reducing the rate of cognitive decline due to aging.\n* However, the current work lacks a discussion on some known results from neuroscience?\n* In particular, I would have liked to see a discussion on how the brain learns the language structure of individuals who speak a single language vs. bilingual and neural networks.\n* It is unclear which linguistic properties are missing because of adding noise to neural network models?\n* Can the authors associate the linear mapping between these single or multi-task network models with mono or bilingual brain datasets? \n* Why do authors add noise to the weights of language models? Can we infer anything from the human brain? A clear justification is needed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "gzRreSwoiU4", "reviewer": "Reviewer_NDUb", "summary": "This paper investigates the relationship between multitask learning and robustness to neuron failures. The paper mathematically shows that  multitasking models are more robust to noises, and diverse tasks lead to higher robustness than similar tasks. The paper also presents experiments that demonstrate higher robustness in multitasking models (both linguistic and non-linguistic). \n", "questions": "- What would be practical implications and insights?", "limitations": "The authors addresses limitations and ethical considerations. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the strengths of this paper includes:\n- clearly written\n- interesting motivation bringing cognitive science and bilingualism into the space\n- interesting problem \n- experiments are conducted on several datasets and tasks. \n- the results could inspire other researchers \n- thorough appendix\nThe weaknesses are:\n- the connection between the human and what the paper does sounds interesting, but I cannot fully understand it.\n- I'm having trouble in understanding any practical implications and insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zr_w0CgYfpn", "reviewer": "Reviewer_WfPh", "summary": "The paper introduces the hypothesis that neural networks are more resilient to structural noise when they have been trained to solve multiple tasks, rather than a single one. The hypothesis finds inspiration in a linguistic phenomenon under which cognitive functions of multilingual speakers are more resilient to dementia and cognitive decline. The paper proposes a formal explanation for this phenomenon based on an analysis of a linear case in which the task is that of approximating a number $T$ of target vectors $c_i$ using linear combinations of an orthonormal weight matrix $W^T$. In this case, they show that increasing the number of approximated tasks effectively decreases both the expected MSE of the solutions and the Frobenius norm of the linear coefficients (thus inducing L2 regularization on them), thus making a connection between structural robustness and L2 regularization. Then, the authors experiment both with linear representation layers on image recognition datasets and on language modelling, observing that models trained on more than 1 task tend to be more robust to different kinds of perturbations.", "questions": "1. It is not clear to me whether the experiments were adequately controlled. Were the \"monotask\" and \"multitask\" models trained for a comparable number of iterations on a comparable number of data points?\n2. $T$ as in number of tasks and $T$ in $W^T$ are completely different $T$s, right? The second stands for transpose? If so, this is a tad confusing. I would strongly recommend using a different symbol for the transpose (see https://tex.stackexchange.com/questions/30619/what-is-the-best-symbol-for-vector-matrix-transpose) or using a different letter for the number of tasks.\n3. The image recognition experiments are performed on the linear case. What do you think would change if using a CE loss or corrupting a non-linear network as in the LM experiments? ", "limitations": "The authors have acknowledged that the studied networks are not state-of-the-art, but that is also not relevant to the question they want to study. They also make clear that beyond the loose inspiration on the cognitive phenomenon of \"Bilingual Cognitive Reserve\", no extrapolations should be made without significant work. To stay true to that acknowledged limitation, the authors could revise the title fragment \"A Neural Model for Bilingual Cognitive Reserve\", which for the uncareful reader could warrant such extrapolations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper presents an intriguing hypothesis on the properties of optimal solutions to multi-task learning setups.\n2. It presents an insightful theoretical analysis of the linear case that could possibly be extended in the future to account for more realistic scenarios.\n3. Backs the theoretical claim with multiple experiments on different datasets.\n4. The paper is reasonably clear, but there are several points that would need clarification (see Weaknesses below).\n\n**Weaknesses**\n1. Some parts of the experimental paradigm look a bit arbitrary. For instance, the authors use IMDB as an evaluation set. Why that is not clear. Do the results replicate, say, on validation sets of English Wikipedia? A second experiment uses the LAMBADA dataset as a secondary task. LAMBADA is *not* a language modeling dataset. It's just an evaluation dataset containing only a few selected passages, so it is not clear why it is a good choice for these experiments. It is also not clear what is the connection between the attention weights and the predicted effect on L2 regularization.  \n2. There seem to be missing references that come up on a quick google search, such as this one: https://arxiv.org/pdf/2004.11072.pdf\n3. The paper introduces experiments, whose results are only presented in the appendix. This is not appropriate. Either these experiments should be only briefly mentioned in the main body, or the full results should be discussed in it. Only introducing them seems like a way of working around the page limit.\n4. There seems to be a strong jump between the predictions in the linear case to the non-linear one. The latter ones are only explored in the Transformer LM experiments. It would have been good to exploit the simpler image recognition experiments to also explore whether the results followed the predictions in non-linear networks trained with CE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3J6wbCFiBgF", "reviewer": "Reviewer_1N4w", "summary": "This paper argues that multi-task training functions as a regularizer in neural network optimization, yielding models which are more robust to different types of structural failures (perturbations or dropouts of weights). They demonstrate this formally by analyzing a representation learning task in which a model learns to predict multiple downstream task representations, and show a connection to L2-regularization under assumptions of the structural failure / noise process and orthogonality of task target representations. They next test the idea empirically in a broad set of experiments in multiple modalities (vision and language), and show that multi-task learning leads to slower performance decay (relative to single/fewer tasks) as the degree of various weight perturbations increases.", "questions": "1. The theory section of the paper reaches for an interesting connection between multitask training and L2 regularization, and this is later channeled in the weight norm table (Table 1) of the experimental section. But why not make this connection stronger in the experimental section, by using L2 regularization as a baseline / comparison case? I would expect to find that strong L2 leads to better performance, but that L2 models would still decay more quickly than multi-task trained models. The remaining gap in \"cognitive reserve\" between L2 and multi-task models would be explanatory work for future papers :)\n2. The tasks in figure 3 don't seem to show a crossover point like the later language tasks (in which models trained on more tasks perform poorly in low-noise regimes, but perform better than fewer-task models in high-noise regimes). Why not?\n3. Minor comments\n\t1. Figure 4 (right) makes it hard to see the intended trend (change in performance as a function of epochs). Can we see a slice of the current x-axis with epoch on the x-axis? Or plot pruning probability as color and epoch as x-axis?\n\t2. Figure 5 caption doesn't match text description. Is Task 2 bilingual training (according to caption) or training on English LAMBADA (according to main text description)?\n\t3. Not sure, but it might be better to plot perplexity on a log scale in figure 1, 4 so that we can better see the differences between the two models.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper presents an interesting idea, and the theory and experiments do a good job at backing up the claims as far as I can see.\n\n- The paper combines a clear theoretical presentation with several stages of theoretical experiments. But the theory addresses the properties of an ideal (SVD) solution rather than investigating actual optimization methods. This gap would be less problematic if the authors included L2 regularization as a baseline / comparison case in the experimental section (see Question 1 below).\n- The intuitive presentation is quite clear and worth the space in the paper, I think.\n- The link to bilingualism is interesting but I think not entirely necessary. While the path leading the authors to their discovery is interesting, there doesn't seem to be a necessary conceptual link in either direction: the theory or experiments aren't necessarily about bilingualism (certainly not -- there are experiments on vision tasks!), and the authors themselves say that these results shouldn't be taken to imply anything strong for the cognitive neuroscience of language/bilingualism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept