On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning
Authors: Julian G. Zilly
Alessandro Achille
Andrea Censi
Emilio Frazzoli
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer learning
Deep learning
Sparsity
Invariance
Sequential learning
Critical learning periods
Curriculum learning

Abstract: Plastic neural networks have the ability to adapt to new tasks. However, in a continual learning setting, the configuration of parameters learned in previous tasks can severely reduce the adaptability to future tasks. In particular, we show that, when using weight decay, weights in successive layers of a deep network may become "mutually frozen". This has a double effect: on the one hand, it makes the network updates more invariant to nuisance factors, providing a useful bias for future tasks. On the other hand, it can prevent the network from learning new tasks that require significantly different features. In this context, we find that the local input sensitivity of a deep model is correlated with its ability to adapt, thus leading to an intriguing trade-off between adaptability and invariance when training a deep model more than once. We then show that a simple intervention that "resets" the mutually frozen connections can improve transfer learning on a variety of visual classification tasks. The efficacy of "resetting" itself depends on the size of the target dataset and the difference of the pre-training and target domains, allowing us to achieve state-of-the-art results on some datasets.

On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning
Julian Zilly
Alessandro Achille Caltech eachille@caltech.edl edu
Andrea Censi ETH Zürich acensi@ethz.co ch
Emilio Frazzoli ETH Zürich efrazzoli@ethz.
ETH Zürich jzillyBethz.ch
Abstract
Plastic neural networks have the ability to adapt to new tasks. However, in a continual learning setting, the configuration of parameters learned in previous tasks can severely reduce the adaptability to future tasks. In particular, we show that, when using weight decay, weights in successive layers deep network may become "mutually frozen". This has a double effect: on the one hand, it makes the network updates more invariant to nuisance factors, providing a useful bias for future tasks. On the other hand, it can prevent the network from learning new tasks that require significantly different features. this context, we find that the local input sensitivity of deep model is correlated with its ability to adapt, thus leading to an intriguing trade-off between adaptability and invariance when training a deep model more than once. We then show that simple intervention that "resets" the mutually frozen connections can improve transfer learning on a variety of visual classification tasks. The efficacy of "resetting" itself depends the size of the target dataset and the difference of the pre-training and target domains, allowing us datasets.
achieve state-of-the-art results
Introduction
Lifelong learning 11 holds both the promise of benefitting from past experiences and the challenge of having to continually adapt to new problem settings. The characteristics of this intriguing sequential learning problem make it balaneing act between retaining knowledge and adapting new experiences referred to as the stability-plastiiity dilemma
3 Similar to a coach or teacher, we want understand when a sequence of tasks helps hinders further learning. Clearly, some pre-training tasks are highly beneficial 7 while others can hurt performance [ 9
The existing literature pre-training and curriculum learning sheds some light the desired characteristics of sequential learning. In some cases, following a sequence of learning tasks can lead to better results than simply training the target task from scratch [[O]. Work by Achille and Soatto & contrasts this picture by highlighting that neural networks can suffer from pre- pre-training on some tasks. In extreme case, when learning data is corrupted for sufficiently long period, switching back to clean data does not allow the neural network to recover its original performance.
Contribution: We formalize the basic ingredients of sequential task learning, introduce mutually frozen weights and establish a connection between plasticity and invariance of deep networks. To understand mutually frozen weights, we investigate their weight update dynamics, show that such frozen weights are different from regular sparse weights that are not connected to other sparse weights and that frozen weights lead to decrease in performance when retraining new task. Based on this analysis, we show that applying an intervention which "resets" frozen weights can improve retraining performance number of tasks long sufficiently many samples available for retraining and even achieve state-of-the-aa results on FashionMNIST image classification.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

"Frozen weights" impede retraining future tasks
Pre-training can create "frozen weights"
fg Model path (So. Jtmal)
fo
(Tg. w3)
(Ty, W2)
(T):w1)
Learning path T
M
In the context of the negative pre-training effect, e.g. when pre-training on blurred images before training unblurred images [8], we explore the effect of "frozen weights", which can hinder retraining in a sequential learning setting, on the generalization performance of neural networks A point on the learning manifold M is supervised learning task r := (p(ar, y), C),, composed of data distribution of input and labels and a loss function, and learning process w which determines how a model / is adapted given task т. The learning path T determines model path changes from initial to final trained model final- We focus on the effect of frozen weights and the trade-off that emerges when sequentially training deep network with weight decay.
Figure 1:
2 Related work
We call the problem setting in this work optimal task tracking which involves training model on sequence of tasks to improve performance on final target task. This strongly resembles the objectives of lifelong learning [. continual learning [ and meta-learning [2] and connects to
many time-dependent learning aspects in relation to neural networks.
We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting [15], critical learning periods [ and time-sensitivity
Time-dependent learning:
of regularization and data augmentation [6] Achille et al. S show that by dropping out certain frequencies during the beginning of training, networks are unable to recover to full performance, even when those frequencies are reintroduced. Similarly, Liu et al. 177 show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards. Recent information-ten theoootii analysis [18] suggests that neural network have distinct learning phases: First phase where network parameters grow in information content, before, in second phase, self-regularizing and pruning away irrelevant information. More commonly known time-dependent or sequential heuristics are learning rate schedules [19] which may even be cyclical 20), regularization annealing such as dropout annealing (1,, student-teacher model transfer [22], and pre-training (23) Finally, our introduced concept of mutually frozen weights also touches upon the lottery ticket hypothesis and may help explain some of its underlying gradient dynamics.
is the technique of training agents by showing them a progression of tasks from easier variants to harder examples attempt to solve the final target task. Both the utility of transferring between similar or dissimilar tasks is debated, with some results pointing towards similarity 22 28 and others towards dissimilarity 29 B0. Meta-
Curriculum learning and generation: Curriculum learning 25 10
learning IS K1 is similarly related by focusing speeding up or enhancing performance on target task by pre-training on other tasks to first learn to learn. Unlike curriculum learning, curriculum generation seeks to find path through task space, generating agent that can solve some final,
held-out task
Invariance- plasttccy wwtetipittttggteerllllleee generalization: Empirically, Mehta et al. 377 showed that ReLU networks with batch normalization trained with gradient descent lead filter-level sparsity. Evci et al. [ demonstrate that training sparse networks usually leads to worse outcomes than without sparsity. On the other hand, Bartoldson al. [39 showed that increased sparsity through pruning can lead to better generalization. Bringing sparsity emergence and difficulty of training together, we will demonstrate how certain types of sparsity can lead to problems in retraining a model after pre-training in the context of mutually frozen weights. From a theoretical viewpoint, Neyshabur et al. 40 41 connect the puth norm of network to generalization via Rademacher bounds, where the path norm is based on the sum across all paths of the product of weights associated to individual paths.

---

3 Optimal task tracking
We are interested in better understanding what influences the outcome of an optimal task tracking problem, i.e. given fixed task sequence how do we train a model to optimize the performance on the final target task. To clarify our further writing, we summarize essential elements involved in sequential learning and visualize the overall setting in Fig. I
task consists of probability distribution p(z; y) of data points and desired outputs as well loss function L: (p, f) + R that maps an input-output behavior of a model to a cost score. The learning mapping (J. + fnew, e-g. stochastic gradient descent, uses the information in task and initial model finit to train a model fieam optimized for the task until convergence. The tasks and learning mappings (Tr,w) E T applied sequentially starting from an initial model finit to produce the final target model furget to be evaluated on the target task Targett
4 Plasticity and frozen weights
With the basic building blocks place, we ask in which cases, even with an infinite number of data points, previous tasks can hinder learning on new tasks. We are interested in understanding what causes model to be more difficult to adapt to a new task than another. This is evidently the case when some of the model weights 0 cannot be updated when training on the new task even when this would yield a better model. Based on this reasoning, we introduce the concept frozen weights, i.e. weights that are unchanged when training
learning task s.t. plasticity is clearly lacking.
Definition (Frozen weights). We define a frozen weight frozen of model f as parameter that is not changed through learning on task т. Byozen (f) frozen (w(
Associated to frozen weights, plasticity captures how well model can adapt to new task.
Definition (Plasticity). We define plasticity as the rate of change of the model parameters subject gradient descent updates P == = [f
The definitions of frozen weights and plasticity beg the question in which scenarios frozen weights can appear. A promising candidate for such a constellation are mutually frozen weights.
Definition 3 (Mutually frozen weights). For model f(0 with parameters 0 and input x, consider n non-overlapping sets of parameters {0,) as mutually frozen for each set, all parameters zero and if each parameter in each set is exclusively multiplied with all other sets of parameters. We call such parameters n n-sparse frozen weights. As the leading example, weights f;, fy. 0g E R are triple-spare sparse mutually frozen if. first. the weights 0,, 0,.0j. O2 only appear multiplied together within the model as Bigk = 0; 0j 0g. S.I. [(0: x) = - D)) where 0 refers to all parameters except 0,, 0f-00 O2 and, secondly if 0; 03 O2 O. Given loss L(0;.r) - TR see that the gradient of 0g is proportional to the remaining weights 0j 0g 0:
dL(0;=) 0L(0;x) (1(0; x) 00 0l(0;x) 05(0; 0,0x
00,
0f(0;z) 00:5k
00:
0f(0;z) 00.jk
Figure 2: Scenario for double- and triple-sparse frozen weights: (Left) Double-sparse frozen weights can occur if, with same index i, zero row vector "u" (sparsity denoted in blue) or zero convolutional filter meets zero column vector "v" zero-entries for the associated channel in the following filters. (Right) Triple-sparse frozen weights can occur if additionally a diagonal matrix of the same index i entry is zero. Scalar multiplication such commonly used in normalization layers can serve the part of the diagonal matrix in this case. We are not currently aware of higher-order frozen weight scenarios beyond triple-sparse frozen weights in available NN architectures.

---

Netra by tmars ner
Epoda
Epada
Figure 3: Existence and dynamics of mutually frozen weights: (Left) Number of frozen weight filter-pairs between consecutive layers as function of the training epoch in ResNet18 8 trained on CIFAR- 10 images with weight decay À =le-3. The top rows indicate that, soon after the beginning of training, mutually frozen weights are created in the initial layers (lighter color). (Middle) Whether mutually frozen weights are created depends on the amount of weight decay. The heatmap shows the overall number of frozen weights across the entire network for different values of weight decay in ResNet18 trained on CIFAR-IO: only higher values weight decay lead the emergence of frozen pairs. (Right) Same center, but for architecture without skip connections (All-CNN 45
Therefore with condition 8, 6, = 0 = 0 and assuming that the partial gradients Ty (f(aa) are bounded, the gradient of the mutually frozen weights 0,, B,. 02 weights are zero. 00.g
L(0:z) and
We note that mutually frozen weights describe n-tuples (n > 1) that "freeze together". As depicted in Fig. 2 we will focus on double-sparse and triple-sparse frozen weights in the remainder of the text. Of course also higher-order freezing involving n-tuples are possible yet do not necessarily contribute further insights. Likewise we deem this less likely to occur in neural network architectures since the influence of individual parameters spreads much further when involving more than three layers.
Example 1 (Linear regression). Linear regression does not have "mutually frozen weights". Given (0;=z) = 0²'x + b, y. E R, a. 0 E IR", we have 0f(!):)] d0 = x. Hence, unsurprisingly, no weights are multiplied with other weights a way that could create mutually frozen weights. Thus we would not expect linear regression to incur loss in plasticity in a sequential task learning setup.
Example 2 (Neural networks). A simple example of triple-sparse mutually frozen weights appears for the model f0,, 0,, x4) 0,0,0yz 2 with x, 0,, 0j. E R. In this case, one can readily see that if = 02 = 02 then the gradients for all weights will also be zero. This can be regarded as a sub-part of a larger piece-wise linear neural network which will be further trained due to mutually frozen weights. Note that with 3 or more weights multiplied as shown, the zero gradient is locally stable (see Sec. 4.7)
The mutually frozen weights condition is particularly relevant for deep neural networks which consist of many pairwise multiplications of weights. In practice paired weights need not be zero exactly but simply have very small absolute values compared to the remaining weights. Other situations of training problems due to insufficient gradients exist such as the vanishing gradient problem [42], wherein the magnitude of the gradient is reduced as it progresses through the network due small weight norms. This problem however appears to be largely solved by methods such as identity connections such in residual networks 43 and normalizations such as batch normalization 44]
From afar it might appear that the answer to the frozen weights problem is simply to avoid them, i.e. to ensure through training model to not end up with frozen weights. However as we will document in the following, weight decay, which can cause frozen weights, often connected to better generalization and thus makes frozen weights non-trivial to avoid. This naturally causes curious trade-off between how we first train a model on a pre- pre-training and subsequently on a target task.
4.1 Existence and weight decay dynamics associated with mutually frozen weights
An important question at this point is whether and how such mutually frozen weights can occur. In Fig. S we show that this indeed can happen in standard training settings (we use the same settings as Mehta et al. (37] Moreover, we notice that the number of mutually frozen weights increases as the training progresses and that the amount of mutually frozen weights created depends on the

---

Mesple
Toogle
1o 3 8
.0033
Figure 4: Dynamics of regular mutually frozen weights: For pairs of column and row filters of an All-CNN trained from scratch on CIFAR-I0 we plot the sum of their associated row and column L1-norm. (Left) Reference dynamies. We select randomly initialized pair of rows and column weights from each layer and plot the dynamics of the sum of their norms of the weights during training, for different values of weight decay. Larger weight decay is associated with lower overall norms (as expected) but in most cases the weights reach a stable equilibrium aside from zero. (Middle) Exactly zero dynamics. We select specific pair of rows and column weights which are initialized to be exactly zero. We find that these double zero weights remain at zero, that the weights remain frozen at zero. (Right) Close zero dynamics. If the weights are very close but not exactly zero, weight decay is likely to move them back to zero (stable equilibrium).
amount of weight decay applied. While weight decay is not usually expected to create sparsity in the weights for shallow model [66). show that for deep models weight decay promotes the existence of mutually- frozznn weights by modifying the learning dynamics in such a way that makes mutually-frozen weights attractor-points of the dynamics.
For the following argument we pick a triplet of weights 0, oz. 0g E which are multiplied for all forward passes of neural network. Furthermore as in the case of ReLU activations. Then we can rewrite the model as f(0; x) = A(0 ; 1z) 0g- + B(O ;z), where f- indicates that @ does not contain 0,, 0g. Oz. With weight decay, weights 0, 0z. Ox evolve as follows:
assume that the network is piece-wise linear
aL(0;z) 0f(0;a) + +X01111k= - [L((0))) x) A(O ; 0,0,0k O/jijk = -1) 0f(0;z) 00./1/k
( 0f(0;x)
+ XB315/kk
02/s/k
Assuming a quadratic loss [(0:x))
(115(0;=) v12. we can further see that 0 -n((A((=:=)00.0) + B(0 : -) A(;; (0a,((;;000000 0-0,00 20.) -=)(((====))0ppp) (B(0;;))) =))(0==z)0,66 + A.,). We calculate the stability dynamics of 0; 0,. Ag linearized around equilib-
rium weights (0,, Gj 0z) (0, 0). With this linearization obtain:
0
4
0
17
0
0
These dynamics have cigenvalues 01,2,3 = -nA. Therefore weights 0, = -, 0 00 0 are locally stable 47). The above dynamics calculation was performed on pair of input-output data x, y but should similarly hold in expectation over the entire dataset. We would expect that triple-sparse mutually frozen weights and even higher-order frozen weights once formed remain stable and are not retrained. For double-sparse frozen weights, we find that, under certain conditions, weight can escape the local equilibrium as shown in a similar analysis in Appendix 6.1
4.2 Invariance-plastiiie connection
We have established that mutually-frozen weights arise naturally during the training of a DNN, and, perhaps surprisingly, while their effect seems negative as they lock some parameters of the network, their presence is associated with weight decay, which has instead an overall positive regularization effect. To clarify the apparent contradiction, in this section we show that mutually frozen weights may help to codify "learned invariances" to transformations of the data. As invariance we define the sensitivity of model to local input changes.
Note that can always locally approximate the loss ((0) quadratically around the equilibrium point.
5

---

Definition 4 (Invariance imput sensitivity). Given a smooth input transformation 4p (x) H B
applied to input with (h= = I, we quantify invariance as Inv (f() 3h
(xp 0). with parameter vectors 0g associated For composite model functions f f fn to each intermediate composite function fi. we can find a connection between invariance and plasticity characteristics of model. For this we first note that via chain-rule, we can decompose the model invariance and model plasticity derivative as follows:
Of (z). 0) Of(pn(z).0) 1f 0fn-1
(f)())()))0)) 0f (p)((())).) (p)(2)
Oh
Of:
a,
@pn(z)
Ōh
0 (f 0) Of(px(z),0)
0f+ (PL)(()()).0)) 0f((p(()).))
00,
0fn-1
0f.
00,
Shared
From the above we can see that the input sensitivity derivative shares many factors with the model derivative. The earlier the intermediate composite function is applied to the input, the more overlap in factors we have. Thus, for compositional models, input sensitivity and plasticity are naturally linked. Lemma (Plasticiy--invriianc connection). We assume a composite model f(0, (Pn(x)) = (a fn-1 10,-1, fi (0100 ( (x))), 0) and that that the factors unique to the input sensitivity and unique to the model derivative are non-zero (f.11(9)(0)8)) (f(())()) (pp(a)
Oqula)) 8h + 0 and
0f.
(f((0.(2).0) * O. Then a lack of plasticity in some direction de implies that there exists an in- 88,
put transformation s.t. (f(pn(2).0) dh = 0 in which the model is invariant to transformation (pt- 8h
Proof. Knowing that the plasticity zero in d0 direction and that the individual, non-shared parameter derivative factors are non-zero implies that the shared factors have non-empty nullspace. Since the shared factors also appear for the model input sensitivity derivative we can make statement about possible implied invariances. Case I: The individual factors of the input have a range space that does not overlap with the nullspace of the shared factors. Then we can deduce that the individual factors of the input sensitivity derivative have non non-emply null space. Case 2: The individual input factors have a range space that does overlap with the nullspace of the shared factors. For both cases, there exists a transformation yhh with direction dh for which the model input derivative vanishes 8f there is either nullspace in the shared or individual factors that projects derivatives zero D
ah dh = 0 since
The above shows that mutually frozen weights are naturally linked invariance if the zero parameter derivative arises in shared derivative factors, which is the case for mutually frozen weights. The assumption of shared factors becomes more and more restrictive the later an intermediate composite function is applied such that we would expect that later layers are easier to retrain as observed by Achille et al. [.. In the limiting case of linear regression, there are no shared factors and we would, in general setting, not expect any relation between input sensitivity and parameter gradients. For very deep networks (especially without skip connections), the relation should most pronounced for the layers that are closest to the input since more factors are shared.
5 Experiments
We explore the ideas and connections of mutually frozen weights and invariance and their impact in number of sequential learning settings across different datasets, network architectures, learning rates, and weight decay settings. Detailed specifications will be provided in the appendix. The following subsections are meant to give further evidence for the following statements:
Mutually frozen weights occur and are different from weights that zero (sparse) but not mutually frozen.
Both sufficiently high learning rates and weight decay essential for the occurrence of mu- tually frozen weights and final test performance as tested across two different architectures. Mutually frozen weights the beginning of training be harmful yet can be removed through a "resetting intervention".
Across a number of task changes, removing frozen weights is beneficial as long as sufficiently many retraining samples are available.
We provide an analysis summary relating frozen weights, invariance, and performance.

---

Table 1: Mutually sparse weights fundamentally different from sparse weights. We compare the performance of All-CNNs 45 trained on CIFAR-I0 with/without data augmentation of horizontal flip and random shift of magnitude 0.5 while initializing the networks with different types of sparsity, Mono, double-sparse, and triple-sparse weights are randomly set to zero as described in Appendix 6.5 to ensure similar levels of sparsity. Displayed across 3 random seeds. Triple-sparse weights: We randomly select filter pairs of layers and batch normalization (BN) (triplet) that are directly multiplied together and all associated weights within those filters to zero. In terms of matrix operations this would be equivalent to setting a row of the first matrix and the associated column of the second matrix well as its BN weight to zero. Double-sparse: For this case we randomly set any weight to zero to arrive at the sparsity probability in the top row. We exempt batch normalization weights from being set to zero since these weights are naturally tied to many other weights thus frequently lead to triple-frozen weights. Mono-sparse: For this case mimic the double-spee double-sparse case. However to ensure that no weight is multiplied with paired zero- prro-filter, we add an identity initialization to existing filters which barely changes the overall sparsity level. It appears that training sparse networks in itself is not difficult, training double and triple-sparse weights is.
mean accuracy results with standard deviation
Type/ / Average sparsity at init.
0.5
0.8
0.92
0.98
Mono-sparse accuracy
92.1 + 0.2
91.8 + 0.4
92.4 0.1
92.1 + 0.0
Mono-sparse, no data-aug.
88.6 + 0.1
88.6 + 0.2
88.5
87.1
Double-sparse accuracy Double-sparse, no data-aug.
92.1 0.1
89.2
77.7
67.0 9.0
88.6 - 0.3
5.1
6.0
63.4 23.2
Triple-sparse accuracy Triple-spars,, no data-aug.
87.8 1 4.9
0.5
6.62
14.0
86.8 + 0.3
0.2
7..1
16.0 + 10.4
Weight deeay effect- blurred to sharp CIFAR-10
Learning rate effect-
blurred to sharp CIFAR-10
0.9
Tist aveuracy
0.8
Test aceurayy
0.8
0.7
0.7
0.005
0.0005
0.05
0.05-0.0005
0.6
0.001
0.0001
0.6
0.0005-0.05
0.0005
0.0001
0.001
0.5
0.005
0.5
Allcony
R8snet18
Resnet18
Allcoue
Network Architecture
Network Architecture
Figure 5: In a negative pre- pre-training setting, decreasing the pre-training weight decay and pre- training initial learning rate helps since it helps to avoid mutually frozen weights For pre- training on blurred CIFAR-I0 images and then switching to regular sharp images, we show the test loss for the ResNet- 18 and All-CNN architectures across different initial learning rates and weight decay settings. We apply exponential learning rate decay tuned to the initial learning rate. (Left) Varying weight decay across pre- rre-training while keeping the learning rate schedule constant shows that too much weight decay hurts and too little is similarly detrimental. For negative pre-training turns be better have little weight decay the blurred images and more on the standard sharp images. (Right) Varying initial learning rates shows that, up to point, higher initial learning rates yield better retraining results. Even better however is again pre-training with a low initial learning rate and retraining with a higher learning rate.
5.1 Mutually frozen weights exist and are differrnt from regular sparse weights
To begin our experiments, we want to demonstrate that mutually frozen weights indeed exist in deep neural network training and that mutually frozen weights are qualitatively different from "regular" sparse zero weights that are not paired with other zero weights. Evei et [X] showed that convolutional networks that are sparse can indeed be hard to train to similar test performance as
7

---

Table 2: Retraining performance with/without frozen weights intervention: We report test ac- curacies with and without resetting for the bascline case before retraining with mean and standard deviation across 3 seeds as well as the mean number of double/triple frozen weights (FW) (see Ap- pendix 6.2p. srate-of-the-ar results in bold. For most tasks with sufficient data, intervening/resetting by removing frozen weights allows pre-trained network to better adapt to new task. For few samples, we hypothesize that the constraints provided by pre- pre-training training are essential when retraining.
Task change
Model
Reset Test acc.
Test acc.
Double/Triple FW
CIFAR- 10
CIFAR- 100
10
ResNet18
74.2 = 0.3
73.1 0.5
60k/50k
CIFAR-I0 10 49
+ FashionMNIST
ResNet18
96.9 = 1.3
95.6 0.5
100k/90k
MNIST 50
FashionMNIST [5]
ResNet18
97.1 + 1.0
96.7 1.1
200k/170k
SVHN 5
CIFAR-10 4b
ResNet18
88.9 + 0.2
87.6 + 0.1
3.5M/3.5M
ImageNet- MongeNet-Fasttionnie FisS 51
ResNet50
97.8 + 1.6
97.2 0.2
90k/70k
I [
ResNet50
9.1
9.1
90k/70k
ImageNet-EEMNISTT (letters)
ResNet50
95.
95.3
90k/70k
ImageNet-<CIFAR 100
49
ResNet50
84.5 0.2
84.2 0.3
90k/70k
ImageNet-RESISCA.
55
ResNet50
90.2 0.3
89.7 0.1
90k/70k
ImageNet-<Patch Camelyon
56
ResNet50
89.3 0.6
87.8 1.4
90k/70k
networks that are not sparse. However they did not provide a mechanism of how sparsity developed or that double-sparse and triple-sparse weights in particular make it hard to train.
In Fig. 3 and 4 we count a weight 3-tuple as nutually frozen if the absolute value of each weight is smaller than le-4. We find that when starting training with high learning rate and using non- negligible weight decay the number of mutually frozen weights steadily increases with the number of trained epochs as shown in Fig. B While weight decay affects all parameters, as shown on the left of Fig. triple-sparse weights remain stably at zero as shown in the middle and right of Fig.
To establish that mutually frozen weights are meaningful concept, we compare the performance when training randomly initialized All-CNNs 45 at different sparsity levels on CIFAR- 10 image classification when altering the initial number of small weights. As a point of reference we measured that around 50% of weights fall below our sparsity threshold even without setting any weights to zero. We create mutually frozen weights by randomly sampling entire filters, associated filter columns and associated batch normalization weights to be set to zero. "Double-sparse" weights we create by randomly setting weights in a filter to zero but excluding any batch normalization weights. This ensures that triplets of zero weights are very unlikely. Finally, we create "mono-sparse" weights by following the same procedure as for "double-sparse" weights but adding identity filters to the existing filters by "resetting" (subsection 5.2). Thus we can avoid that any weight has an associated filter which is completely zero. As shown in Tab. [ training network is hardest with triple-sparse frozen weights, less difficult with double-spart sparse weights and not difficult with mono-sparse weights. We deduce that, in this setting, sparsity in itself does not necessarily have strong effect training performance outcomes, triple-sparse or double-sparse weights however do.
5.2 Interventions Resetting mutually frozen weights Adding scaled identity
To further test the effect and removal of frozen weights, we devise a simple "resetting" intervention. The intervention adds small constant times identity initialized weights, similar to the idea of DiracNets [8), to all filters. To the convolutional filters, we add very small zero-mean Gaussian random noise with standard deviation 1c-5 to avoid identical updates by breaking symmetry of updates. Batch normalization (BN) multiplicative weights cither, depending on the magnitude of the intervention, increased by the same small constant across all weights or set directly to 1.0 for all BN weights. By design, this removes all mutually frozen weights since weight of one filter is associated to only zero weights of an adjacent layer the effect of which is also shown in Tab.
To demonstrate the effect of the proposed "resetting" intervention, we apply it on number of pre-training to retraining task changes as recorded Tab. 2 As long as sufficiently many samples are available for retraining, this can lead to a consistent increase in performance, albeit small increase if features do not have to be adapted much. For domains that are sufficiently different, we can achieve meaningful gains such as when moving from ImageNet to FaSHionMNIST. where we surpass the
8

---

Table 3: Re-training performance with and without the use of trainable batch normalization weights: We report test accuracies with and without using the "affine" option to use trainable multiplicative and additive scalars in batch normalization We pick the seminal blurring transition [ as well as a belated data-augmentattin of up to 50% random shift [I6] to showcase the influence of frozen weights. In this setting, we compare results of ResNet18, AlI-CNN, and AlexNet [58] architectures, where we especially note that AlexNet does not have multiplicative normalization weights, e.g. via batch normalization. Intriguingly, without the "affine" option, networks are less prone to negative pre-training yet also perform less well as shown by the performance after "resetting". We calculate the mean number of double- and frole-frozen weights without intervention and before the second task (FW) as described in Appendix 6.2
CIFAR-10 augmentation transitions.
CIFAR-10 Task change
Model
Reset Test acc.
Test acc.
Double/Triple FW
Blurred- Regular sharp
ResNet18
89.0 0.5
87.2 0.2
0.4M/0.3M
Blurred- + Regular sharp
No "affine" R18
87.4 0.5
85.1 3.9
0/0
Blurred- + Regular sharp
All-CNN
88.7 :
83.1
60k/60k
Blurred Regular sharp
No "affine" A-CNN
85.9 3.0
88.0 +: 0.1
0/0
Blurred Regular sharp
AlexNet
87.3 + 0.2
87.5 0.1
0/0
Regular Random shift
ResNet18
94.0 0.1
93.0 0.3
90k/80k
Regular + Random shift
"affine" RI8
93.0 0.0
93.0 0.1
0/0
Regular + Random shift
All-CNN
90.5 1.3
89.7 + 0.1
4k/4k
Regular + Random shift
No "affine" A-CNN
90.5 - 0.2
90.4
0/0
Regular + Random shift
AlexNet
86.2 0.22000 0.6
86.5 = 0.5
0/0
57 and reach 97.8% accuracy. To demonstrate the crucial role that BN weights play, we further provide experiments with and without intervention in Tab. 3 on the "critical learning periods" task of pre-training on blurred images then switching to regular sharp images and the time-sensitive data augmentation task of introducing random shift only after having trained without it. We note that without BN weights, time-sensitive effects are greatly reduced, however the maximal performance is reduced. As a fine-tuning baseline for all experiments, we follow the comprehensive visual benchmark approach M. Specific details such as studying the number of samples vs. retraining performance are listed in the appendix.
previous state-of-the-art of 96.9% accuracy
5.3 Learning rate and weight decay mutually frozen weights
We might wonder whether training could simply be done without weight decay trained with small learning rates to avoid the formation of many mutually frozen weights. In a negative pre-training setting of pre- training blurred and retraining sharp CIFAR-10 images [8) we trained either ResNet18 8 [ or All-CNNs 451. With the currently popular setup employing weight decay we find pre-training with low weight decay or initial learning rate but retraining with high initial learning
rates and sufficiently high weight decay leads to better test performance as shown in Fig. S
5.4 Analysis: Invariance-plasticity connection
A compelling part of weight decay is the potential double-edged nature it has which leads to both better test performance yet also to mutually frozen weights. Fig. we showease the relationship of performance the number of frozen weights the beginning and the end of training. Likewise, we highlight the relation between invariance and performance on the very same experiments. To quantify invariance we utilize the Fisher discrimation score [59 60) and adapt it analysis of high-dimenssinal features extracted before the last layer of a Resnet18 where we compute mean ja and variance Σ for each class individually. We not only interested in absolute invariance but also care about how invariant a model is within a class but also how this variance is scaled relative to the distance between classes.
2 between
(w0 (MT
100 IT Eo + En) 170)
(7
S
0 within
wT (E, )
(1
As shown in Fig. when retraining. After retraining however, a low but non-zero initial frozen weight count is best while
we find that zero initial frozen weights are best to later achieve good performance

---

0.7
Model
0.6
ott
After zetraining
Before retraining
Test loss
Test loss
0.5
0.5
Model
0.4
After retraining
Before retraining
0.3
0
1000
2000
3000
4000
5000
2
4
6
# of frozen weights
Fisher discriminative score
trade-off: For the CIFAR- 10 dataset, aggregate results across different weight decay and learning rate settings when pre- pre-training a ResNet18 either on a) blurred or b) 10% of training then fine-tuning the full standard CIFAR-10 10. We contrast the measured statistics between before fine-tuning on CIFAR-10 10 and after fine-tuning on CIFAR-10 10 but always report the final target standard CIFAR-10 test loss (lower is better). (Leff performance-frozer weights) We find that, before fine-tuning, more frozen weights relate to lower attainable retraining performance. After fine-tuning it appears be best to have some, but not too many, frozen weights such that a small number of frozen weights leads to the best performance. (Right performance- invariance) We measure the Fisher discriminative score (eq. D before and after fine-tuning. Before fine-tuning more invariance, as part of a fareto-front, is associated with lower final performance whereas after fine-tuning higher fisher score relates better final performance. The results hint at possible trade-off between the performance on the pre-training task, which can create higher initial invariance and create frozen weights, and the ability retrain to the target task.
Figure 6: Plasticit-invariiann
more frozen weights decrease performance. When regarding Fisher score invariance, the models that are most invariant before retraining appear to incur a performance penalty after retraining, while the models that are more invariant after retraining outperform their less invariant counterparts.
Discussion and conclusion
We started our investigation intrigued by the initial observation that deep networks can exhibit critical learning periods (8] such that pre-training on blurred images can lead to worse performance when retraining sharp images. From this starting point we showed that mutually frozen weights play role making harder to retrain neural networks by limiting how gradients flow in a network which qualitatively different kind of sparsity that, to our knowledge, has not been discussed before.
Mutually frozen weights may not occur in all commonly used networks. Likewise is possible that mutually frozen weights can be avoided in the future by improving upon weight decay. Training of DNN involves numerous other factors which may overshadow the effect of frozen weights and make the analysis more difficult. Due to the potential generality of our proposed concepts it is likewise always helpful add further datasets and data and model types to assess whether the same effects present. The scope of this paper is to further the understanding of transfer learning. As such it may have an impact on any setting associated to transfer learning and may help make deep learning solutions more accessible.
proookig about nutually frozen weight resetting that very sparse network, whose mutually frozen weights are completely removed, can be trained to the same accuracy network trained from scratch. Furthermore, retraining even ImageNet pre-trained models can often be slightly improved by removing existing frozen weights and even helped obtain Better understanding mutually frozen weights may enable to to get the most out of pre-trained networks when retraining them on datasets that require a focus on different features. Finally, we touched upon a potentially fundamental trade-off in sequential learning between invariance and plasticity for deep networks. We hope this may spark new research in sequential learning how best to train a network sequentially, which types of networks are especially conducive to sequential learning and which trade-offs may be inevitable in sequential learning.
results.
10

---

References
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15 (1-2):25-46 46, 1995.
[2] Daniel Silver, Qiang Yang, and Lianghao Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.
[3] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology. 4:504, 2013.
dilemma: Investigating
[4] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? Proceedings the thirteenth international conference on artificial intelligence and statistics, pages 201-208. JMLR Workshop and Conference Proceedings. 2010.
[5] Minyoung Huh, Pulkit Agrawal, and Alexei Efros. What makes imagenct good for transfer learning? arXiv preprint arXiv: 1oxii:66088 1608/0608/1111 0868008644, 14,14,77 2016.
[6] Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. A linearized framework and new benchmark for model selection for fine-tuning. arXiv preprint arXiv:2102.0000 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
[8] Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv: / 1,8856, 2017.
[9] Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? arXiv preprint arXiv:2006. 10455, 2020.
[10] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41 41-48. ACM, 2009.
[11] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz- Rodríguez. Continual learning for robotics. arXiv preprint arXiv: 2019.
[12] Sehastian Thrun and Lorien Pratt. Learning learn. Springer Science & Business Media, 2012.
[13] Tom Schaul and Jürgen Schmidhuber. Metalearning, Scholarpedia, 5(6):4650. 2010.
[14] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference Artificial Neural Networks, pages 87-94. Springer, 2001.
[15] Robert M French. Catastrophic forgetting in connectionist networks. Trends cognitive sciences, 3(4): 128 135, 1999.
[16] Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep net- works: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems, pages 10677-10687, 2019.
[17] Shengchao Liu, Dimitris S. Papailiopoulos. and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. CoRR, abss19006.02613. 2019.
[18] Ravid Shwartz-Ziv and Nafaai Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 00810. 2017.
[19] Christian Darken and John Moody. Note on learning rate schedules for stochastic optimization. In Advances in neural information processing systems, pages 332-838, 1991.
[20] Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 644-472, IEEE, 2017.
[21] Steven J Rennie, Vaibhava Goel, and Samuel Thomas. Annealed dropout training of deep networks. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 159- IEEE, 2014.

---

[22] Renato Vicente and Nestor Caticha. Functional optimization of online algorithms in multilayer neural networks. Journal of Physics A: Mathematical and General, 30(17):L599, 1997.
[23] Dumitru Erhan, Antoiee Manzagol, Yoshua Bengio, Sany Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect of unsupervised pre-training, In Artificial Intelligence and Statistics, pages 153-160, 2009.
[24] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprint arXiv: 8833036335 2018.
[25] Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
[26] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-leariig. Proceedings of the IEEE International Conference on Computer Vision, pages 6430-6439, 2019.
[27] Haitham Bou Ammar, Eric Eaton, Matthew Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure of mdp similarity for transfer in reinforcement learning. In Workshops at the Twenty-Eigtth AAAI Conference on Artificial Intelligence, 2014.
[28] Hippolyt Ritter, Alcksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pages 3738- 3748, 2018
[29] Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv: 2018.
[30] Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv: 1et::::0800099999 2019.
[31] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
[32] Nicolas Heess. Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1 707.02286, 2017.
[33] Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. Proceedings of the 34th International Conference on Machine Learning-Volan Volume 70, pages 1311-1320. JMLR. org, 2017.
[34] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse lcarning environments and their solutions, 2019.
[35] OpenAT, Tige Akkaya, Marein Andrychowicz, Macick Chociej. Mateusz: Litwin, Boh McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Rihas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand, 2019.
[36] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization, 2019.
[37] Dushyant Mehta, Kwang In Kim, and Christian Theobalt. On implicit filter level sparsity in convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 520-528, 2019.
[38] Utku Evci. Fabian Pedregosa. Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:1906.10770 10732, 2019.
[39] Brian R Bartoldson, Ari S Morcos, Adrian Barbu, and Gordon Erlebacher. The generalization-stabll stabillity tradeoff in neural network pruning, arXiv preprint arXiv:1906 09990000722.. 2019.
[40] Behnam Neyshabur, Ryota Tomioka. and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pages 1376-1400 PMLR, 2015.
[41] Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: Path- normalized optimization in deep neural networks. arXiv preprint 100.v1150. 2015.
12

---

[42] Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal Uncertainty, Fuzzinesx and Bnowledge-Baseee Systems, 6(02):107-116. 1998.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[44] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[45] Jost Tohias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6000 2014.
[46] John Moody, Stephen Hanson, Anders Krogh, and John A Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4 4(1995):550-957, 1995.
[47] Philip Hartman. A lemma in the theory of structural stability of differential equations. Proceedings of the American Muthematical Society, 11(())610-66) 620. 1960.
[48] Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip- connections, arXiv preprint 176.088706.00888, 2017.
[49] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[50] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( I 1):2788-3324, 1998.
[51] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17Xiv:1708.07747,
[52] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[53] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 2018.
[54] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Seeworks pages 2921-2926. IEFE, 2017.
[55] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification; Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865 1885, 2017.
[56] Bastiaan Vceling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In International Conference on Medical image computing and computer-assiiseed intervention, pages 210-218. Springer, 2018.
[57] Muhammad Suhaib Tanveer, Muhammad Umar Karim Khan, and Chong Min Kyung. Fine-tuning darts for image classification. In 2020 25th International Conference Pattern Recognition (ICPR), pages 4789-4796. IEEF, 2021.
[58] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 2012.
[59] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179-188, 1936.
[60] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K.R. Mullers. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing IX: Proceedings of the 1999 JEEE Signal Processing Society Workshop (Cat. No.98TH84688, pages 41 18- 1999. doi: 10. 1999NNN8.1999.788..1.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.00 1912.01703, 2019.
[62] HM Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. arXiv preprint 200.0:2007.033477 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We back up our claims via the main body of the work well numerous figures and tables.
(b) Did you describe the limitations of your work? [Yes] We discuss limitations throughout the text and in particular in the penultimate paragraph of the paper.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes, however there are no clear societal implications since this work more foundational than applied to problem with societal consequences.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] Yes, most theoretical claims are in section 4 for which we state assumptions.
(b) Did you include complete proofs of all theoretical results? [Yes] Likewise, we include proofs to the statements we make in the theoretical part of the paper in section 4.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Depending the type of plot these error bars were standard error statements, boxplot shapes or even individual points on a scatterplot. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We do include the use of resources but do not have precise estimate of the total amount of hours trained.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The most important assets that we use are datasets and code frameworks such as Pytorch which we cite.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as a URL? [Yes]
Yes, we include our code base, to allow others to easily reproduce our results.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vdNaXbyF6kM", "reviewer": "Reviewer_eMiV", "summary": "The paper studies the \"optimal task tracking\" problem under the training setting where we use the \"weight decay\" regularization.\nMore specifically, the paper studies the relationship between plasticity-stability tradeoff and shows that a consequence of training with (large enough) weight decay is the \"mutually frozen\" weights that can damage the plasticity of the network. Moreover, the paper introduces an intervention to reset the mutually frozen weights and improve the performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Overall\n**Originality: Are the tasks or methods new?** Yes    \n**Quality: Is the submission technically sound?** Partially  \n**Clarity: Is the submission clearly written?** Yes   \n**Significance: Are the results important?** Partially\n\n------\n### Strengths\n**(1)** In general, I appreciate the works that aim to understand the behavior of neural networks rather than jumping into problem-solving and beating the benchmarks. This paper belongs to this category, and I appreciate the novelty and the approach of the paper.\n\n**(2)** The overall theoretical reasoning seems sound to me.  \n\n**(3)** The authors provide the code and details for reproducibility.\n\n-----\n\n### Weaknesses\n\n**(1)** While the introduced concept (mutually frozen weights) seems very interesting, I believe the study could benefit if other contributing factors to this concept were studied. For instance, the authors study the impact of weight decay and (to some degree) learning rate (Sec. 5.3) but since the aim is to study the training dynamics. But, several other common factors may or may not contribute to the impact of mutually frozen weights. While I understand it is infeasible to study every contributing factor, the paper could at very least study the prominent factors in training dynamics such as BatchNorm, Dropout, and Optimizer that is shown to have an impact on training dynamics and stability-plasticity in continual learning [1]. More specifically, [2] shows that L2 regularization with Batch Normalization leads to an Exponential Learning Rate Schedule that depends on the momentum and weight decay rate. According to the code that the authors provided, most of the models use Batch Normalization layers. Hence, I believe an additional study on these factors is needed to improve the paper's insights.\n\n**(2)** In the continual learning literature, weight decay is known to generally hurt the performance regarding the stability and catastrophic forgetting [1,3], which contradicts the paper's claim. While the benchmarks in those studies are different, the problem setting seems similar (i.e., distribution shift). Could the authors discuss the relationship of those findings to the findings of their paper? \n\n**(3)** Regarding Table 4, I have three comments/questions:\n   *(3.1)* If I understood correctly, the number of frozen weights is very small compared to the total number of parameters in the network.  How this small number of frozen weights can make such a gap (e.g., ~6.6% on CIFAR-10 -> CIFAR-100)? According to the appendix, the reported numbers are calculated for only one specific learning rate. Could it be because of this?\n   *(3.2)* It is difficult to find the real impact of the intervention on performance, given that only one hyper-parameter is used. Could the authors provide the results for at least a few of the rows (e.g., one from each category) where a grid (even small) is used to find the best parameters? The rationale is some reported numbers are lower than the typical performances reported in the other studies. For instance, training a ResNet18 on CIFAR-100 from scratch should give above 70% accuracy. While one can argue that they might use different techniques for improving performance, I suppose given this sub-optimal training setting, it is clear how much real gain the proposed intervention provides.\n\n**(4)** [Misc]: In addition to some grammatical errors (e.g., L38), I believe some figures (2, 3, 4)  are misplaced and very far from where they have been referenced. Also, the y-axis titles of Fig. 4 are not entirely readable. It could improve the readability of the paper if the authors could improve the figures. However, I should note that this did not contribute to my decision and is only a suggestion.\n\n[References]   \n*[1] Mirzadeh, Seyed Iman, et al. \"Understanding the Role of Training Regimes in Continual Learning.\" Advances in Neural Information Processing Systems, 2020.*   \n*[2] Li, Zhiyuan, and Sanjeev Arora. \"An Exponential Learning Rate Schedule for Deep Learning.\" Eighth International Conference on Learning Representations, 2020.*   \n*[3] Lange, Matthias De, et al. Continual Learning: A Comparative Study on How to Defy Forgetting in Classification Tasks. 2019.*\n\n\n----- Update (After Rebuttal) ----   \nI believe the authors have addressed most of my questions and comments. Hence, I increase my initial score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "r1WZzrUuha4", "reviewer": "Reviewer_RXdT", "summary": "In this paper, the authors present a mechanistic phenomenon linked to plasticity problems incurred in sequential task learning. They call this phenomenon \"mutually frozen weights\", describing a situation where triples (or higher order tuples) of weights that multiply each other in a system, all end up with zero value, which prevents error gradients from influencing their value when learning new tasks. They then argue that such a situation arises in real world setting, and explain how weight decay can promote such a situation. They also demonstrate that such frozen weights play a role in input invariance, and thus should be considered in a tradeoff perspective. Finally, the present a \"resetting\" method that helps mitigate the problems brought by mutually frozen weights, and perform some numerical experiments to demonstrate the efficacy of their method.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: To my knowledge, the concept surrounding mutually frozen weights, and their link to learning trajectory deficiencies in sequential task learning, is novel. The concept is sound and the identification of its impact is new.\n\nQuality & Clarity: The paper is thorough and takes the necessary steps to formally define mutually frozen weights, and test for their impact on training. Experiments are well-documented and follow quality guidelines. However, a number of questions remain unanswered with respect to how extended mutually frozen weights might arise.\n\n1- It is not clear to me why the concept of mutually frozen weights needs 3-tuples of weights to be zero, and not 2-tuples. In Definition 3, couldn't the multple \\theta_j*\\theta_k be replace by a single zero weight ? Maybe I am missing something here, but it seems a 2-tuple is enough to get the phenomenon. In any case, the reason behind why 3-tuples are needed is not explicitly explained, which would greatly improve clarity.\n\n2- In Definition 3, it is stated that mutually frozen weights are 3-tuples that \"only appear multiplied together\" in the network. In practice, this seems like a very special situation in deep networks, such as in filters, etc. (as treated in experiments). Indeed, typical layer weights will multiply several other weights and thus, gradients will be able to reach them in principle. Is the phenomenon of mutually frozen weight restricted to this situation? (maybe I am missing something here) Also, is the mechanism of hindered sequential task learning observed in the literature restricted to architectures that allow mutually frozen weights? If not, a discussion about the limitation of  mutually frozen weights should be present.\n\n3- Table 1 describes an experiment with frozen weights AND perturbation meant to test invariance (horizontal flip and random shift of magnitude). Why combine both these effects ? What happens to training with different sparsity scenarios when no perturbations are introduced ?\n\n4- Ablation experiments that measure gradients to mutually frozen weights before and after resetting would strengthen the conclusion that the phenomenon, and the solution, are mechanistically related to improvements.\n\nSignificance:\nIf clarity issues described above are resolved, I believe this contribution is significant in understanding mechanisms that influence training trajectories for sequential task learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not notice any discussion of potential societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5hrs"}, "extra_text": {}}, {"review_id": "CnxUOpLcm2F", "reviewer": "Reviewer_tyuV", "summary": "The authors of 'On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning' introduce 'mutually frozen weights', a new metric for evaluating how well neural networks trained on one task can learn a novel task. They provide preliminary theoretical analysis relating their new metric to weight decay and show how it can be manipulated to facilitate sequential task learning.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors define the phenomena of 'mutually frozen weights' as a triplet of parameters that appear on the neural function only as their product and when they are all equal to zero, they will stay zero for the rest of training. They authors also claim that with the MSE loss and weight decay, all of the parameters being zero is a stable attractor of the training dynamics. \n\nIt is very unclear to me how often does it happen when three (or more) parameters only appear as their product. This is a critical point of their work as the 'mutually frozen weights' only happens in that scenario. Interestingly enough, fully connected neural network (i.e. Weights multiplication and bias addition followed by non linearity) do not have this feature, and neither do standard Convolutional neural network composed (they don't even have two parameters only appearing as their product), and even with Batch Norm, I believe we are still left with only two parameters sequentially multiplied. I may be missing something, but still, the authors should provide a clear example when this feature occurs in a common neural architecture rather. \n\nThe theoretical analysis in parts 4.1 seems sound aside from the reliance on the MSE loss and weight decay. While I do understand why  all thetai/j/k = 0 is a fixed point in the non weight-decay case, it is not clear why it would occur if it is not an attractor (which it isn't since the eigenvalues of the fixed point would be zero if lambda = 0). An extension to more common loss functions (like, softmax cross entropy) and an explanation of what happens without weight decay (as weight decay is not a very popular regularization measure anymore) would enhance the applicability of this analysis.\n\nThe relationship to sequential task learning is also not so clear. In their analysis in 4.2, the authors show that weight gradient and gradients to temporal change in input have common factors and conclude that they are thus interlinked. Due to the familiarity of the community with vanishing gradients (and why we all now use relu instead of tanh and sigmoid) I believe this could simply be stated in a couple of sentences. Devoting an entire section and introducing it as a lemma seems odd, especially considering that the authors definitely lacked space judging from their 2nd and 3rd figures which are hard to read and should be at least twice their current size.\n\nDue to lack of obvious applicability of this work, I recommend it be rejected from Neurips 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a theoretical analysis of neural networks and I do not see any negative societal impact from their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the learning dynamics in deep networks by making a novel observation regarding weight decay as mutually frozen weights and their role in generalization. The paper initially received reviews that tended towards rejection. The reviewers had difficulty understanding some details and were concerned whether the results will hold true in real-world settings. The authors provided a thoughtful rebuttal that addressed the reviewers' concerns. The paper was discussed and all the reviewers updated their reviews in the post-rebuttal phase. All reviewers switched their score to weak acceptance (note one reviewer still has a score of 4 in the main review, but they have switched to 6 in comments). AC agrees with the reviewers and suggests acceptance. However, the authors are requested to look at reviewers' feedback and incorporate their comments in the camera-ready.
Decision: Accept (Poster)