On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning
Year: 2021

Abstract:
Plastic neural networks have the ability to adapt to new tasks. However, in a continual learning setting, the configuration of parameters learned in previous tasks can severely reduce the adaptability to future tasks. In particular, we show that, when using weight decay, weights in successive layers of a deep network may become “mutually frozen”. This has a double effect: on the one hand, it makes the network updates more invariant to nuisance factors, providing a useful bias for future tasks. On the other hand, it can prevent the network from learning new tasks that require significantly different features. In this context, we find that the local input sensitivity of a deep model is correlated with its ability to adapt, thus leading to an intriguing trade-off between adaptability and invariance when training a deep model more than once. We then show that a simple intervention that “resets” the mutually frozen connections can improve transfer learning on a variety of visual classification tasks. The efficacy of “resetting” itself depends on the size of the target dataset and the difference of the pre-training and target domains, allowing us to achieve state-of-the-art results on some datasets.

heading: 1 Introduction
text: Lifelong learning [1, 2] holds both the promise of benefitting from past experiences and the challenge of having to continually adapt to new problem settings. The characteristics of this intriguing sequential learning problem make it a balancing act between retaining knowledge and adapting to new experiences referred to as the stability-plasticity dilemma [3]. Similar to a coach or teacher, we want to understand when a sequence of tasks helps or hinders further learning. Clearly, some pre-training tasks are highly beneficial [4–7] while others can hurt performance [8, 9].
The existing literature on pre-training and curriculum learning sheds some light on the desired characteristics of sequential learning. In some cases, following a sequence of learning tasks can lead to better results than simply training on the target task from scratch [10]. Work by Achille and Soatto [8] contrasts this picture by highlighting that neural networks can suffer from pre-training on some tasks. In an extreme case, when learning data is corrupted for a sufficiently long period, even switching back to clean data does not allow the neural network to recover its original performance.
Contribution: We formalize the basic ingredients of sequential task learning, introduce mutually frozen weights and establish a connection between plasticity and invariance of deep networks. To understand mutually frozen weights, we investigate their weight update dynamics, show that such frozen weights are different from regular sparse weights that are not connected to other sparse weights and that frozen weights can lead to a decrease in performance when retraining on a new task. Based on this analysis, we show that applying an intervention which “resets” frozen weights can improve retraining performance on a number of tasks as long as sufficiently many samples are available for retraining and even achieve state-of-the-art results on FashionMNIST image classification.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 2 Related work
text: We call the problem setting in this work optimal task tracking which involves training a model on a sequence of tasks to improve performance on a final target task. This strongly resembles the objectives of lifelong learning [1], continual learning [11] and meta-learning [12–14] and connects to many time-dependent learning aspects in relation to neural networks.
Time-dependent learning: We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting [15], critical learning periods [8], and time-sensitivity of regularization and data augmentation [16]. Achille et al. [8] show that by dropping out certain frequencies during the beginning of training, networks are unable to recover to full performance, even when those frequencies are reintroduced. Similarly, Liu et al. [17] show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards. Recent information-theoretic analysis [18] suggests that neural network have distinct learning phases: First a phase where network parameters grow in information content, before, in a second phase, self-regularizing and pruning away irrelevant information. More commonly known time-dependent or sequential heuristics are learning rate schedules [19] which may even be cyclical [20], regularization annealing such as dropout annealing [21], student-teacher model transfer [22], and pre-training [23]. Finally, our introduced concept of mutually frozen weights also touches upon the lottery ticket hypothesis [24] and may help explain some of its underlying gradient dynamics.
Curriculum learning and generation: Curriculum learning [25, 10] is the technique of training agents by showing them a progression of tasks from easier variants to harder examples to attempt to solve the final target task. Both the utility of transferring between similar or dissimilar tasks is debated, with some results pointing towards similarity [26–28] and others towards dissimilarity [29, 30]. Metalearning [13, 31] is similarly related by focusing on speeding up or enhancing performance on a target task by pre-training on other tasks to first learn to learn. Unlike curriculum learning, curriculum generation seeks to find a path through task space, generating an agent that can solve some final, held-out task [32–36].
Invariance-plasticity-generalization: Empirically, Mehta et al. [37] showed that ReLU networks with batch normalization trained with gradient descent lead to filter-level sparsity. Evci et al. [38] demonstrate that training sparse networks usually leads to worse outcomes than without sparsity. On the other hand, Bartoldson et al. [39] showed that increased sparsity through pruning can lead to better generalization. Bringing sparsity emergence and difficulty of training together, we will demonstrate how certain types of sparsity can lead to problems in retraining a model after pre-training in the context of mutually frozen weights. From a theoretical viewpoint, Neyshabur et al. [40, 41] connect the path norm of a network to its generalization via Rademacher bounds, where the path norm is based on the sum across all paths of the product of weights associated to individual paths.

heading: 3 Optimal task tracking
text: We are interested in better understanding what influences the outcome of an optimal task tracking problem, i.e. given a fixed task sequence how do we train a model to optimize the performance on the final target task. To clarify our further writing, we summarize essential elements involved in sequential learning and visualize the overall setting in Fig. 1.
A task ⌧ consists of a probability distribution p(x, y) of data points and desired outputs as well as a loss function L : (p, f) 7! R+ that maps an input-output behavior of a model to a cost score. The learning mapping ! : (f, ⌧) 7! fnew, e.g. stochastic gradient descent, uses the information in task ⌧ and initial model finit to train a model flearn optimized for the task until convergence. The tasks and learning mappings (⌧i,!i) 2 T are applied sequentially starting from an initial model finit to produce the final target model ftarget to be evaluated on the target task ⌧target.

heading: 4 Plasticity and frozen weights
text: With the basic building blocks in place, we ask in which cases, even with an infinite number of data points, previous tasks can hinder learning on new tasks. We are interested in understanding what causes a model to be more difficult to adapt to a new task than another. This is evidently the case when some of the model weights ✓ cannot be updated when training on the new task even when this would yield a better model. Based on this reasoning, we introduce the concept of frozen weights, i.e. weights that are unchanged when training on a learning task s.t. plasticity is clearly lacking. Definition 1 (Frozen weights). We define a frozen weight ✓frozen of model f as a parameter that is not changed through learning ! on task ⌧ . ✓frozen(f) = ✓frozen(!(⌧, f))
Associated to frozen weights, plasticity captures how well a model can adapt to a new task. Definition 2 (Plasticity). We define plasticity as the rate of change of the model parameters subject to gradient descent updates P := r✓L(f(✓, x), y).
The definitions of frozen weights and plasticity beg the question in which scenarios frozen weights can appear. A promising candidate for such a constellation are mutually frozen weights. Definition 3 (Mutually frozen weights). For a model f(✓;x) with parameters ✓ and input x, we consider n non-overlapping sets of parameters {✓i} as mutually frozen if, for each set, all parameters are zero and if each parameter in each set is exclusively multiplied with all other sets of parameters. We call such parameters n-sparse frozen weights. As the leading example, weights ✓i, ✓j , ✓k 2 R are triple-sparse mutually frozen if, first, the weights ✓i, ✓j , ✓k only appear multiplied together within the model as ✓ijk := ✓i · ✓j · ✓k s.t. f(✓;x) = f(✓ , ✓ijk;x), where ✓ refers to all parameters except ✓i, ✓j , ✓k and, secondly if ✓i = ✓j = ✓k = 0. Given a loss L(✓;x) ! R+ we see that the gradient of ✓i is proportional to the remaining weights ✓j · ✓k = 0:
@L(✓;x) @✓i = @L(✓;x) @f(✓;x) @f(✓;x) @✓ijk @✓ijk @✓i = @L(✓;x) @f(✓;x) @f(✓;x) @✓ijk ✓j✓k
Therefore with condition ✓i = ✓j = ✓k = 0 and assuming that the partial gradients L(✓;x) @f(✓;x) and @f(✓;x) @✓ijk are bounded, the gradient of the mutually frozen weights ✓i, ✓j , ✓k weights are zero.
We note that mutually frozen weights describe n-tuples (n > 1) that “freeze together”. As depicted in Fig. 2, we will focus on double-sparse and triple-sparse frozen weights in the remainder of the text. Of course also higher-order freezing involving n-tuples are possible yet do not necessarily contribute further insights. Likewise we deem this less likely to occur in neural network architectures since the influence of individual parameters spreads much further when involving more than three layers. Example 1 (Linear regression). Linear regression does not have “mutually frozen weights”. Given f(✓;x) = ✓Tx + b, y, b 2 R, x, ✓ 2 Rn, we have @f(✓;x)@✓ = x. Hence, unsurprisingly, no weights are multiplied with other weights in a way that could create mutually frozen weights. Thus we would not expect linear regression to incur a loss in plasticity in a sequential task learning setup.
Example 2 (Neural networks). A simple example of triple-sparse mutually frozen weights appears for the model f(✓i, ✓j , ✓k;xi) = ✓i✓j✓kx with x, ✓i, ✓j , ✓k 2 R. In this case, one can readily see that if ✓i = ✓j = ✓k = 0 then the gradients for all weights will also be zero. This can be regarded as a sub-part of a larger piece-wise linear neural network which will not be further trained due to
mutually frozen weights. Note that with 3 or more weights multiplied as shown, the zero gradient is
locally stable (see Sec. 4.1).
The mutually frozen weights condition is particularly relevant for deep neural networks which consist of many pairwise multiplications of weights. In practice paired weights need not be zero exactly but simply have very small absolute values compared to the remaining weights. Other situations of training problems due to insufficient gradients exist such as the vanishing gradient problem [42], wherein the magnitude of the gradient is reduced as it progresses through the network due to small weight norms. This problem however appears to be largely solved by methods such as identity connections such as in residual networks [43] and normalizations such as batch normalization [44].
From afar it might appear that the answer to the frozen weights problem is simply to avoid them, i.e. to ensure through training a model to not end up with frozen weights. However as we will document in the following, weight decay, which can cause frozen weights, is often connected to better generalization and thus makes frozen weights non-trivial to avoid. This naturally causes a curious trade-off between how we first train a model on a pre-training and subsequently on a target task.

heading: 4.1 Existence and weight decay dynamics associated with mutually frozen weights
text: An important question at this point is whether and how such mutually frozen weights can occur. In Fig. 3, we show that this indeed can happen in standard training settings (we use the same settings as Mehta et al. [37]). Moreover, we notice that the number of mutually frozen weights increases as the training progresses and that the amount of mutually frozen weights created depends on the
amount of weight decay applied. While weight decay is not usually expected to create sparsity in the weights for a shallow model [46], we show that for deep models weight decay promotes the existence of mutually-frozen weights by modifying the learning dynamics in such a way that makes mutually-frozen weights attractor-points of the dynamics.
For the following argument we pick a triplet of weights ✓i, ✓j , ✓k 2 R which are multiplied for all forward passes of a neural network. Furthermore we assume that the network is piece-wise linear as in the case of ReLU activations. Then we can rewrite the model as f(✓;x) = A(✓ ;x)✓i✓j✓k + B(✓ ;x), where ✓ indicates that ✓ does not contain ✓i, ✓j , ✓k. With weight decay, weights ✓i, ✓j , ✓k evolve as follows:
✓̇i/j/k = ⌘ ✓ @L(✓;x) @f(✓;x) @f(✓;x) @✓i/j/k + ✓i/j/k ◆ = ⌘ ✓ @L(✓;x) @f(✓;x) A(✓ ;x) ✓i✓j✓k ✓i/j/k + ✓i/j/k ◆
Assuming a quadratic loss1 L(✓;x) = 12 ||f(✓;x) y|| 2 2, we can further see that ✓̇i =
⌘((A(✓ ;x)✓i✓j✓k+B(✓ ;x) y)A(x; ✓ )✓j✓k+ ✓i) = ⌘(A(✓ ;x)2✓2j ✓2k✓i+(B(✓ ;x) y)A(✓ ;x)✓j✓k + ✓i). We calculate the stability dynamics of ✓i, ✓j , ✓k linearized around equilibrium weights (✓i, ✓j , ✓k)eq. = (0, 0, 0). With this linearization we obtain:
0 @ ✓̇i ✓̇j ✓̇k 1 A = ⌘ 0 0 0 0 0 0 ! ✓i ✓j ✓k !
These dynamics have eigenvalues ↵1,2,3 = ⌘ . Therefore weights ✓i = ✓j = ✓k = 0 are locally stable [47]. The above dynamics calculation was performed on a pair of input-output data x, y but should similarly hold in expectation over the entire dataset. We would expect that triple-sparse mutually frozen weights and even higher-order frozen weights once formed remain stable and are not retrained. For double-sparse frozen weights, we find that, under certain conditions, weight can escape the local equilibrium as shown in a similar analysis in Appendix 6.1.

heading: 4.2 Invariance-plasticity connection
text: We have established that mutually-frozen weights arise naturally during the training of a DNN, and, perhaps surprisingly, while their effect seems negative as they lock some parameters of the network, their presence is associated with weight decay, which has instead an overall positive regularization effect. To clarify the apparent contradiction, in this section we show that mutually frozen weights may help to codify “learned invariances” to transformations of the data. As invariance we define the sensitivity of a model to local input changes.
1Note that we can always locally approximate the loss L(✓) quadratically around the equilibrium point.
Definition 4 (Invariance - input sensitivity). Given a smooth input transformation 'h(x) : x 7! x̃ applied to input x with 'h=0(x) = x, we quantify invariance as Inv := @f(✓;'h(x)) @h .
For composite model functions f = fn(fn 1(. . . f1('h(x), ✓), with parameter vectors ✓i associated to each intermediate composite function fi, we can find a connection between invariance and plasticity characteristics of a model. For this we first note that via chain-rule, we can decompose the model invariance and model plasticity derivative as follows:
@f('h(x), ✓)
@h =
@f('h(x), ✓)
@fn 1 . . .
@fi+1('h(x), ✓)
@fi . . .
@f2('h(x), ✓)
@f1
@f1('h(x), ✓)
@'h(x)
@'h(x)
@h
@f('h(x), ✓)
@✓i =
@f('h(x), ✓)
@fn 1 . . .
@fi+1('h(x), ✓)
@fi| {z } Shared
@fi('h(x), ✓)
@✓i
From the above we can see that the input sensitivity derivative shares many factors with the model derivative. The earlier the intermediate composite function is applied to the input, the more overlap in factors we have. Thus, for compositional models, input sensitivity and plasticity are naturally linked. Lemma 1 (Plasticity-invariance connection). We assume a composite model f(✓,'h(x)) = fn(✓n, fn 1(✓n 1, . . . f1(✓1,'h(x), ✓) and that that the factors unique to the input sensitivity and unique to the model derivative are non-zero @fi 1('h(x),✓)
@f1
@f1('h(x),✓) @'h(x) @'h(x) @h 6= 0 and
@fi('h(x),✓) @✓i 6= 0. Then a lack of plasticity in some direction d✓ implies that there exists an input transformation s.t.
@f('h(x),✓) @h dh = 0 in which the model is invariant to transformation 'h.
Proof. Knowing that the plasticity is zero in d✓ direction and that the individual, non-shared parameter derivative factors are non-zero implies that the shared factors have a non-empty nullspace. Since the shared factors also appear for the model input sensitivity derivative we can make a statement about possible implied invariances. Case 1: The individual factors of the input have a range space that does not overlap with the nullspace of the shared factors. Then we can deduce that the individual factors of the input sensitivity derivative have a non-empty null space. Case 2: The individual input factors have a range space that does overlap with the nullspace of the shared factors. For both cases, there exists a transformation 'h with direction dh for which the model input derivative vanishes @f@hdh = 0 since there is either a nullspace in the shared or individual factors that projects derivatives to zero.
The above shows that mutually frozen weights are naturally linked to invariance if the zero parameter derivative arises in shared derivative factors, which is the case for mutually frozen weights. The assumption of shared factors becomes more and more restrictive the later an intermediate composite function is applied such that we would expect that later layers are easier to retrain as observed by Achille et al. [8]. In the limiting case of linear regression, there are no shared factors and we would, in a general setting, not expect any relation between input sensitivity and parameter gradients. For very deep networks (especially without skip connections), the relation should be most pronounced for the layers that are closest to the input since more factors are shared.

heading: 5 Experiments
text: We explore the ideas and connections of mutually frozen weights and invariance and their impact in a number of sequential learning settings across different datasets, network architectures, learning rates, and weight decay settings. Detailed specifications will be provided in the appendix. The following subsections are meant to give further evidence for the following statements:
• Mutually frozen weights occur and are different from weights that are zero (sparse) but not mutually frozen. • Both sufficiently high learning rates and weight decay are essential for the occurrence of mutually frozen weights and final test performance as tested across two different architectures. • Mutually frozen weights at the beginning of training can be harmful yet can be removed through a “resetting intervention”. • Across a number of task changes, removing frozen weights is beneficial as long as sufficiently many retraining samples are available. • We provide an analysis summary relating frozen weights, invariance, and performance.

heading: 5.1 Mutually frozen weights exist and are different from regular sparse weights
text: To begin our experiments, we want to demonstrate that mutually frozen weights indeed exist in deep neural network training and that mutually frozen weights are qualitatively different from “regular” sparse zero weights that are not paired with other zero weights. Evci et al. [38] showed that convolutional networks that are sparse can indeed be hard to train to similar test performance as
networks that are not sparse. However they did not provide a mechanism of how sparsity developed or that double-sparse and triple-sparse weights in particular make it hard to train.
In Fig. 3 and 4, we count a weight 3-tuple as mutually frozen if the absolute value of each weight is smaller than 1e-4. We find that when starting training with a high learning rate and using nonnegligible weight decay the number of mutually frozen weights steadily increases with the number of trained epochs as shown in Fig. 3. While weight decay affects all parameters, as shown on the left of Fig. 4, triple-sparse weights remain stably at zero as shown in the middle and right of Fig. 4.
To establish that mutually frozen weights are a meaningful concept, we compare the performance when training randomly initialized All-CNNs [45] at different sparsity levels on CIFAR-10 image classification when altering the initial number of small weights. As a point of reference we measured that around 50% of weights fall below our sparsity threshold even without setting any weights to zero. We create mutually frozen weights by randomly sampling entire filters, associated filter columns and associated batch normalization weights to be set to zero. “Double-sparse” weights we create by randomly setting weights in a filter to zero but excluding any batch normalization weights. This ensures that triplets of zero weights are very unlikely. Finally, we create “mono-sparse” weights by following the same procedure as for “double-sparse” weights but adding identity filters to the existing filters by “resetting” (subsection 5.2). Thus we can avoid that any weight has an associated filter which is completely zero. As shown in Tab. 1, training a network is hardest with triple-sparse frozen weights, less difficult with double-sparse weights and not difficult with mono-sparse weights. We deduce that, in this setting, sparsity in itself does not necessarily have a strong effect on training performance outcomes, triple-sparse or double-sparse weights however do.

heading: 5.2 Interventions - Resetting mutually frozen weights - Adding scaled identity
text: To further test the effect and removal of frozen weights, we devise a simple “resetting” intervention. The intervention adds a small constant times identity initialized weights, similar to the idea of DiracNets [48], to all filters. To the convolutional filters, we add very small zero-mean Gaussian random noise with standard deviation 1e-5 to avoid identical updates by breaking symmetry of updates. Batch normalization (BN) multiplicative weights are either, depending on the magnitude of the intervention, increased by the same small constant across all weights or set directly to 1.0 for all BN weights. By design, this removes all mutually frozen weights since no weight of one filter is associated to only zero weights of an adjacent layer - the effect of which is also shown in Tab. 1.
To demonstrate the effect of the proposed “resetting” intervention, we apply it on a number of pre-training to retraining task changes as recorded in Tab. 2. As long as sufficiently many samples are available for retraining, this can lead to a consistent increase in performance, albeit a small increase if features do not have to be adapted much. For domains that are sufficiently different, we can achieve meaningful gains such as when moving from ImageNet to FashionMNIST, where we surpass the
previous state-of-the-art of 96.9% accuracy [57] and reach 97.8% accuracy. To demonstrate the crucial role that BN weights play, we further provide experiments with and without intervention in Tab. 3 on the “critical learning periods” task of pre-training on blurred images then switching to regular sharp images and the time-sensitive data augmentation task of introducing random shift only after having trained without it. We note that without BN weights, time-sensitive effects are greatly reduced, however the maximal performance is reduced. As a fine-tuning baseline for all experiments, we follow the comprehensive visual benchmark approach [7]. Specific details such as studying the number of samples vs. retraining performance are listed in the appendix.

heading: 5.3 Learning rate and weight decay - mutually frozen weights
text: We might wonder whether training could simply be done without weight decay or trained with small learning rates to avoid the formation of many mutually frozen weights. In a negative pre-training setting of pre-training on blurred and retraining on sharp CIFAR-10 images [8], we trained either ResNet18 [43] or All-CNNs [45]. With the currently popular setup employing weight decay we find pre-training with low weight decay or initial learning rate but retraining with high initial learning rates and sufficiently high weight decay leads to better test performance as shown in Fig. 5.

heading: 5.4 Analysis: Invariance-plasticity connection
text: A compelling part of weight decay is the potential double-edged nature it has which leads to both better test performance yet also to mutually frozen weights. In Fig. 6, we showcase the relationship of performance to the number of frozen weights at the beginning and the end of training. Likewise, we highlight the relation between invariance and performance on the very same experiments. To quantify invariance we utilize the Fisher discrimation score [59, 60] and adapt it to analysis of high-dimensional features extracted before the last layer of a Resnet18 where we compute mean µ and variance ⌃ for each class individually. We are not only interested in absolute invariance but also care about how invariant a model is within a class but also how this variance is scaled relative to the distance between classes.
S = 2between 2within = (~w · (~µ1 ~µ0))2 ~wT (⌃0 + ⌃1)~w w=µ1 µ0⇡ (~µ1 ~µ0) 4 (~µ1 ~µ0)T (⌃0 + ⌃1)(~µ1 ~µ0) (1)
As shown in Fig. 6, we find that zero initial frozen weights are best to later achieve good performance when retraining. After retraining however, a low but non-zero initial frozen weight count is best while
more frozen weights decrease performance. When regarding Fisher score invariance, the models that are most invariant before retraining appear to incur a performance penalty after retraining, while the models that are more invariant after retraining outperform their less invariant counterparts.

heading: 6 Discussion and conclusion
text: We started our investigation intrigued by the initial observation that deep networks can exhibit critical learning periods [8] such that pre-training on blurred images can lead to worse performance when retraining on sharp images. From this starting point we showed that mutually frozen weights play a role in making it harder to retrain neural networks by limiting how gradients flow in a network which is a qualitatively different kind of sparsity that, to our knowledge, has not been discussed before.
Mutually frozen weights may not occur in all commonly used networks. Likewise it is possible that mutually frozen weights can be avoided in the future by improving upon weight decay. Training of a DNN involves numerous other factors which may overshadow the effect of frozen weights and make the analysis more difficult. Due to the potential generality of our proposed concepts it is likewise always helpful to add further datasets and data and model types to assess whether the same effects are present. The scope of this paper is to further the understanding of transfer learning. As such it may have an impact on any setting associated to transfer learning and may help make deep learning solutions more accessible.
Thought-provoking about mutually frozen weight resetting is that a very sparse network, whose mutually frozen weights are completely removed, can be trained to the same accuracy as a network trained from scratch. Furthermore, retraining even ImageNet pre-trained models can often be slightly improved by removing existing frozen weights and even helped us obtain state-of-the-art results. Better understanding mutually frozen weights may enable to us to get the most out of pre-trained networks when retraining them on datasets that require a focus on different features. Finally, we touched upon a potentially fundamental trade-off in sequential learning between invariance and plasticity for deep networks. We hope this may spark new research in sequential learning on how best to train a network sequentially, which types of networks are especially conducive to sequential learning and which trade-offs may be inevitable in sequential learning.

Review summaries:
The paper studies the "optimal task tracking" problem under the training setting where we use the "weight decay" regularization. More specifically, the paper studies the relationship between plasticity-stability tradeoff and shows that a consequence of training with (large enough) weight decay is the "mutually frozen" weights that can damage the plasticity of the network. Moreover, the paper introduces an intervention to reset the mutually frozen weights and improve the performance.

In this paper, the authors present a mechanistic phenomenon linked to plasticity problems incurred in sequential task learning. They call this phenomenon "mutually frozen weights", describing a situation where triples (or higher order tuples) of weights that multiply each other in a system, all end up with zero value, which prevents error gradients from influencing their value when learning new tasks. They then argue that such a situation arises in real world setting, and explain how weight decay can promote such a situation. They also demonstrate that such frozen weights play a role in input invariance, and thus should be considered in a tradeoff perspective. Finally, the present a "resetting" method that helps mitigate the problems brought by mutually frozen weights, and perform some numerical experiments to demonstrate the efficacy of their method.