Powerpropagation: A sparsity inducing weight reparameterisation
Authors: Jonathan Schwarz
Siddhant Jayakumar
Razvan Pascanu
Peter E. Latham
Yee Whye Teh
Venue: NeurIPS 2021 Poster (2021)
Keywords: Sparse Neural Networks
Continual Learning

Abstract: The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. In
this work, we introduce Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, our method gives rise to weight updates exhibiting a “rich get richer” dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have a distribution
with markedly higher density at zero, allowing more parameters to be pruned safely. Powerpropagation is general, intuitive, cheap and straight-forward to implement and can readily be combined with various other techniques. 

To highlight its versatility, we explore it in two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constrained settings. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use
of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity. In all cases our reparameterisation considerably increases the efficacy of the off-the-shelf methods.

Powerpropagation: A sparsity inducing weight reparameterisation
Jonathan Schwarz DeepMind & Gatsby Unit, UCL schwarzjnßgoogle.
Siddhant M. Jayakumar DeepMind & University College London
Razvan Pascanu DeepMind
Peter E. Latham Gatsby Unit, UCL
Yee Whye Teh DeepMind
Abstract
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models training and evaluation. well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. baight-prrameteriaation for
this work, introduce Powerpropagatton, neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, method gives rise weight updates exhibiting "rich get richer" dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have distribution with markedly higher density at zero, allowing more parameters to be pruned safely. implement
Powerpropagation is general, intuitive, cheap and straight-forwat and can readily be combined with various other techniques. To highlight its ver- satility, we explore
two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constraing settings. Here, we combine Powerpropagation with a traditional brgning-runing technique as well as recent state-of totte--o-th--rr toarse-o-sspase algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use of sparsity in overcoming catastrophic forgetting, where compressed representa- tions allow accommodating a large number tasks at fixed model capacity. all cases our reparameterisation considerably increases the efficacy of the off-the off-the-shelf methods.
1 Introduction
Deep learning models are emerging as the dominant choice across several domains, from language [c.g. 2] vision [e.g. 3. RL [c.g. 5, 6]. One particular characteristic of these architectures that they perform optimally in the overparameterised regime. In fact, their size seems to be mostly limited by hardware constraints. While this potentially counter-intuitive, given a classical view on overfitting, the current understanding is that model size tends to have dual role: It leads to better behaved loss surfaces, making optimisation easy, but also acts as a regulariser. This gives rise the double descent phenomena, where test eтror initially behaves as expected, growing with model size due to overfitting, but then decreases again as the model keeps growing, and asymptotes as the model size goes to infinity to better performance than obtained in the classical regime [7, 8).
35th Conference Neural Information Processing Systems (NeurIPS 2021).

---

As consequence, many of the state of the art models tend to be prohibitively large, making them inaccessible to large portions of the research community despite scale still being a driving force in obtaining better performance. To address the high computational cost of inference, a growing body of work has been exploring ways to compress these models. As highlighted by several works [e.g. 9, 10, 111, size is only used as a crutch during the optimisation process, while the final solution requires a fraction of the capacity of the model. A typical approach therefore is to sparsify or prune the neural network after training by eliminating parameters that do not play vital role in the functional behaviour the model. Furthermore, there is a growing interest in sparse training [e.g. 12, 13, 14], where the model is regularly pruned or sparsified during training in order reduce the computational burden.
Compressed or sparse representations are not merely useful to reduce computation. Continual learning, for example, focuses on learning algorithms that operate on non-iid data [e.g. 15, 16, 17]. In this setup, training proceeds sequentially on a set of tasks. The system is expected to accelerate learning on subsequent tasks as well as using newly acquired knowledge to potentially improve on previous problems, all of this while maintaining low memory and computational footprint. This is difficult due to the well studied problem of catastrophic forgetting, where performance on previous tasks deteriorates rapidly when new knowledge is incorporated. Many approaches to this problem require identifying the underlying set of parameters needed to encode the solution to a task in order to freeze or protect them via some form of regularisation. In such a scenario, given constraints model size, computation and memory, it is advantageous that each learned task occupies as little capacity possible.
In both scenarios, typically, the share of capacity needed to encode the solution determined by the learning process itself, with explicit force to impose frugality. This is in contrast with earlier works on Lo regularisation that explicitly restrict the learning process to result in sparse and compressible representations [18]. The focus of our work, similar to the Lo literature, is on how to encourage the learning process to be frugal in terms of parameter usage. However instead of achieving this by adding explicit penalty, we enhance the "rich get richer" nature of gradient descent. In particular we propose a new parameterisation that ensures steps taken by gradient descent are proportional to the magnitude of the parameters. In other words, parameters with larger magnitudes are allowed to adapt faster in order to represent the required features to solve the task, while smaller magnitude parameters are restricted, making it more likely that they will be irrelevant in representing the learned solution.
2 Powerpropagation
The desired proportionality of updates to weight magnitudes can be achieved in surprisingly simple fashion: In the forward pass of a neural networks, raise the parameters of your model (clement-wise) to the a-th power (where a > 1) while preserving the sign. It easy to see that due to the chain rule of calculus the magnitude of the parameters (raised to a - 1) will appear in the gradient computation, scaling the usual update. Therefore, small magnitude parameters receive smaller gradient updates, while larger magnitude parameters receive larger updates, leading to the aforementioned "rich get richer" phenomenon. This simple intuition leads to the name of our method.
More formally, we enforce sparsity through an implicit regulariser that results from reparameterising the model similar to [19, 20]. This line of research builds on previous work on matrix factorisation [e.g. 21, 22]. In [19] a parameterisation of the form w = v O v - u O is used to induce sparsity, where stands for element-win wise multiplication and we need both v and negative values, since the parameters are effectively squared. In our work we rely on a simpler formulation where w 2/le|0 - for any arbitrary power a 2 (as compared to fixing a to 2). which, since we preserved the sign of e, can represent both negative and positive values. For a = this recovers the standard setting.
be able to represent
If we denote by θ = R.M the original parameter space or manifold embedded in RM,, our reparam- eterisation can be understood through invertible map V, where its inverse - projects 0 E θ into o E 4 where & is new parameter space also embedded in RM M,M,i i.c. +- = R As The map is detined by applying the function / R + R, (x) x||| element-wise, where by abuse of notation we refer to both the vector and element level function by V. This new parameter space or manifold & has curvature (or metric) that depends on the Jacobian of T. Similar constructions have

---

been previously used in optimisation, as for example in the case of the widely known Mirror Descent algorithm [23]. where the invertible map is the link function. For deep learning, Natural Neural Networks [24] rely on reparameterisation of the model such that in the new parameter space, at least initially, the curvature is close to the identity matrix, making a gradient descent step similar to second order step. Warp Gradient Descent [25] relies on a meta-learning framework to learn nonlinear projection of the parameters with a similar focus of improving efficiency of learning. In contrast, our focus is to construct a parameterisation that leads to an implicit regularisation towards sparse representation. following [19], rather then improving convergence.
Given the form of our mapping V, in the new parameterisation the original weight 0; will be replaced by 6s. where (())) = di 0=10 n = 0₂ and indexes over the dimensionality of the parameters. Note that we apply this transformation only to the weights of a neural network, leaving other parameters untouched. Given the reparameterised loss L(, V (()), the gradient wrt. to & becomes
ac(, (()) 0Φ
OC (v)
dc (v)
diag a (ag((a))0a1)))
(I)
(v) 06
Note that diag indicates a diagonal matrix, and |0|00-1 indicates raising element-wise the entries of vector |o] to the power a - 1. ac is the derivative wrt. to the original weight = 0|ø|-0-1
[v(Φ)
which is the gradient in the original parameterisation of the model. This is additionally multiplied (element-wise) by the factor a al. a-1, which will scale the step taken proportionally the magnitude of each entry. Finally, for clarity, this update is different from simply scaling the gradients in the original parameterisation by the magnitude of the parameter (raised a 1), since the update is applied to ø not 6, and is scaled by Ø. The update rule (1) has the following properties:
(i) O is critical point for the dynamics of any weight Φt. if a > 1. This is easy to see as AC = O whenever oi = 0 due to the a 0i 1o-1 84
factor.
(ii) In addition, 0 is surrounded by a plateau and hence weights are less likely to change sign (gradients become vanishingly small in the neighbourhood should negatively affect initialisations that allow for both negative and positive values, but it might have bigger implications for biases.
due to the scaling). This
(iii) This update is naturally obtained by the Backpropagation algorithm. This comes from the fact that Backpropagation implies applying the chain-rule from the output towards the variable of interest, and our reparameterisation simply adds another composition (step) in the chain before the variable of interest.
At this point the perceptive reader might be concerned about the effect of equation (1) on established practises in the training of deep neural networks. Firstly, an important aspect of reliable training is the initialisation (le.g. 26, 27, 28)) or even normalisation layers such as batch-norm [29] or layer- norm [30]. We argue that our reparameterisation preserves all properties of typical initialisation schemes as it does not change the function. Specifically, let 6; ~ p(O) where p(0) is any distribution of choice. Then our reparameterisation involves initialising Φ < sign(0.) v/0.T. ensuring the
neural network and all intermediary layers are functionally the same. This implics that hidden activations will have similar variance and mean as in the original parameterisation, which is what initialisation and normalisation focus on.
Secondly, one valid question is the impact of modern optimisation algorithms ([e.g. 31, 32, 33)) on our reparameterisation. These approaches correct the gradient step by some approximation of the curvature, typically given by the square root of running average of squared gradients. This quantity will be proportional (at least approximately) to diag(ar()) [o/=-11)) This reflects the fact that our projection relies on making the space more curved and implicitly optimisation harder, which is what these optimisation algorithms aim to fix. Therefore, a naive use with Powerprop. would result in reduction of the "rich get richer" effect. On the other hand, avoiding such optimisers completely can considerably harm convergence and performance. The reason for this is that they do not only
'To see this assume the weights do not change from iteration to iteration. Then each gradient is scaled by the same value diag(a (a (6 ()) which factors out the summation of gradients squared, hence the correction from the optimiser will undo this scaling. In practice & changes over time, though slowly, hence approximately this will still hold.
3

---

-18
3-4
73
weghts
-28
Remaiiiing weights
Final weight
10c
10f
30
26
Fniiin
-14
-10"
-10'
022
50%
-22
-10
-0.2
0.0 Inttial weight
0.2
-0.2
0.0
0.2
Weihhs Remariin 154
13
Fonssprap.
Initial weight
(a) Sparsity Performance
(b) Mask overlap
(c) Baseline
(d) 4.0
Figure I: Powerpropagation applied to Image classification. (a) Test accuracy at increasing levels of sparsity for MNIST (b) Overlap between masks computed before and after training (c) & (d) Analysis of weight distributions for Baseline model and at a high a. We use 10K weights chosen at random from the network. For a) & b) we show mean and standard deviation over runs. We provide code to reproduce the MNIST results (a) in the accompanying notebook.
correct for the curvature induced by our reparameterisation, but also the intrinsic curvature of the problem being solved. Removing the second effect can make optimisation very difficult. We provide empirical evidence of this effect in the Appendix.
To mitigate this issue and make Powerprop. straightforward to use in any setting, we take inspiration from the target propagation literature [34, 35, 36, 37J which proposes an alternative way of thinking about the Backpropagation algorithm.
In our case, we pretend that the exponentiated parameters are the de-facto parameters of the model and compute an update wrt. to them using our optimiser of choice. The updated exponentiated parameters are then seen as a virtual target, and we take a gradient descent step on 6 towards these virtual targets. This will result in descent step, which, while it relies on modern optimisers to correct for the curva- ture of the problem, does not correct for the curvature introduced by our parameterisation. Namely we denote optim:
: R + RM as the function that implements the correction to the gradient done by some modern optimiser, our update becomes AΦ = optim
ac 8
diag(a: (6) 10)))...
The proof that our update is indeed correct follows the typical steps taken in the target propagation literature. From a first order Taylor expansion of L( - m)d), we have that in order for AΦ to reduce
the loss, the following needs to hold: (Aó, 36 0. But we know that (
(optim
ac
(v))
as this was a valid step on (()) Because (iag(a)/)) (0-1) positive definite (diagonal matrix with all entries positive), we can multiply it on both sides, proving that (Aś, (b) > 0. We provide more details in the Appendix. We will rely on this formulation our empirical evaluation.
Effect on weight distribution and sparsification
At this point empirical demonstration might illuminate the effect of equation (1) model parameters and our ability to sparsify them. Throughout this work, we will present results from neural networks after the removal of low-magnitude weights. We prune such parameters by magnitude (i.e. min (0]), following current best practice [e.g. 38, 39, 13]. This is based on a Taylor expansion argument [14] of sparsely- parameterised function f(x O.) which would like approximate its dense counterpart f (a, 8): fxx 0.) zs f(0,x) x)) + g" (0, 0) + 2 (0, 0) H(,, #) + +... where is the gradient vector and H the Hessian. As higher order derivatives
impractical to compute for modern networks, minimising the norm of (0, - 0) is a practical choice instead.
Following the experimental setup in [10] we study the effect of Powerpropagation at different powers (a) relative to standard Buckpropagation with otherwise identical settings. Figure la shows the effect of increasing sparsity on the layerwise prggituudepruminng setting for LeNet [40] on MNIST [41]. In both cases we notice a significant improvement over an otherwise equivalent bascline. While the choice of does influence results, all choices lead to an improvement in the MNIST setting. Where does this improvement come from? Figures 1c & Id compare weights before and after training
2 For numerical stability reasons we typically suggest a choice of a E (1,3) depending on the problem.

---

on MNIST at identical initialisation. We prune the network to 90% and compare the respective weight distributions. Three distinct differences are particularly noticeable: (i) Most importantly, weights initialised close to zero are significantly less likely to survive the pruning process when Powerpropagation is applied (see green Kernel density estimate). (ii) Powerpropagation leads to a heavy-tailed distribution of trained weights, as (1) amplifies the magnitude of such values. These observations are what we refer to as the "rich-get-richer" dynamic of Powerpropagation. (iii) Weights are less likely to change sign (see Figure Id), as mentioned in Section 2.
One possible concern of (i) and (ii) are that Powerpropagation leads to training procedures where small weights cannot escape pruning, i.e. masks computed at initialisation and convergence are identical. This is undesirable as well established that pruning at initialisation is inferior [e.g. 11, 42, 43, 39]. To investigate this we plot the overlap between these masks at different pruning thresholds in Figure lb. While overlap does increase with a, at no point do we observe an inability of small weights to escape pruning, alleviating this concern. Code for this motivational example on MNIST is provided.
4 Powerpropagation for Continual Learning
While algorithms for neural network sparsity are well established means to reduce training and inference time, we now formalise our argument that such advances can also lead to significant improvements in the continual learning setting: the sequential learning of tasks without forgetting. As this is an inherently resource constraint problem, we argue that many existing algorithms in the literature can be understood as implementing explicit or implicit forms of sparsity. One class of examples are based on weight-space regularisation [e.g. 17, 44] which can be understood as compressing the knowledge of specific task to small set of parameters that are forced to remain close to their optimal values. Experience Replay and Coreset approaches [e.g. 45, 46, 47] on the other hand compress data from previous tasks to optimal sparse subsets. The class of methods on which we will base the use of Powerpropagation for Continual Learning on implement gradient sparsity [e.g. 48, 49], i.c. they overcome catastrophic forgetting by explicitly masking gradients parameters found to constitute the solution to previous tasks.
In particular, let us examine PackNet [48] representative of such algorithms. Its underlying principle is simple yet effective: Identify the subnetwork for each task through (iterative) pruning to fixed budget, then fix the solution for each task by explicitly storing a mask at task switches and protect each such subnetwork by masking gradients from future tasks (using a backward Mask M Given a pre-defined number of tasks T, PackNet reserves 1/T of the weights. Self-evidently, this procedure benefits from networks that maintain high performance at increased sparsity. which becomes particularly important for large T. Thus, the application of improved sparsity algorithms such as Powerpropagation are natural choice. Moreover, PackNet has the attractive property of merely requiring the storage of binary mask per task, which comes at a cost of bit per parameter, in stark contrast to methods involving the expensive storage (or generative modelling) of data for each past task.
Nevertheless, the approach suffers from its assumption of a known number of maximum tasks T and its possibly inefficient resource allocation: By reserving fixed fraction of weights for each task, no distinction is made in terms of difficulty or relatedness to previous data. We overcome both issues through simple yet effective modifications resulting in a method we term EfficientPuckner (EPN), shown in Algorithm 1. The key steps common to both methods are (i) Task switch (Line 3), (ii) Training through gradient masking (Line 4), (ii) Pruning (Line 8) (iv) Updates the backward mask needed to implement gradient sparsity.
Improvements of EPN upon PackNet are: (a) Lines 7-11: We propose simple search over range of sparsity rates S = [s]...., sn], terminating the search once the sparse model's 's performance falls short of minimum accepted target performance P (computed on held-out validation set) or once a minimal acceptable sparsity is reached. While the choice of 7 may appear difficult at first, we argue that maximal acceptable loss in performance is often natural requirement of practical engincering application. In addition, in cases where the sparsity rates are difficult to set, a computationally efficient binary search (for 7P) up to a fixed number of steps can be performed instead. Finally, in smaller models the cost of this step may be reduced further by instead using
"https://githir http://hhweeeeeeeeeeeeeeeeeeeeee ptteer//////ee///aeeeppppppppppp
5

---

Algorithm 1: EtficientPacKNet (EPN) + Powerpropagation.
Require: T lasks (.1...)......4 (Xy, ):: Luss & Eval. lunctions L, E; Initial weight distribution p(u) (egg. Uniform); a (for
Powerprop.J: Target performance y E [0, Sparsity rates [811--:: (at---::0 (9)] where #i 2 82 and as E (O,
Output: Trained model (i Task-speciffi Masks (M²)
M 1/54 Backward mask
sign(#) 3/10,1:8, p(0) Initialise paraneters
for
[1, .... do
arg ming L(X,, Vt: M,) Train
explicit gradient masking through My e(X,, Ve: 4/// Validation perfornance dense modal on
1
Txp raturna indicea of k largest alenenta
M; = {
if TopK(d, dim(d)]) otherwise j // Find nev Forvard maek at aparsity 81
P. - E(XT. O ( M') // Validation performance of sparse model
[
while 11
M c- V M/// Update backuard mask to protect all tasks 1,
12
13
Re-initialise pruned weights
Optionally retrain with nasked veights @ M on X before task suitch
enror estimate based a Taylor expansion [50]. This helps overcome the inefficient resource allocation of PackNet. (b) Line 8: More subtly, we choose the mask for a certain task among all network parameters, including ones used by previous tasks, thus encouraging reuse to existing parameters. PackNet instead forces the use of a fixed number of new parameters, thus possibly requiring more parameters than needed. Thus, the fraction of newly masked weights per task is adaptive to task complexity. (c) Line 13: We re-initialise the weights as opposed to leaving them at zero, due to the critical point property (Section 2). While we could leave the weight at their previous value, we found this to lead to slightly worse performance.
Together, these changes make the algorithm more suitable for long sequences of tasks, show in our experiments. In addition, they overcome the assumption of an a priori known number of tasks T. Another beneficial property of the algorithm is that the backward pass becomes increasingly sparse, the method becomes more computationally efficient with larger T. A possible concern for the method presented thus far is the requirement of known task ids at inference time, which are needed to select the correct mask for inputs. We present an algorithm to address this concern in the supplementary material.
will
5 Related work
Sparsity in Deep Learning has been an active research since at least the first wave of interest in neural networks following Backpropagation [51]. Early work was based on the Hessian [c.g. 52, 50, 53), an attractive but impractical approach in modern architectures. The popularity of magnitude-bant based pruning dates back to least [54], whereas iterative pruning was first noted be effective in [55]. Modern approaches are further categorised as Dense Dense-SSparse or Sparse methods, where the first category refers the instantiation of dense network that is sparsified throughout the training. algorithms on the other hand maintain constant sparsity throughout, giving them clear computational advantage.
Among SSanse methods, To regularisation [18] uses non-negative stochastic gates to pe- nalise non-zero weights during training. Variational Dropout [56] allows for unbounded dropout rates, leading to sparsity. Both are alternatives to magnitude- based pruning. Soft weight threshold reparameterisation [57] is based on learning layerwise pruning thresholds allowing non-uniform budgets across layers. Finally, the authors of [10] observe that re- re-training weights that previously survived pruning from their initial value can lead to sparsity superior performance. Examples of Sparse-SSarsee methods are Single Shot Network Pruning [58] where an initial mask is chosen according to the salience and kept fixed throughout. grow weights while maintaining fixed sparsity budget. Deep Rewiring for instance [59] augments SGD with random walk. addition, weights that are about to change sign are set to zero, activating other weights at random instead. Rigging the Lottery (RigL) [13] instead activates new weights by
addition, key insight is to iteratively drop

---

Rase
Powerpos (c- 1.23)
Powerprop,
Powerpepp [e 1.1250
Powerppep 1.375
Teet Accurary H4
80
AuCURAO Smal
70
Density
80
1.0
1.25
6.0
80
1.5
30
30
Remaining
Wilghts Remaining 1%6
-0.04
-0.02
0.02
0.04
(a) CIFAR-10
(b) ImageNet
(c) 100K smallest weights
Figure 2: One-shot pruning results on CIFAR-10 (a) and ImageNet (b and c). Performance sparsity curves are shown (a) and (b), highlighting significant improvements wrt. the baseline. This is due to Powerprop.'s 's effect on the weight distribution, which we show for the smallest 100k weights on ImageNet in (c). > pushes the weight distribution towards zero, ensuring that more weights can be safely pruned. For CIFAR-10 we show mean and standard deviation over five runs. Repeated runs lead to almost identical results for ImageNet and are thus omitted.
highest magnitude gradients, whereas Top-KAST [14] maintains two set of masks for (sparse) forward and backward-passes. This also allows the authors to cut the computational budget for gradient computation. In addition, a weight-decay based exploration scheme used to allow magnitude-hasee based pruning to be used for both masks. Other notable works in this category are Sparse Evolutionary Training [12] and Dynamic sparse reparameterisation [39].
Powerpropagation is also related to Exponentiated Gradient Unnormalized [60] (Example 2) which is reparametrisation equivalent to Powerpropagation and should be understood development, Indeed, the authors provide a preliminary experiment of the last layer of a convolutional network on MNIST, showing similar inherent sparsity results. We view Powerprop. generalisation of this and related ideas [c.g. 19, 20, 21] with an explicit focus on modern network architectures, introducing necessary practical strategies and demonstrating the competitiveness of reparameterisation approaches at scale. In addition, the technique has not yet been used for Continual Learning.
concurrent
While continual learning has been studied as early as [15, 16], the work on Elastic Weight Consolida- tion (EWC) [17] has recently started a resurgence of interest in the field. Contemporary approaches are subdivided into methods relying on regularisation [e.g. 44, 46, 47], data replay [e.g. 61, 62, 45, 63] architectural approaches [e.g. 64, 65]. Works such as PackNet and SpaceNet [48. 49] have also introduced explicit gradient sparsity as a tool for more efficient continual learning. Other approaches include PathNets which [66] uses evolutionary training to find subnetworks within larger model for individual tasks.
Experimental Evaluation
We now provide an experimental comparison of Powerpropagation to a variety of other techniques, both in the sparsity and continual learning settings. Throughout this section we will be guided by three key questions: (i) Can we provide experimental evidence for inherent sparsity? (ii) so, can Powerprop. be successfully combined with existing sparsity techniques? (iii) Do improvements brought by Powerprop. translate to measurable advances in Continual Learning?
This section covers varying experimental protocols ranging from supervised image classification to generative modelling and reinforcement learning. Due space constraints focus primarily on the interpretation of the results and refer the interested reader to details and best hyperparameters in the Appendix.
6.1 Inherent Sparsity
Turning to question (i) first, let us start our investigation by comparing the performance of a pruned method without re-training. This is commonly known as the one-shot pruning setting and is thus a Dense > Sparse method. improvement should show most clearly in this setting. Figure 2 shows this comparison for image classification on the popular CIFAR-10 [67] and ImageNet [68] datasets using smaller version of
Powerprop. does indeed lead to inherently sparse networks its
7

---

Method
0%
Dense
76.8 + 0.09
80%
90%
95%
80% (ERK)
90% (ERK)
Static [13]
d: 0.06
0.04
59.5 0.11
72.1 d: 0.04
67.7 +: 0.12
DSR [39]
73.3
71.6
SNFS [38]
74.2
72.3
75.2 + 0.11
72.9 = 0.06
SNIP [58]
72.0 4: 0.10
67.2 0.12
S7.8
RigL [13)
74.6 + 0.06
72.0 - 0.05
75.1 + 0.05
73.0 = 0.04
Iterative pruning [I]]
75.6
73.9
70.6
Iterative pruning (ours)
75.3++0007
13.7
70.6 0.05
Powerprop. ler Pruning
0.07
74.4 0.02
72.1 0.00
TopKAST [14]
75.47
74.65 = 0.03
0.10 72.89
72.73
75.71 + 0.06
74.79 0.05
Powerprop. TopKAST!
75.75 1: 0.05
0.04
75.84 1: 0.01
74.98 0.09
TopKAST [14] Powerprop. TopKAST
76.08 +: 0.02
75.13 + 0.03
73.19
0.02
76.42 + 0.03
75.31
0.05
76.24 1: 0.07
75.23
0.02
73.25
0.02
76.76 1 0.08
75.74
0.08
Table 1: Performance of sparse ResNet-50 on Imagenet. Baseline results from [13]. ERK: Erdos-RRnyl Kernel
[13]. */t: TopKAST at 0%/50% Backward Sparsity. Shown are mean and standard deviation over 3 seeds.
AlexNet [3] and ResNer50 [4] respectively.' In both situations we notice a marked improvement, up 33% for CIFAR and 18% for ImageNet specific sparsity rates. Fig. 2c we show the density of the smallest 100K weights, in which we observe notable peak around zero for > 1.0, providing strong evidence favour of the inherent sparsity argument. Finally worth noting that the choice of does influence the optimal learning rate schedule and best results were obtained after changes to the default schedule.
Powerprpp
6.2 Advanced Pruning
Pruning
Powerprop. TsPRAST
Method
80% Sparsity
90% Spansity
Top sccurace Eual
Tter, Pruning (1.58)
76.5 [0.84x]
75.2 [0.76x]
RigL (6x)
76.630 os [1.14x]
75.7 on [0.52x]
ERK
Rigl. (8x)
77.1 os [2.09x]
76.4 es [1.23x]
Powerprop. TopKAST" (2x)
77.51+o..0 [1.21x]
76. 76.91 [0.97)]
Powerprop. + TapKAST (8x)
77.64=o.08 [1.81x]
77.16:b..1 [1.46x]
Sparsity (8)
Table 2: Extended training. Numbers in square Brackets show FLOPs relative to training of a dense model for a single cycle.
Figure 3: Extreme sparsity
We now address question (ii) by comparing Powerprop. when used in conjunction with and compared exiting state-of-the-ar algorithms ImageNet. We orient ourselves primarily on the experimental setup in [13] & [14], both of which present techniques among the strongest in the literature. To obtain Dense- method combine Powerprop. with Iterative Pruning and combine TopKAST [14] with Powerprop. as our method of choice from the Sparse literature.
We distinguish between three sub sub-settings a) Table I shows ResNet 50@ (80%, 90%, 95%} sparsity, for which the largest number of baseline results are available. We also provide results using the Erdos-Renyi Kernel [13], a redistribution of layerwise sparsity subject to the same fixed overall budget. While this improves results, it increases the floating point operations (FI.Ps) at test time. b) Table 2: Extended training, where we scale training steps and learning rate schedule by factor of the usual 32K steps. While this tends to lead to better results the computational training cost increases. Figure 3: Extreme sparsity at 95% 99% thus testing methods at the limit of what is currently possible. All experiments are repeated 3 times to report mean and standard deviation. As Powerprop. introduces a negligible amount of extra FLOPs (over baseline methods) we only show such values in the extended training setting to provide a fair comparison to the setup in [13]. A longer discussion of a FLOP trade-off can be found in the work on TopKAST [14].
Examining the results, we note that Powerprop. improves both Iterative Pruning & TopKAST in all settings without any modification to the operation of those algorithms. In all settings a large array
"The motivating example in Scction also follows this protocol.

---

of other techniques are outperformed. To the best of our knowledge those results constitute new state-of-the art performance all three settings. We emphasise that many of the methods we compare against are based on pruning weights by their magnitude and would thus also be likely to benefit from our method.
6.3
Continual Learning
6.3.1
Overcoming Catastrophic forgetting
urcirttacow:
Poweora (o- #Fcrrttacone
Laptace
Powerpes. 3a- 251 -O Ofcenthoond
Nike
Smrppti Hal.:eeelog
Algurithm
MNIST
noLMNIST
fowrppes 3150 DricenPecote
100
Naive
-258. 23
-797.85 88
Laplace [69.65]
-225.54
EWC [17]
105.78
-212.62
[44]
-11.47
-190.48
VCL [46]
94.27
-187.55 .4
40
Eff. PackNet (EPN)
-94.50
- 177.08
Powerppop- + Laplace
96.72
-166.87
20
Powerprop- EWC
-95.79
1188.56
Powerprop. EPN
-92.15
-174.54
100
150
203
250
300
50
150
200 250
Number of Tasks
Numbor ofTasks
Table 3: Results the Continual generative modelling experiments. Shown is an importance-samplle estimate of the test log- looliik.
Figure 4: Powerpropagation quences. Left: Average accuracy over all tasks. Right: Fraction of unmasked parameters. Shown are mean and standard deviation over 10 repetitions of the experiment.
extremely long task se-
We now turn to question (iii,, studying the effectiveness of sparsity methods for Continual Learning. We first consider long task sequences on the Permuted MNIST benchmark, first popularised by [70]. While usually regarded to be an idealised and simple setting, we push its boundaries by drastically increasing the number of tasks, which was previously limited to 100 in [71] and 78 in [72], whereas we learn up 300 in a single model. In all cases all tasks, following the protocol by and comparing to [72]. Figure 4 shows the average accuracy and fraction of used weights. We significantly outperform baselines with an average acc. of = 86% after 300 tasks, whereas the closest baseline (Laplace [65]) falls to 85.61% % after merely 51 tasks. This also highlight the effect of a more noticeably, showing increased performance for higher Note also that the gradient of the used weights curve flattens, showcasing the dynamic allocation of larger number of parameters tasks early in the sequence, whereas later tasks reuse larger fraction of previous weights. While we also attempted a comparison with standard PackNet, the model failed, as its default allocation of <0.4% of weights per task is insufficient for even single MNIST task (see Figure la). The perspective reader might notice higher performance for considered baselines at small number of tasks. This is due to our choice of y 0.9 (see Section 4) which we optimise for long sequences. It is worth mentioning that can be increased up to a certain limit at which point training for high number of tasks can become unstable. We will investigate this effect in future work.
network with single output layer across
Moving onto more complex settings, we use Powerprop. on the continual generative modelling experiment in [46] where 10 tasks are created by modelling single character (using the MNIST and notMNIST datasets) with a variational autoencoder [73]. We report quantitative results in Table 3. In both cases observe Powerprop. + EfficientPack effectiveness for Catastrophic Forgetting is not limited to PackNet. In particular, we provide results for the Elastic Weight Consolidation (EWC) method and its online version (Laplace) [17, 65] both of which are based on Laplace approximation (using the diagonal Fisher information matrix). As Powerprop. results in inherently sparse networks, the entries of the diagonal Fisher corresponding to low magnitude weights are vanishingly small (i.e. the posterior distribution has high variance), resulting in virtually no regularisation of those parameters. We observe significant improvements with no other change to the algorithms.
outperforming the baselines. Interestingly, its
Finally, we move onto Catastrophic Forgetting in Reinforcement Learning using six tasks from the recently published Continual World benchmark [74], a diverse set of realistic robotic manipulation tasks. This is arguably the most complex setting in our Continual Learning evaluation. Following the authors, we use Soft actor-critic [75], training for 1M steps with Adam [33] (relying on the formulation in Section 2) on each task while allowing 100k retrain steps for PackNet (acquiring no
'nttp://yaroslavva www.:a/wwwwwbbbbbb..b...o....... http://ppopppp.opoooo/////////// http://////ttmmttaaaaaaaaaaeee wat.iisttdttttttt.ttt

---

Table 4: Reinforcement Learning Results on Continual World [74]. Error bars provide 90% confidence intervals.
Method
Avg. success
Forgetting
Fine-tuning
0.00 1o00, 0.00)
0.87 00.84.0.99
A-GEM [62]
0,02 joor, 0.94)
0.89 10866 0.11
EWC [17]
0,66 [061, 0.71]
0.07 [0.04, 0.110
MAS [78]
0.61 [0.56,
-0.01 [-0044 0.01]
Perfect Memory
0.36 [0.32, 0.40]
0.07 (o.05, 0.10]
VCL [46]
0.52 (0.47, 0.57]
-0.01 [-0.03, 0.01]
PackNet [48]
0.80 (0.76, 0.84]
0.02 (0.04, 0.00)
Power. + EffficientPacKNet (am11.376)
0.82 (0.77,0.87)
0.01 po.03, 0.01]
Power. + EffficientPacKNet (am11.5)
0.86 (0.52, 0,90)
0.00 (-0.22, 0.02]
additional data during retraining). We report the average success rate (a binary measure based on the distance to the goal) and a forgetting score measuring the difference of current performance to that obtained the end of training (but importantly before pruning) certain task. Thus, if Powerprop. does improve results, this will show in the Forgetting metric. Results are shown in Table 4. Indeed, we notice that a higher a leads reduction of forgetting and hence overall improvement over PackNet, resulting in superior performance. Finally, it is worth mentioning that (Efficient) still assume hard task boundaries that known during training. However, it has previously been shown that it is possible to overcome these limitations using changepoint detection algorithm [e.g. 76, 77].
Discussion
In this work we introduced Powerpropagation, demonstrating how its effect on gradients leads to inherently sparse networks. As we show in our empirical analyses, Powerprop. is easily combined with existing techniques, often merely requiring few characters of code to implement. While we show its effect on only a few established algorithms. we hypothesise that combination with other techniques could lead to further improvements. Bridging gaps between the fields of sparse networks and Continual Learning, we hope that our demonstration of its effect on Catastrophic Forgetting will inspire further interest in such ideas. In terms of future work, we believe that devising automated schemes for learning a may be worthwhile, making the use of our method even simpler. Particular attention ought be paid to its interaction with the learning rate schedule, which found worthwhile optimising for particular choices of a. Finally, we also hope further work will bring additional theoretical insights into method.
References
[1] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt. Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. arXiv preprint arXiv:1506.03344 2015.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.. 14165, 2020.
[3] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-
1105, 2012.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[5] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897. PMLR, 2015.
10

---

[6] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-3333
2017.
[7] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical varianee trade-off. Proceedings of the National Academy of Sciences, 11663):15849-155888
2019.
[8] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak. and llya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint 1rXin:19/..02299, 2019.
[9] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410-2419. PMLR, 2018.
[10] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 03635, 2018.
[11] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: /92.00544 2019.
[12] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
[13] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943 2952.
PMLR, 2020.
[14] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744- 20754, 2020.
[15] Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 123):123-146, 1995.
[16] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4): 1999.
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinnke. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, page 201 61 1835, 2017.
[18] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through |_0 regularization. arXiv preprint arXiv: 011122, 2017.
[19] Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[20] Peng Zhao, Yun Yang, and Qiao-Chu He. Implicit regularization via hadamard product over- parametrization in high high-dimensional linear regression, 2019.
[21] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In I. Guyon, U. V. Luxburg, S. Ben- gio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[22] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over- weer-prameteriiee matrix sensing and neural networks with quadratic activations. Proceedings of the 31st Conference On Learning Theory, pages 2-47, 2018.
[23] Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. (16 May 2003.
[24] Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and koray kavukcuoglu. Natural neural networks. In Advances in Neural Information Processing Systems, volume 28, 2015.

---

[25] Sebastian Flemnerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
[26] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural nefworks: Tricks of the trade, pages 9-48. Springer, 2012.
[27] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249.256.100 JMLR Workshop and Conference Proceedings. 2010.
(28) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[29] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[30] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1607.06400 2016.
[31] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[32] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8), 2012.
[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[34] Yann LeCun. Learning process in an asymmetric threshold network. In Disordered Systems and Biological Organization, pages 233-240. Springer Berlin Heidelberg, 1986.
[35] Yann Lecun. PhD thesis: Modeles connexionnistes de l'apprentissage (comnectionist learning models). Universite P. et M. Curie (Paris 6), June 1987.
[36] Miguel Parreira and Weiran Wang. Distributed optimization of deeply nested systems. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, pages 10-19, 2014.
[37] Dong-Hyun Hyun Lec, Saizheng Zhang, Antoine Biard, and Yoshua Bengio. Target propagation. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
[38] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:/907, 9907004440 2019.
[39] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655, PMLR, 2019.
[40] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551 1989.
[41] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.leac www.lleccccolec.oo.. www//xxd//nnisstt.o.omommmmm 2, 2010.
[42] Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:1906.100 10732, 2019.
[43] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint 1903.v:903000000 8, 2019.
[44] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. arXiv preprint arXiv:/703.0000 042000 2017.
[45] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experi- ence replay for continual learning. arXiv preprint 11I.v:1811.11688 2018.
12

---

[46] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10626 2017.
[47] Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. arXiv preprint arXiv: /901. 1356, 2019.
[48] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018
[49] Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. Neurocomputing, 439:1-11, 2021.
[50] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[51] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[52] Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in neural information processing systems, pages 107-115, 1989.
[53] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
[54] Georg Thimm and Emile Fiesler. Evaluating pruning methods. In Proceedings of the Interna- fional Symposium on Artificial neural networks, pages 20-25. Citeseer, 1995.
[55] Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In Fifth European Conference on Speech Communication and Technology, 1997.
[56] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-2507, PMLR, 2017.
[57] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pages 5544-5555. PMLR, 2020.
[58] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint 11X1v:1810.023400, 2018.
[59] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint 1711vv/711.088 03,6, 2017.
[60] Ehsan Amid and Manfred K Warmuth. Reparameterizing mirror descent as gradient descent. Advances in Neural Information Processing Systems, 2020. Version of the article (https:// // arxiv. .rwg wwrg/abs/20022 10487v1) contains experiments with the last layer of a neural network.
[61] Hanul Shin, Jung Kwon Lec, Jachong Kim, and Jiwon Kim. Continual learning with deep 2017.
generative replay. arXiv preprint
[62] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv: /8/2.0044 2018.
[63] Christos Kaplanis, Claudia Clopath, and Murray Shanahan. Continual reinforcement learning with multi-timescale replay. arXiv preprint arXiv:2004.07530, 2020.
[64] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.0666 2016.
[65] Jonathan Schwarz, Jelena Luketina, Wojeiech M Czamecki, Agnieszka Grabska- Barrinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. arXiv preprint arXiv:1805.066 867700 2018
[66] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint 170::1701.0874 2017.
13

---

[67] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[68] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115())211-252, 2015.
[69] Eleazar Eskin, Alex J Smola, and SVN Vishwamathan. Laplace propagation. In Advances in neural information processing systems, pages 441-448, 2004.
[70] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 131.62131 2013.
[71] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. arXiv preprint arXiv:1805.0740 2018.
[72] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating con- tinual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv: /2488, 2018.
[73] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13/2.6/ 14, 2013.
[74] Maciej Wolczyk. Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Con- tinual world: A robotic benchmark for continual reinforcement learning. arXiv preprint I00109,905.0099, 2021.
[75] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[76] Kieran Milan, Joel Veness, James Kirkpatrick, Michael Bowling, Anna Koop, and Demis Hassabis. The forget-me-not process. Advances in Neural Information Processing Systems, 29:3702-3710, 2016.
[77] Ryan Prescott Adams and David JC MacKay. Bayesian online changepoint detection. arXiv preprint arXiv:0710.374.. 2007.
[78] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154, 2018.
[79] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.077 077344 2019.
[80] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. arXiv preprint arXiv:2006. 14769, 2020.
[81] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6672 2014.
[82] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[83] Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv: 1612:11612.01774, 2016.
[84] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint Stti::2033097660 2021.
[85] Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE Transactions Pattern
Analysis and Machine Intelligence, 2017.
[86] David Lopez-Paz et al. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, pages 6470-6479, 2017.
[87] Gido M Van de Ven and Andreas S Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv preprint arXiv:1809. 10635, 2018.
14

---

[88] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364-372, 2019.
[89] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548-4551 PMLR, 2018.
[90] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint 1rXfv:1706.026777 2017.
[91] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-11000 PMLR, 2020.
15


Reviews:
[{"review_id": "vPQShvFPJb3", "reviewer": "Reviewer_kFtF", "summary": "The paper proposes a method to parameterize weights such that the resulting model is inherently sparse. This is achieved by dynamically updating the parameters proportional to their magnitude i.e., the higher magnitude weights receive larger updates while the low magnitude weights remain unaffected. The idea is novel and useful for training sparse networks geared towards applications in resource-constrained environments. Additionally, as elaborated in the paper, it is also useful in the cases of continual learning where a single model is used to learn multiple tasks. The claims are supported by experimental evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The idea of transforming the parameters with an exponent is a simple yet effective technique.\n- The paper is well written with necessary mathematical derivations. Experiments adequately substantiate the proposed method and claims.\n- The perspective of using sparsity in learning better models in continual learning is interesting and is very useful.\n\nWeaknesses\n\n- Section 3 last paragraph: The authors mention the concern that powerpropagation can lead to identical masks at initialization and at convergence which is intuitive as the small weights are not modified frequently. But it is also mentioned that the empirical results indicate that this is not the case. Can you please provide a possible justification?\n- Can you please discuss how the exponentiation might impact the efficiency of the network in resource-constrained devices as exponentiation can be an expensive operation. This can increase the training complexity multiple folds.\n- Section 6.3: Please mention the model architecture used for training the permuted MNIST dataset. The results in this section are impressive, however, it is limited to only one dataset in each of standard image classification and reinforcement learning settings. Moreover, permuted MNIST is synthetically generated. Hence, including another continual learning benchmark like CIFAR100 split into 10 or 20 tasks will add more value to the paper.\n\nOverall a good paper. Minor improvements will help add more value to the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed. No adverse societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "tS5kYW-I5", "reviewer": "Reviewer_m5ux", "summary": "This paper is focused on weight reparameterization, proposing a versatile method that leads to inherently sparse models.  The authors theoretically analyze this extension and backup the effectiveness of their proposed method with encouraging empirical results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The basic idea of altering the gradient descent behavior by an approximated factor, exhibiting a \"rich get richer\" dynamic, is well-worthed and original.  Several experiments are conducted to demonstrate the performance of the proposed model. Promising results in terms of accuracy have been achieved in combining the proposed method -- Powerpropagation, with interactive pruning methods and sparse-to-sparse algorithms with a slight increase in FLOPs. Still, it is not clear what is the impact of alpha. Can you provide a more systematic analysis? \n\nI like the diversity and the choice of the type of experiments performed and the special attention accorded to the catastrophic forgetting phenomenon in continual learning. However, the bridge between sparse training networks and continual learning appears possible in previous works (e.g. [1]). Please can you comment on the expected impact of SpaceNet on Figure 4?  Perhaps this shall be reflected in some nuances added to the discussion (rows 342-344).\n\n[1] G. Sokar et al. \"SpaceNet: Make Free Space For Continual Learning\", https://arxiv.org/abs/2007.07617, 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "ekKldhTswHn", "reviewer": "Reviewer_3XMd", "summary": "This is another paper where the goal and idea of the paper are almost summarized well in the title and the abstract. The only missing part is the parameterization itself which is very simple yet effective. A weight, w, is reparameterized using another underlying variable, v, as w = v|v|^($\\alpha$ - 1), essentially reparameterizing the weight (real number) using an exponentiation operation as the name suggests. \n\nThis decoupling as the paper discusses in L106-115 leads to sparse solutions, due to the gradient scaling of the underlying variable resulting in the rich getting richer due to the exponentiation involved. The paper uses this observation, coupled with some toy understanding on MNIST gave the motivation to use this reparameterization to naturally induce sparser solutions.\n\nThe paper focuses on coupling this with sparsity-inducing methods and continual learning methods that use sparsity to encode multiple sub-networks and shows that combining this reparameterization with the SOTA techniques further improves accuracy with minimal overhead. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Let me start with a general comment before going into more details.\n\nThe paper is very engaging to read but sometimes becomes dense because of the things and ideas moving around. I liked it very much once I understood what the authors were trying to do, but long bodies of text couple with notations at places made it harder at times. It is a simple idea and the explanation could be made simpler.\n\nEspecially Section 2, it very strong but at times tries to explain everything at a stretch creating a chance to miss things. It would be great to have a simple running example with a gradient update equation would have been great like. Instead of update being w - $\\eta\\Delta$w the update now becomes something like w = sign(v)(v - $\\eta\\Delta$v)^$(\\alpha)$ = sign(v)(v - $\\eta\\Delta$w$\\alpha$|v|^$(\\alpha-1)$)^$(\\alpha)$ would have made things clearer. Even now there are two cases when v > 1 and v < 1 due to the nature of exponentiation, making gradient scaling while monotonic, not trivial to understand. Most weights are < 1 so the dynamics of a weight with $\\alpha=2$ and v = 0.1 vs v = 0.2 do differ when looked at from w's perspective. \n\nI strongly hope either author comment on this or fix this in the next revisions making it easier to follow. \n\nI think that is my only major concern in the whole paper. Now coming to the strengths. \n\nThe authors identified extremely appropriate applications and showed statistically significant improvement when using powerpropagation instead of the normal weights. Both the sub-network-based CL and sparse network learning benefit from this simple yet powerful idea. \n\nThe experiments are very thorough and the authors should be appreciated for their efforts. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are decently discussed and the authors seem to be aware of the potential of the method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "Cq7ecmAN6n2", "reviewer": "Reviewer_o3yJ", "summary": "Proposes a reparameterization of neural network models, where each weight is raised to the power of alpha, where alpha > 1, but keeps its original sign.\nThe proposed reparameterization and its implications on the optimization process are briefly discussed mathematically, with the general intuition being that \"the rich get richer\", i.e., low-magnitude weights tend to stay close to zero, while high-magnitude ones tend to become more significant in the resulting model.\n\nThus, the reparameterization implicitly acts as a regularizer that prefers models with few high-magnitude weights. The resulting model is more suitable for post-pruning schemes that create sparse models.\n\nFinally, the reparameterization is empirically shown to enhance a Continual Learning scheme that uses iterative pruning (PackNet [Arun and Lazebnik 2017]), improving results on various continual learning baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Generally, the general idea is interesting and extends previous ideas in the field of pruning and sparse models.   \n* The paper is mostly well written and ideas are presented clearly (the plots are great).  \n* Empirically, there are decent results in various settings (pruning for sparse models; and implications in an iterative pruning model for continual learning).\n* I have some reservations, mostly regarding some unclear paragraphs or sentences. \n* I am ambivalent about the rating, **unsure between 6 and 7**. I believe the author response can help in this regard.\n\n---\n\n### A. Originality and significance\n\n\n1. **Originality.** I believe the proposed reparameterization is original *enough*.   \nSimilar reparameterizations were previously studied in the sparse context, e.g. the authors here adequately cite [Vaskevicius et al. 2019] that reparameterize a linear regression model as $u\\circ u- v\\circ v$, which at a certain initialization scale limit converges to a minimum-L1 solution (inducing sparsity, see also [Woodworth et al. 2020]).  \nHowever, the reparameterization in the paper reviewed here is more *general and flexible* (the exponent can be any $\\alpha>1$ and moreover preserves the signs of the exponentiated weights, thus there is no need to split the parameters into $u$ and $v$).\n\n2. **Significance.** The results are interesting to me and seem important. The reparameterization is rather elegant, can evidently yields sparser models. I believe it is immediately (and easily) applicable to general sparse schemes and more specifically pruning schemes. The authors show (Table 1) how it improves several pruning schemes and advances SOTA algorithms on sparse ResNet-50 for ImageNet.  \nThe observations on the continual learning setting are interesting but limited, since the paper focuses on a setting where the task identities are known during inference, and it seems that some of the conclusions only apply to models like PackNet that are based on iterative *pruning*, and do not apply to the wide continual learning scope.\n\n---\n\n### B. Quality and clarity\nMost of the paper is clearly written and overall the paper has a good flow. I also believe the experimental results are strong enough.  \nHowever, some of the theoretical analysis/intuitions are not really clear to me, and there are a few paragraphs that would benefit from rephrasing. Moreover, I was left with some questions and remarks I would like to specify.\n\n1. I could not understand what exactly do the authors claim in lines 121-124. Please clarify.\n2. I could not understand what is the difference between plain backpropagation and what the authors refer to in lines 136-145. This paragraph was extremely unclear to me. Could the authors please clarify? (I believe this paragraph needs to be improved).\n3. The description of Algorithm 1 in the beginning of Section 4 is not self contained (some parts cannot be understood without reading the PackNet paper). For instance, it is not clear enough that this setting assumes the task identities are known during inference (thus it was unclear to me while reading why do you store *all* forward masks and not only the current one and the backward one). I also believe it should be explained that the space complexity requirement is valid for both train and inference. Moreover, I could not understand the exact origin of the sparsity range in $S$ in the algorithm. How is it determined?\n4. Should the line 12 in Algorithm 1 be split into two lines? This is confusing.\n5. Figure 2c: which dataset and model is presented here? CIFAR or ImageNet? Not stated.\n6. In line 208: should it really say $s_n\\cdot P$? If so, why $s_n$?\n7. In line 212-213: the authors state that the algorithm \"overcomes\" the assumption of known task number. Is it actually true? Shouldn't one at least know the order of $T$ to set the sparsity ranges?\n8. In Section 6.3 (continual learning experiments). Is the comparison to EWC (for instance) fair? \nSince EWC doesn't assume known task-identities during inference like EPN does.\n9. Do any of the experiments use explicit regularization (weight decay)? Could not find the answer in the paper or Appendix.   \nI guess the answer is no, but I want to make sure. This should also be explicitly written in the paper.\n10. It can be interesting to understand or discuss the layer-wise effect of Powerpropagation. For instance, perhaps earlier layers are more dense while later layers are more sparse, etc.\n\n---\n\n### C. Minor remarks\nFollowing are some minor remarks by line numbers. Please consider fixing them.\n* 52: occupy => occupies\n* 100 (and more): I don't believe the $\\circ$ in the exponent was defined.\n* Figure 1: should the caption say $\\alpha=3.0$ instead of $4.0$?\n* 188: Percent sign seems unnecessary\n* Algorithm 1, line 12: Should the index in the superscript be $i$?\n* 221, 225, 231: Spare => Sparse\n* 223: Throughout what?\n* 263: **D**atasets should be lowercase.\n* 284: shows => show\n* 289: constitues => constitute\n* 300: significant => signficantly\n* 303: Do you actually mean the *gradient* flattens? Or that the plot flattens?\n* Generally: Instead of writing \"details in the Appendix\", consider writing the exact section of the appendix.\n\n* Table 4: Isn't PackNet's forgetting better than EPN? A negative forgetting is often called \"backward transfer\" and is a good property of an algorithm.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors addressed these issues adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "I believe there are no such issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests a simple re-parameterization, motivated by recent theoretical results, which for improves SOTA results in the areas of pruning and continual learning. All reviewers seemed to like the idea, the writing and the results. The rebuttal addressed remaining concerns, and all reviewers voted for acceptance.

Decision: Accept (Poster)