Learning where to learn: Gradient sparsity in meta and continual learning
Authors: Johannes von Oswald
Dominic Zhao
Seijin Kobayashi
Simon Schug
Massimo Caccia
Nicolas Zucchet
Joao Sacramento
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
continual learning
few-shot learning
deep learning
sparsity
sparse gradients

Abstract: Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.

Learning where to learn: Gradient sparsity in meta and continual learning
Johannes von Oswald*.1 Dominic Zhao Seijin Kobayashi', Simon Schug', Massimo Caccia", Nicolas Zucchet', João Sacramento'
Equal contribution
Institute of Neuroinformatiis, University of Zürich and ETH Zürich Mila,, University of Montreal & ServiceNow {voswaldj, dozhao)@eehhz.dl ch
Abstract
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on problem-by- problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges more expressive model where learning rates are meta-lcarned Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is powerful inductive bias for meta-learning systems.
Introduction
Meta-learning holds the promise of discovering inductive biases that improve the performance of primary learning process. Such set of assumptions can materialize in various elements of the learner. The well- koolnn model-agnostic meta-learning [MAML; algorithm aims to learn a neural network initialization that generalizes well to new learning tasks. More sophisticated meta- augment
this procedure by additionally modulating the inner-loop learning dynamics 35 32
12
It has been recently shown that applying MAML while adapting only last-layer weights leads to almost no decrease performance standard few-shot learning benchmarks. Our study builds upon the surprising effectiveness of this form of meta-learning, known as almost no inner-loop training [ANIL: 44). Here, instead deciding which weights to freeze a priori, we endow the meta-learner with the possibility to explicitly stop changing certain weights in the inner-loop learning process. We do this by introducing an adjustable binary mask which elementwise multiplied with gradient updates. This can be understood as simple form of learned gradient modulation that induces sparsity. Overlitting can thus be prevented and learning sped up by focusing adaptation to a sparse parameter subset, discovered by meta-learning.
We find that our sparse -MMM algorithm recovers a behavior that is reminiscent of ANIL. It induces high gradient sparsity in earlier layers of the network while allowing for adaptation in deeper layers including the network's output. Despite this reduction the number of adaptable parameters, the sparse learning patterns formed by sparse-MAML do not overly specialize to the family of tasks
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

observed during meta-learning: both ANIL and MAML are outperformed by the resulting sparse learners in cross-adaptation problems involving shift in task distribution g 40) Furthermore. sparsity adapts intuitively to the number of inner-loop gradient steps as well as its learning rate, few-shot dataset size, and network specifications. This leads to a robust and interpretable variant of MAML, that improves generalization by self-regularizing the parameters that the model should learn.
An exciting avenue of meta-learning research concerns continual learning. Learning tasks sequentially by gradient descent generally leads to poor results, as past tasks tend to be rapidly forgotten due to interfering weight updates. Such interference can be reduced with online meta-learmin methods which optimize the base learning algorithm using both present and past data, kept in replay buffer [5]. Our findings translate to this setting. We analyze the state-of-the thettrrrrt look-ahead MAML algorithm [La-MAML; [3] which introduces per-paune parameter meta-learned learning rates and find that sparse learning emerges, as a large fraction of learning rates drops to zero. Notably, similarly high performance can be reached when meta- meta-learning binary gradient masks only. Moreover, performance improves after endowing a version of MAMI. adapted for online learning 7 with binary gradient masks. Thus, sparse learning can improve generalization, accelerate future learning, and reduce forgetting, and these benefits can be realized within online meta-learning.
2 From MAML to sparse-MAML
The MAML algorithm seeks neural network weights @ from which only few gradient descent steps suffice to reach high performance on a given task т, that is assumed to be drawn from a certain distribution p(r). Formally, a task is defined by an outer loss function L and an inner loss function in We will later make explicit the form the two loss functions can take depending on the problem being solved. The result of the inner loss minimization is evaluated by the outer loss leading to the following optimization problem:
MAMI.
min E,)p(rr [Iout (0+, (0))] s.t. 0T,K+1 Φr.6 V (d+,k) and Φ-,0 0
(1)
with o,,* denoting the task-specific weights after k steps of gradient descent, a the inner-loop learning rate and K the inner-loop length. The initialization 0 is then obtained by iterative updating, using
- T BT=-(r) de L (or,x(0))]
with Da the outer-loop learning rate. Note that we need the total derivative de in Eq. 2 and not the partial derivative Va due the complex relationship between Φr.A and 0. In practice, the expectations over the task distribution that appear above are estimated by Monte Carlo integration. The updates in 0 therefore correspond to stochastic gradient descent on the expected outer loss.
In MAML, the total derivative w.r.t. to 0 is obtained by backpropagating through the inner opti- mization, resource-intensive procedure. First-order MAML (FOMAML) drastically reduces the computational cost by setting to zero the second-order derivatives that appear when differentiating the inner-loop update.
Learning the learning rates. Some variants of MAML focus on learning the learning rate and consider inner-loop updates of the following form:
Φr, = Φr.k a M V a I. (dr,)),
for some learnable preconditioning matrix M, that is optimized similarly to the initialization A. Through M, these algorithms learn some information on the geometry of the loss with the hope of faster inner-loop optimization. Meta-SGD [35] considers diagonal M, i.e., learnable learning rates, meta-curvature [4 considers a block matrix, while vanilla MAML corresponds to the M case.
Sparse-MAML In line with these approaches, we introduce sparse- MAM.. Together with an initial set of weights 0, our algorithm dynamically learns the parameters which will be updated and the ones that will not. Hence, sparse-MAML learns where to learn. To do so, we use a vector m (instead of matrix M) that modulates the gradient in the inner-loop update in the following way:
+ Φr, - a m20 V (
(4)
with 1:00 R. -> {0, 1)2 the step function that is applied elementwise to the underlying parameter vector E R and a the pointwise multiplication. We differentiate the step function by considering

---

it linear: this method is called the straight-throghh estimator [3 and it was recently used for similar large-scale masking [4]] Following FOMAML, we ignore second-order derivatives. This leads to the update
K E
- m + I' Brrp(r)
v VoL:" k))
VoL
(5)
detailed derivation of the mask update, alongside the presentation of the initialization update, can be found in the supplementary material (SM).
Our mask update depends on the alignment between the outer-loss gradient g:th V I.g (or, k and the inner loss gradient g.th = Kー VoL." (Øx) accumulated over the inner loop trajectory. Learning tends to be shut off on coordinates for which these two quantities are of opposing sign, E, [ 9r,99 < O. Such freezing of learning when parameter updates are conflicting on the training and validation sets can decrease negative interference across tasks, which can turn improve generalization performance (39 49
3 Few-shot learning
Finding a network that performs well when trained on few samples of unseen data can be formulated meta-learning problem. We study here the supervised few-shot learning setting where tasks comprise small labelled datasets. A loss function C(ó, D) measures how much the predictions of network parameterized by & deviate from the ground truth labels dataset D. During meta-learning, the data of given task r is split into training and validation datasets, D and D., respectively. The sparse-MAML MAML formulation of few-shot learning then consists in optimizing the meta-parameen and m that, given the training set, in turn yield parameters & that improve validation set performance:
sparse-
min Exppp(r) (C( (0, m), 2")]
(6)
s.t. 0r = Q. - a m20 V VoL(or,k: Dz. and Φ,.0 = 8,
This corresponds to setting the outer- and inner-loop loss functions introduced in Section 2 (o"(() = ( D2) and (o) [( D.).
to
We apply sparse-MAML to the standard few-shot learning benchmark based on the minilmageNet dataset [47]. Our main purpose is to understand whether our meta-learning algorithm gives rise to sparse learning by shutting off weight updates, and if the resulting sparse learners achieve better generalization performance. Furthermore, we analyze the patterns of sparsity discovered by sparse- MAMI. over range of hyperparameter settings governing the meta-learning process.
Our experimental setup follows refs. 11 56 unless stated otherwise. particular, by default, our experimental results are obtained using the standard 4-convolutional-lan neural network (ConvNet)
model that has been intensively used to benchmark meta-learning algorithms. As is also conventional,
Source code available at: httt::///ghhbbb..c//ooaaaallaaad
citere_to_leary
100
-80
Final Sparsity (%%)
Sparsity (%)
Iik: 100%
80
Layer
Test (% (9%
65.0
MMML
78%
Sparsity
60
Layer
64.5
Inl: 25%
Layer 4
40
Int: 0%
Outpur
64.0
Sporsiy
0
20
200
400
200
400
0
20
50
100
Epochs
Epochs
Initial sparsity (%)
Figure 1: Gradient sparsity emerges in 5-shot, 5-way minilmageNet classification, standard ConvNet model. Results averaged over 5 seeds + std. Left: Averaged gradient sparsity adapts for different sparsity initializations. Center: Different levels of gradient sparsity for convolutional and output layer weights emerge, with gradually less sparsity from earlier to deeper layers, while all being initialized 50% sparsity. Right: Sparse-MAML reaches higher test set accuracy for higher initial levels of gradient sparsity.

---

we consider two data regimes: 5-shot 5-way, and 1-shot 5 5-way (the term 'shot" denotes the number of examples per class, and 'way" the number of classes). As we vary the hyperparameters of our algorithms, we monitor few-shot learning performance and the gradient sparsity level, defined for parameter group or the entire network as m [Immal?/di((mm) All experimental details can found in the SM.
3.1 Gradient sparsity decreases with layer depth
Our first finding validates and extends the phenomena described by Raghu et al. [4] and Chen et al. [ As shown Figure
Table 1: 5-way few-shot classification accuracy (%) on minilmageNeet, standard ConvNet model. We report mean + std. over 5 seeds. All results except ours taken from the respective papers (we use the symbol --- to indicate missing results). The results for meta-curvature (MC) are not directly comparable as additional data augmentation was used.
MAML. dynamically adjusts gradi- ent sparsity across the network, with very different values over the layers. As an ex- ample, we show the average gradient spar- sity of the four convolutional weight ma- trices and the output layer during training. The same trend is observed for other pa- rameter groups in the network except the output bias (for which sparsity is always high; see SM). Sparsity clearly correlates with depth and gradually increases towards the early layers of the network, despite the similar value before training (around 50%), i.e. MAALL suppresses inner-loop updates of weights in earlier layers while allowing deeper layers to adjust to new tasks. This effect is robust across differ- ent sparsity initializations, with final few- shot learning performance correlating with sparsity, cf. Figure] |
Method
1-shot
5-shot
MAML
48. 48.07=1.75
63.15=00.91
ANIL
46.70=0.40
61.50
BOIL
49.61 16
66.45-t0.37
Meta-SGD 335
50.47#1.87
64.03
MT- 32
51.70 1.84
MC (+data aug.)
41
54.23 1.888
68.47:10.69
Shrinkage p
47.7
exp-MANE MAML
48.38-t0.45
60.01
sparse-ReL ReL.UMMML
40.84
66.80+0.43
sparse-MAMI. sparse-MAML"
50.35
67. 67.03
51.04 10.59
68.05 1.0.54 10.84
These findings validate that our method can discover sparse learning algorithms. Moreover, they show that the level of sparsity is anti-correlated with depth. This result can be interpreted in the light of neural network models with human-engineered patterns of frozen features, which freeze layers of features based on their depth (in combination with MAML, see e.g. ANIL and BOIL, [44] 40)). Our method justifies these approaches, while outperforming them, cf. Table T. suggesting that might be preferable to meta-leamy which features to freeze. We note that another related method for automatic discovery of task-shared weights based on learning per-parantn per-parameter L2 regularization strengths [Shrinkage. [9] yields a similar trend of high freezing for lower-level features, without however improving performance against standard MAML.. Our findings hold when applying our method to deeper and wider residual neural network (ResNet-12 12) model, see Tables 2 and ST (SM). where we observe the same trend of decreasing gradient sparsity with depth emerge.
Sparse-MMLL
Final sparsity (%)
80
set (%)
65
MAMI.
50
Test set (56)
66
sparse-MAML
Final sparsity (%))
63
45
64
40
Sparsity
Sparaily
15
25
20
Gradient steps
10
10
10a
Inner loop learning rate
Figure 2: Sparse learning algorithms found by sparse-MAML work best combination with highly- plastic models. Both gradient sparsity and generalization performance increase with number of inner-loop learning steps (left) and learning rate (right). Standard MAML, which does not employ sparse learning, requires more careful learning rate tuning and does not benefit much from large learning rates. In all experiments, gradient sparsity is initially ~ 50%. The inner-loop learning rate is set to 0.1 when varying the number of steps. Results are for 5-shot, 5-way minilmageNet, averaged over 5 seeds + std.

---

3.2 Sparse learning prefers highly-plastic models
We hypothesize that restricting learning to an appropriate parameter subset allows for longer training and larger changes without overfitting, beyond meta-learning initial parameter values. To verify this hypothesis we scan over different inner-loop learning rates and lengths and compare the resulting test set performances of MAML and sparse-MAML.
First, we test three different inner-loop durations (5, 15 or 25 gradient steps, see Figure 2 left). We find that neither MAML nor sparse-MAML exhibit overfitting for the duration range considered here (for reference, the original study of MAML applied 5 inner-loop steps during meta-training). In contrast to MAML, the solutions found by sparse-MAML generalize significantly better for longer adaptation phases. This improvement in generalization performance is accompanied by an increase in gradient sparsity. Furthermore, applying sparse-MAML. in the very-low data regime of 1-shot learning results in higher levels of gradient sparsity, even though the exact same model and training setup is used for both 1- and 5-shot learning experiments.
We further investigate increasing the learning rate can result in improved generalization performance in combination with sparse learning. We scan the inner-loop learning rate over a large range, cf. Figure (right), and find clear trend towards gradient sparsity going along with better test-set accuracy for larger learning rates. Interestingly, similar effects have been reported in standard (non- meta-learned) neural network training where both freezing layers throughout training [5] and the use of large learning rates 33 seem improve generalization performance.
3.3 Sparse learning vs. more expressive gradient modulation methods
Sparse-MAML can be understood as a bi-
Table 2: 5-way few-shot classification accuracy (%) on minilmageNee with a ResNet- model. We report mean + std. over 3 seeds. We report MetaOptNet [31 figures when no additional regularization techniques are applied. Results from BOIL and MetaOptNet are taken from the respective papers.
nary gradient modulation method. Second- order methods such as meta-curvature [4]
modulate gradients by meta- meta-learning pre- conditioning matrices; in meta-SGI) [35],
these matrices are restricted to be diagonal; sparse-MAMI. further restricts the diago- nal values to be binary. From this point of view, sparse-MAML the least expressive form of gradient modulation.
Method
1-shot
5-shot
MetaOptNet
51.13
70.88
Surprisingly, we find that despite its re- duced expressiveness, sparse-MAML re- covers the performance improvements achieved by the more sophisticated alter- natives, significantly improving the perfor- mance of standard MAML (cf. Table [ We point out that sparse-MAMi MAMML uses a first-order update (Eq, 5), while all three gradient modulation methods we compare to (meta-SGD, meta-curvature and MT-nets) use second-order derivatives that are more costly to evaluate.
MAML
53.91 53.9140.61
#:0611
69.36:11.23
ANIL BOIL
55.25 10.33
70.03
70.50 =0.28
sparse-MAML
55.02
70.02
sparse-eeLU-MAML.
56.39-0033
73.01 +0.24
Sparse learning emerges when meta- learning learning rates. We also im- plement variant of meta-SGD which uses rectified learning rates (sparse-Rel.U) MAML). Concretely, we replace the step function 2 in the inner-loop dynamics (Eq. 4 by the positive part of m, (m)+ := 1 m2 m. Then, we learn the underlying learning rate parameter m using our first- order straight-through update of Eq. to prevent learning rates from getting stuck at zero. Besides standard meta-SGD, which allows learning rates to go negative, we compare this method to an alternative exponential learning rate param- eterization (53), exp m, which like sparse-Rel Re-e-RRUUUMMMM........ enforces non-negativity while avoiding permanently frozen updates (cXp-MAML, Table It is, however, harder to reach sparse learning rate
Table 3: Average gradient sparsity levels (%) after meta- learning on 5-way minilmageNet few-shot tasks, stan- dard ConvNet model. Mean + std. over 5 seeds.
Method
1-shot
5-shot
Oparse-ReLU-MAI sparse-MAML
77.5 77.53=0073
73.53
19.04
74.98
sparse-MAML +
78.05 1.67
76.66 11.13 16.66
5

---

distributions under this parameterization, as the meta- deta-gradient dm L becomes exponentially small as m approaches zero.
We analyze the distributions of learning rates that Ceanse-Rel.U-MMA
yields on minilmageNet and observe that gradient sparsity once more emerges, cf. Table 3 We find that the levels of gradient sparsity and generalization performance when meta-learning binary (sparse-MAML) or rectified learning rates DiparseReLLUMMMMLL are approximately the same, with both methods outperforming exp-MAML on both I-shot and 5-shot tasks. These results support the hypothesis that shutting off weight updates one of the essential gradient modulation operations few-shot learning. We note that while sparse-MAML MAML and sparse- sparse-ReLUs ReLL-MMMAAAN quickly disable learning in a large fraction of weights, exp- 1MMM tends to push learning rates down, in particular for layers close the input, but at much slower pace; increasing the meta-lcarning rate Y'm cannot compensate for this slowdown as learning becomes unstable (data not shown).
This picture changes for the the deeper and larger ResNet-12 model, cf. Table 2 When using this more complex architecture, we find that sparse rectified learning rates Apparse-ReLU-MMNN beneficial over binary gradient masks (sparse-MAML). In particular, the combination of sparse learning (see Table [ST for additional details) with learning rate modulation found by sparse-ReLU- MAML outperforms all other methods, including standard (dense-learning) MAML, as well as methods based on manually freezing layers in the inner-loop: BOIL 40), ANIL 44 and the closely- related MetaOptNet [3]] method. Like ANIL. MetaOptNet only adapts the final classification layer in the inner-loop. but uses more sophisticated solver instead of a few steps of gradient descent to learn task-specific solvers. Thus, once more, learning by sparse gradient descent is an effective strategy to improve the generalization performance of few-shot learner.
are
Stochastic gradient masking. We further investigate whether stochastic binary gradient masks can improve few-shot learning performance. Our interest in studying stochastic masks is two-fold: as way improve meta-optimization based on our straight-through estimator; and to determine if stochastic masking is beneficial at meta-test time. We thus investigate sparse-MAML.- a variant of algorithm in which gradient masks are generated from low-dimensional Gaussian vector, with noise intensity determined by meta-learning (see SM), As before, we adjust meta-parameters using a first-order update. We find that this mask generation method does result in improved performance, cf. Table I Interestingly, mask randomness is entirely suppressed by meta-learning; eventually, + and we recover single deterministic mask m. The performance improvements observed on few-shot learning therefore stem from improvements to the meta- ootrmoizttiiiionnnn process, likely related to the challenges of optimizing binary variables with grradditgrddeentt.. based methods.
3.4 Sparse learning improves performance in cross-domain adaptation tasks
We now investigate whether the patterns of gradient sparsity discovered by our method overfit to the particular task family where they were obtained, namely, to few-shot minilmageNet classification
Table 4: Few-shot classification accuracy (%) when meta-learning minilmageNet but meta-testing on TeeredImageNet. CUB and Cars. Mean + std. over seeds.
Problem
Method
TirredImageNet
CUB
Cars
MAML
51.61 20.20
40.01
33.57#0.14
ANIL
52.82
41.12=0.15
34.77+0.31
BOIL
53.23 10.41
14.20-L..1
36.12
1-shot
sparse- Reuse-RRLU-MMMLL
53. +3.12
41.86=0.95
35.46+0.67
sparse-MAML
53.47
41.37=20778
35.90
sparse-MAML+
53.91 +0.67 #0.67
43.43=1.01
37.14
MAML
65.76
53.09
44.56-10.21
ANIL
66.52
55,82
46.55_10.29
BOIL
69.37
60.92+0.11
50.64+0022
5-shot
sparse-ReLU-MAML
69.06
69.06+t0.28 +0.28
19,55
51.21 $1.21 50.89
sparse-MAML
60.65
60.58+1.10
52.63
sparsc-MAML
69.92 10.21
62.02
53.18 L0.44

---

tasks. This is an important question, since excessive parameter freezing may prevent adaptation to tasks that are too different from those presented during meta-learning
We therefore move our analysis of few-shot learning to a cross-domain adaptation setting. In evaluate
cross-domain adaptation problems, the family of tasks presented post-meta-learning algorithms is shifted by sampling classes from a different dataset. In particular, we train our meta- learner on the minilmageNee dataset and then evaluate learning performance on the TieredImageNet, CUB and Cars datasets. It has previously been demonstrated that manually freezing either the head (BOIL) or the body (ANIL) during meta-testing improves performance in this setting [40], compared to letting all weights adapt (MAML). In Table we compare the performance of our method to these baselines. We find that meta-learning the freezing pattern with sparse- -AMM as opposed to manually selecting it consistently improves cross- doss-n adaptation.
4 Continual learning
We now turn to a continual learning setting, where tasks must be learned sequentially. A successful continual learner is able to learn similar tasks faster, as in the few-shot learning case, while retaining high performance on previously seen tasks. We conjecture that sparse learning can improve memory retention and accelerate future learning by reducing interference with past updates.
4.1 Gradient sparsity emerges when learning continually with Look-ahead MAMI.
We investigate the benefits of sparse gradients the recently proposed La-MAML algorithm [5]. This algorithm combines online meta-learning in conjunction with small replay buffer which holds representative examples from the past in memory. Standard replay methods (51), define joint objective using present and buffered data and directly optimize this objective. La-MAML follows a technique known as meta-experience replay [49] and introduces a bi-level optimization problem. The outer loss L is the multi-task objective optimized with standard replay methods, while the inner loss i is evaluated on the new incoming data only. Riemer et al. [49] have shown that such meta-learning promotes gradient alignment over tasks, which is a way reduce interference [36].
Like the variants of MAML reviewed Section 2 La-MAML introduces meta-learned per-parameter learning rates. We now briefly review single iteration of the algorithm; complete pseudocode provided in the SM. Each iteration of La-MAML consists of processing a new batch data B as follows: (i) starting from do 0 taking inner-loop step on each sample in B, with (Øx) = L(dx, B&); (ii) defining an outer-loss (Øx,, B U R) on both new data B and a batch of past data R sampled from the replay buffer;
(ii)) taking outer-loop step on the learning rate parameter using a first-order update followed by an outer-loop step (using the newly updated learning rate) on the neural network parameters 0; (iv) re-populating the replay buffer with data in The
100
Retained acc (96)
80
30
Final spersity (%)
Sparsity (%)
Sparsity (%)
50
1ys
W
1
ly1
70-
Accuraty Sparsty
20
1y
60
0
10
15
20
5
10
15
20
20
50
100
250
500
Tasks
Tasks
Layer size
Figure 3: Gradient sparsity when learning MNIST rotations with the La-MAML and sparse-La- MAML algorithms. Results averaged over 3 seeds + std. Left: Sparsity emerges on the original La- La-MAML algorithm across the three layer network and monotonically increases with the number of tasks and with depth for both weight (W1, W2, W3) and bias parameters (61, b2, by). Center: A similar behavior is observed when replacing meta-learned learning rates by meta-learned binary gradient masks (sparse-La-MAMM)
Right: Overall sparsity of Larrs-La-MMAL. decreases with increased network capacity accompanied with higher retained accuracy (RA). Network capacity is varied by changing the number of neurons in the two hidden layers simultaneously.
7

---

sequence of inner-loop updates is given by
@ σ2 - (a)+ V VoLK((nn)
s.t. Φ = 0,
where is a vector of learning rates whose components are constrained be non-negative by elementwise application of the positive part function. We note that while the main text of ref. [T5] presents an inner-loop learning rate parameter that is allowed to go negative, the implementation for the experiments reported in ref. [16] uses rectified learning rates. In this implementation. a learning rate that is updated below zero will never recover, which can lead to dead coordinates and promote sparsity. The inner loss T.f (Φ) is defined on different data sample on each step k. A first-order update is applied to again modulated by the adaptive learning rate:
0 0 - (a)+ VoL (Φx).
Sparse-La-MAMI. Our sparse- -AMML can be readily applied to continual learning problems by modifying the inner- and outer-loop updates of La-MAML. We replace the meta-lcarned learning rates in equations 7 By by meta- learned binary gradient masks, a au m with og E R+ some scalar (fixed) learning rate value. To learn the underlying parameter m, we again resort to our first-order update (equation 5)
Spaadd (%5)
50
6
W
6
0
1
2
3 Tasks
4
5
Figure 4: Structured sparsity emerges and tends to converge within a task in multi-pass continual learning. Results shown for La-MAML after training on MNIST rotations, averaged over 3 seeds std., for weight layers (W1, W2, W3) and bias parameters (b1, b2, by). Sparsity increases with depth.
Sparse learning improves continual learning. We hy- pothesize that large fraction of learning rates approaches zero when the hyperparameters of La-MAML and sparse- La-MAML are tuned for best continual learning perfor- mance. To test this hypothesis, we follow the exact same setup as in the original study of La-MAML [1S. We perform grid search over the learning rates (ig and T'm and search for best continual learning performance, not sparsity (cf. SM). The remaining hyperparameters are kept to the values provided in [5].
We study the three MNIST 30 continual learning problems rotations, permutations and many permutations using a singlc-headed network using the code accompanying ref. [1]]. Task information is not given to the network, and each data point is seen only once, unless noted otherwise. Full details as well as additional experiments using the CIFAR-10 10 (28 dataset are provided in the SM.
We verify that our initial hypothesis is correct: La-MAML shuts off learning in many coordinates (cf. Figure 3 full results may be found in the SM, Table S5 reaching even higher levels of sparsity than Lparse-La-MAML. This can be explained by the fact that dead coordinates can arise in La- MAML, which can lead to excess sparsity. By contrast, our straight-through update dynamically and continually adjusts the pattern of sparsity allowing previously frozen parameters to be unfrozen. This results in matching or slightly improved performances when using our binary gradient mask across all three MNIST variants, see Table 5) both in terms of final retained accuracy (RA) and backward- transfer and interference (BTI; the change in accuracy measured at the end of the experiment minus just after learning a task, averaged over tasks). Moreover, the patterns of sparsity adjust to the capacity of the network, decreasing and eventually vanishing for larger models (Figure B as retained accuracy goes up, indicating that the task is not sufficiently difficult to create interference on large capacity models.
As in our few-shot learning experiments, structured sparsity emerges across the different parameter groups of the network (cf. Figure B}. We observe that now sparsity is highest closest to the output layer, the exact opposite of the trend found in our few-shot learning experiments. This provides evidence that online meta-learning can discover how to rewire low-level features without interference in order to accommodate different tasks that share high-level structure. We further investigate a multi-pass setting, where the examples from each task are visited multiple times (10 epochs instead of 1) before proceeding to the next task. In this setting. it can be seen that sparsity levels (displayed in Figure 4 tend to converge within tasks and then raise again when tasks switch, presumably to preserve past memories via gradient sparsification. Taken together, our results support the hypothesis that gradient sparsity is beneficial for continual learning and that appropriate patterns of sparsity can be discovered by simple online basedd meta-learning
8

---

Table 5: Retained accuracy (RA) and backward-transfer and interference (BTI) for three different MNIST continual learning problems: rotations, permutations and many permutations. We report mean std. over 5 seeds. Negative BTI values closer to zero imply less forgetting and are therefore better. Results of related work are taken from [[]; for completeness we include the GEM [36 and MER 44 methods next to a stochastic gradient descent baseline. Although sparse-La-MMAM (sp-LaMM) is strictly less expressive than the original La-MAMI. algorithm, it shows competitive performance across all variants and both metrics. The lower baseline BTI values can be explained by
lower overall accuracies achieved by La-MAML.
Method
Rotations
Permutations
Many permutations
RA
BTI
RA
BTI
RA
BTI
Baseline
53,38 1.53
1..4
55.42
-13.76
32.62 #044
-19.66 00.86
GEM
67.38 -18.02=1.00
55.42=
-24.42=t1.10
32. 32.14 10.50
-23.52=0.87
MER
77.42
-5.60
73.46-b0.45
-9.96=0.455
47.40 0.335
-17.78
La-M
77.4
-8.64 #0.40
74.34+
-7.60=0.51
48.46
-12.96+0.07
sp-LaM
77.77=0.68
-8.16 #0.61
76.88+00.72
-8.39=0.63
50.81 70.79
-13.73#0073
4.2
Sparse online learning
We finally consider another online learning setting in which the underlying task is con- cealed from the learner and can randomly change at each step. potentially going back
Table 6: Sparse learning improves continualMMAML performance. Cumulative online accuracy on Omniglot- MNIST-FFshhonMMNIST benchmark. Tasks switch with probability p. Results from previous work taken from 77. Mean + std. over 5 seeds.
to previously seen tasks [50 19 7. At each time step t, the data D, is an i.ddd sam- ple from a stationary distribution that only depends on the current task. The learner, whose current state is denoted by ot. is evaluated whenever new data is presented and modifies its behavior accordingly. The goal is then to minimize the cumulative loss Σi= L( De) measuring the perfor- mance before adaptation takes place.
Method
p 0.98
p 0.9
Online Adam 25
73.9
23,8+112
Fine-tuning
72.7
22.1 #11
MAML 11
84.5#1.7
75.5+0.7
ANIL 44
75.3+2.0
69.1
BGD
660
87.8
63,4
MetaCOG 19
88.0+1.0
63.6
MetaBGD
91.1
74.
This online learning protocol differs from the one adopted in the previous section, where tasks were visited only once and only the final loss ΣT L(Ør, Dr) evaluated ér mattered. The cumulative loss criterion emphasizes fast learning and adaptability while memory is still needed avoid re-learning, since tasks can be re-encountered. Recently, it has been shown that a simple modification of MAML Continual-MAMM.. 7 can outperform a number of algorithms specifically tailored for this setting as well as plain stochastic gradient descent 4 Briefly, Continual-MAMi extends MAML by introducing task-switer detection mechanism based on changes in loss; data is buffered until switch is detected. When this occurs, the buffered data is used to perform meta-parameter update; the buffer is reset; and the inner-loop optimization restarts. Here, we merge continual-MAMI. with MAArse-MAML, and modulate inner-loop gradients according to equation 4 We present complete pseudocode for the algorithm in the SM.
C-MAML
92,8=t0.6
83.3+0.1
sparse-C- sparse-C-MAMI. C-MAMI
94.2=004
86.3+0.1
Dparse-ReLU-C--MNN
93.5
86.1
We reproduce the experiments of 7 in which a sequence of 10000 examples from the Omniglot [29], MNIST [30] and FaShionMNIST [59] datasets presented for online learning single-headed neural network, using the code provided by the authors. We carry out grid search to tune the inner-loop learning rate ag and the mask learning rate "m introduced by sparse-MAML for best performance, not sparsity (see SM). We observe again structured (layer-dependent) gradient sparsity emerge when using this algorithm (sparse-C-MAMI)
as shown Figure SS5 and an increase in cumulative online learning accuracy over the original continual-MAMI. algorithm (cf. Table (). Finally, we observe the same qualitative behavior (see SM for sparsity levels) and obtain similar performance when replacing our binary masks by rectified learning rates (sparse- Sparse-RRLL-CCE ReLU-C-MALL

---

meta-learned with our throughhtth update. These findings once more support the hypothesis that sparse learning, and not learning rate modulation, lead the improved performance reported here.
5 Discussion
We studied gradient- based meta-learning systems with the ability of learning where to learn. This was modeled by adding binary variables which masked gradients on per-parameter basis, therefore determining which parameters are allowed to change. We observed gradient sparsity emerge stan- dard few-shot and continual learning problems, without introducing an explicit bias towards sparsity. This form of sparse learning, which may be understood as sparse gradient descent, was accompanied by overall improvements in generalization, as well as reduced interference and forgetting.
Previous work on gradient modulation has focused on estimating task-shared loss geometry pre- condition the optimization procedure [35 62 12
32 61 In addition, stochastic variant of gradient masking was featured in the MT-net algorithm 32 as part of more complex model. Our approach differs from these previous studies in its simplicity. We restrict gradient modulation to be binary and deterministic and use an inexpensive first-order update to learn the gradient masks. In contrast to traditional methods for inducing sparsity via regularization [54] (here, gradient regularization) our approach does not require evaluating second derivatives, which would result from differentiating gra- dient regularizers. Despite these simplifications, we find competitive performance our experiments. These results point towards sparse gradient descent as a powerful learning principle.
The idea of meta-learning learning rates can be traced back to the seminal work of Sutton [53 who proposed to estimate learning rate meta- grata-ggaiiets online using forward mode automatic differen- tiation, and to use consecutive batches of data to define inner- and outer-loop loss functions. This approach, known as stochastic meta-descent (SMD), was extended to nonlinear models by Schrau- dolph 52 using fast Hessian-vector product techniques. Using SMD to optimize neural network is an interesting question whether gradient sparsity emerges when applying SMD to online learning problems that are not clearly structured in tasks, as considered here. Furthermore, this line of work suggests that it might be possible to obtain finer binary gradient mask updates in online fashion using forward-mode automatic differentiation.
models an ongoing area of research [55 58
22 24].
recent study has put into question whether any useful adaptation still takes place when MAML few-shot learners are presented with a novel task after meta-learning 44 Our findings shed light on this question, by demonstrating that few-shot learning performance can be improved when learning an adequate small subset of parameters. The additional plasticity of our meta-learner sparse learners led to a significant performance increase over handwired schemes based on frozen layers, in particular when encountering tasks drawn from a different family of problems than that used for meta-learning This finding complements recent work showing that modular recurrent networks with sparse updating mechanisms hold great promise in improving out-of- diitr---ittiiitiion generalization performance [14] 38
Our results may be of special interest the design of neuromorphic hardware. Updating weights on-chip implies a significant power overhead whose cost scales with the number of plastic weights Reducing the number of plastic weights can therefore result in immediate improvements in energy efficiency and scalability. Likewise, synaptic plasticity is costly in biological neural networks. Given the high energy demands of the brain there has likely been selective pressure reduce costs associated with synaptic change (34. It is therefore conceivable that the brain developed mechanisms to restrict learning to an appropriate subset of synapses to save energy. Our study presents further evidence in favor of sparse synaptic change, given its potential benefits in the biologically-relevant scenarios of fho-shot and continual learning investigated here.
42
Limitations. To arrive at our simple mask update (cquation 5 we introduced two important approximations. First, we dropped all terms involving second-order derivatives, and second, we used straight-through estimation to differentiate through the step function. Despite their frequent use in previous work, both approximations remain poorly understood. For this reason it possible that our algorithm fails unexpectedly outside the experiments considered here; potential problems stemming from the con-differentiability of the step function are likely unavoidable in our approach. Our update is also likely inappropriate for long inner loops (large K) [58]. Finally, scaling our few-shot learning experiments to more complex neural network models is potentially difficult, a challenge that our approach shares with other methods based on MAML.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by an Ambizione grant (000002771177)) awarded to João Sacramento by the Swiss National Science Foundation. Johannes von Oswald is funded by the Swiss Data Science Center (J.v... P18-03). Dominic Zhao is supported by AlayaLabs (Montreal, Canada). Massimo Caccia was supported through MITACS during his part time employment with Element Al the ServiceNow company, and by Amazon, during his part time employment there. We thank Charlotte Frenkel, Frederik Benzing, Angelika Steger and Laura Sainz for helpful discussions.
References
[1] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. arXiv preprint arXiv: 1908.00420 2019.
[2] Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In International Conference on Learning Representations, 2019.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
[4] Léon Bottou. On-line learning and stochastic approxinations. In On-line Learning in Neural Networks, pages 42. Cambridge University Press, 1998.
[5] Andrew Brock, Theodore Lim, J. M. Ritchie, and Nick Weston. FreezeOut: Accelerate training by progressively freezing layers. arXiv preprint arXiv: 0470000049, 2017.
[6] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard E. Turner. Tasknorm: Rethinking batch normalization for meta-learning. In International Conference on Machine Learning, 2020.
[7] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Cacciaa Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation (OSAKA): a new approach to continual learning. Advances in Neural Information Processing Systems, 2020.
[8] Wei-Ye Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
[9] Yutian Chen, Abram L. Friesen, Feryal Behbahani, David Budden, Matthew W. Hoffman, Arnaud Doucet, and Nando de Freitas. Modular meta-learning with shrinkage. Advances in Neural Information Processing Systems, 2020.
[10] Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torch- meta: A meta-learning library for Py PyTochh arXiv preprint vrXiv:1999006577 2019.
[11] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning. 2017.
[12] Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
[13] Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-135, April 1999.
[14] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2021.
[15] Gunshi Gupta, Karmesh Yadav, and Liam Paull. La-MAMI:: Look-ahead meta learning for continual learning. In Advances in Neural Information Processing Systems, 2020.
11

---

[16] Gunshi Gupta, Karmesh Yadav, and Liam Paull. Official LeMMaiL Downloaded http:///ww.o...oo....ooooaaaaaaa
from: https://github. py 2020. Accessed: 2020-05-25.
[17] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):3573322 2020.
Kevin Sheppard, Tyler
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: surpass- ing human-level performance ImageNet classification. In IEEE International Conference on Computer Vision, 2015.
[19] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv: 1906.99000000 2019.
[20] J. D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9 (3):90-955 2007.
[21] Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 2015.
[22] Andrew Jacobsen, Matthew Schlegel, Cameron Linke, Thomas Degris, Adam White, and Martha White. Meta-descent for online, continual prediction. In AAAI Conference on Artificial Intelligence, 2019.
[23] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, 2019.
[24] Alex Kearney, Vivek Veeriah, Jaden Travnik, Patrick M Pilarski, and Richard Sutton. Learning feature relevance through step size adaptation in temporal-difference learning. arXiv preprint arXiv:190..03252. 2019.
[25] Diederik P. Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In International Conference on Learning Representations, 2015.
[26] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinste Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521-3526, 2017.
[27] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), 2013.
[28] Alex Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
[29] Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2011.
[30] Yann LeCun. The MNIST database of handwritten digits. Available at http://yann. lecun. com/exdb/mnist, 1998.
[31] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[32] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, 2018
12

---

[33] Aitor Lewkowyez, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02216 02218, 2020.
[34] Ho Ling Li and Mark C.W. van Rossum. Energy efficient synaptic plasticity, eLife, 9:e50804, 2020.
[35] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few shot learning. arXiv preprint arXiv: /07.0098777098333 2017.
[36] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
[37] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter op- timization through reversible learning. In International Conference on Machine Learning. 2015.
[38] Kanika Madan, Rosemary Nan Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. In International Conference on Learning Representations, 2021.
[39] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint 10X3vv8803.020000 2018
[40] Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. BOIL: Towards representa- tion change for few-shot learning. In International Conference of Learning Representations, 2021.
[41] Eunbyung Park and Junier B. Oliva. Meta-curvature. In Advances in Neural Information Processing Systems, 2019.
[42] Jeongwoo Park, Juyun Lee, and Dongsuk Jeon. A 65-nm neuromorphic image classification processor with energy-efficient training through direct spike-only feedback. IEEE Journal of Solid-State Circuits, 55(1)) 2019.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems 32, 2019.
[44] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In International Conference on Learning Representations, 2020.
[45] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular vector canonical correlation analysis for Deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, 2017.
[46] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in a randomly weighted neural network? In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[47] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In Interna- tional Conference on Learning Representations, 2017.
[48] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenen- baum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for cuni-supervised few-shot classification. In International Conference on Learning Representations, 2018.
[49] Matthww Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
13

---

[50] Samuel Ritter, Jane Wang, Zeb Kurth-Nelson, Siddhant Jayakumar, Charles Blundell, Razvan Pascanu, and Matthew Botvinick. Been there, done that: Meta-learning with episodic recall, In International Conference on Machine Learning, 2018.
[51] Anthony Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7 12):123-146, 1995.
[52] Nicol N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference Artificial Neural Networks, 1999.
[53] Richard S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI Conference on Artificial Intelligence, 1992.
[54] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1:2667-288 1996.
[55] Vivek Veeriah, Shangtong Zhang, and Richard S. Sutton. Crossprop: Learning representations by stochastic meta-gradient descent in neural networks. In Machine Learning and Knowledge Discovery in Databases, 2017.
[56] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances Neural Information Processing Systems, 2016
[57] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-UCSD birds 200. Technical report, CNS-TR-2010-001 California Institute of Technology, 2010.
[58) Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations, 2018.
[59] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for 2017..
benchmarking machine learning algorithms. arXiv preprint
[60] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes with fixed-point updates. arXiv preprint arXiv:010.00373. 2020.
[61] Dominic Zhao, Johannes von Oswald, Seijin Kobayashi, João Sacramento, and Benjamin F.
Grewe. Meta-learning via hypemetworks. In NeurIPS Workshop on Meta-Learning, 2020.
[62] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
14


Reviews:
[{"review_id": "dfT9ffvcIS9", "reviewer": "Reviewer_Lbaj", "summary": "This paper proposes a strategy called Sparse-MAML, which extends the MAML algorithm to learn sparse updates.\n\nThe Sparse-MAML algorithm is applied in two settings: few-shot learning with MiniImageNet, and continual learning with MNIST rotations and permutations. The results show a mixed performance – some gains, some losses, compared to different baselines. The authors also provide many ablation studies and analysis which show that Sparse-MAML does exhibit the sparsity in the gradient updates of MAML.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclaimer: I am not familiar with the benchmarks presented in the paper, namely miniImageNet and MNIST rotations/permutations. Therefore, I cannot appreciate the experimental results in the paper. I can only provide my assessment of the paper based on comparing the numbers provided therein, without much further insight.\n\nOn the methodology side, sparsity is a preferred property in many deep learning settings, so the motivation of learning sparse updates in this paper is reasonable. The formulation of Sparse-MAML, which uses a ReLU vector to shut off the gradient updates to various parameters is simple and not computationally intensive. On top of this, the paper is clearly written and easy to follow.\n\nOn the experimental side, the authors do provide analysis which confirms that Sparse-MAML indeed leads to sparser updates to the model parameters. My main concern is the seemingly weak performance of the Sparse-MAML in terms of its resulting model’s qualities. For instance, in Table 1, Sparse-MAML loses to MT-Net and Meta-SGD, and in Table 2, Sparse-MAML sometimes loses to ANIL and BOIL. While the performance of Sparse-MAML+ is sometimes better, Sparse-MAML+ seems to be less sparse than Sparse-MAML (as the authors explain that “gradient masks are generated from a low-dimensional Gaussian vector”). This raises the question whether sparsity should be enforced to the extent of Sparse-MAML.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I cannot comment since I have almost no insight about the experiments in the paper.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.\n", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "TzbUt1XpYww", "reviewer": "Reviewer_bJTT", "summary": "The paper proposes a sparse meta gradient update method during MAML by adopting binary masks that modulate the gradient during the inner update. The masks depend on the angle-similarity between the averaged inner loss gradients and outer loss gradients, where two gradients are opposite directions, then mask to be zero, otherwise one. The method shows better performance compared to MAML and existing meta-learning baselines using the modulated learning rate strategies. Furthermore, the method can be combined with meta continual learning methods, such as La-MAML, and shows superior performances.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and easy to follow. \nThe methodology is simple and reasonable, several ablation studies and analyses are helpful for a better understanding of the proposed method. Although the performance gain compared to recent baselines are not much impressive and sometimes looks marginal, I think this is good work. \n\nQuestions and comments are as follows:\n\n- In Figure 1, performance is also robust across the different sparsity initializations?\n\n- Recent Continual Learning literature deploy CL model on modern deep neural networks, such as ResNet 18 or MobileNet v2. Even La-MAML validates their method on simple CNN, I strongly recommend validating the proposed method on modern nets.\n\n- In Figure 4, why does the sparsity decrease when layer sizes increase? As the network has additional neurons, it looks natural that there are more redundant/unnecessary updates (sparsity can be $\\uparrow$).\n\n- Please describe the merits/drawbacks of meta-learning-based continual learning, compared to other CL directions (e.g., regularization-based, rehearsal-based, architecture-based).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No issue.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GltOf3NNyly", "reviewer": "Reviewer_u5xJ", "summary": "The authors introduce a new version of the MAML algorithm which has learnable sparsity on the weights. Using a linear, first order approximation to the masking gradients, they present a tractable algorithm for meta-learning the sparsification of the gradients along with the rest of the parameters. They show small improvements on relevant fewshot learning and continual learning tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciated the exposition which described the sparse MAML algorithm; it gave a clear path to converting understanding of MAML to an understanding of sparse MAML, complete with an interpretation of the update steps. In fact, this interpretation helped me understand why the gradient approximation may be reasonable, and under what conditions it might fail.\n\nIn the fewshot and continual learning settings, the sparse MAML algorithm seems to generally perform better than vanilla MAML, and is competitive with other SOTA aproaches. In addition, the analysis of the learning dynamics of sparsification as well as the layer-wise analysis of the sparsification is interesting.\n\nOne concern is the fact that the final sparsification factors are not large (2x - 4x gradient reduction) - which to me suggests that the sparse MAML may be a useful final step after all other optimizations have been done, but will never provide qualitatively different results. Researchers may be better served by tuning architecture hyperparameters over tuning sparsity. A related concern is how the sparsifying method fares when pushed into a larger input dimension/dataset/model size regime. This concern was noted by the authors in their conclusion, but may limit the overall impact of the work.\n\nOne interesting thing to know would be the rate of reversion of the masks - that is, how often is a gradient turned on after it has been turned off? If reversion is rare, then it may have implications for the efficiency of the algorithm, as well as the general learning dynamics it induces.\n\nUpdate: The authors addressed many of my concerns. They showed that the sparsification has to be learned simultaneously with the overall training. They also provided an interesting analysis of the rate of reversion of masks (most don't revert, but a non-trivial fraction do). Given these strong additions, I am changing my review from a 5 to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8Oojcc-h_Al", "reviewer": "Reviewer_HSzX", "summary": "The author proposes modulating learning using masking some learning updates. The value of the mask is generated using a simple LTU unit with zero threshold. The mask is learned using the first-order approximation to MAML (FOMAML). The authors use the straight-through estimator because LTU is not differentiable. \n\nThe paper claims that (1) update sparsity emerges through this meta-learning process, (2) different layers have different level of sparsity, and the level of sparsity depends on the problem, (3) binary sparsity performs better than simply meta-learning the step-sizes, (4) sparsity improves the meta-testing performance on unseen task distributions and (5) sparsity improves continual learning performance for online meta-learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, and empirically tests the proposed approach on multiple environments and settings. The central idea --- meta-learning to modulate learning in different parts of the network --- is sound, and generalizes both MAML and ANIL.\n\nIt's not clear to me why a binary mask learned using a biased method (Straight-through estimator) would be better than simple meta-learning step-sizes. I would expect both to do well, and it's unclear to me how much of this difference would actually generalize to a bigger set of problems. \n\nOne issue I have with the paper is that it ignores all prior work done on meta-learning step-sizes that wasn't done in the past 5 years. Two relevant pieces of work (more thoroughly discussed in Limitations sections) are IDBD (Sutton, 1992) and SMD [2]. There has been follow-up work to SMD as well that the authors should include in discussion. I'm happy to recommend this paper for acceptance once a discussion on prior work and an exponentiated version of step-size has been added as a baseline. \n\n\nSutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (pp. 171-176).\n[2] Local Gain Adaptation in Stochastic Gradient Descent∗ Nicol N. Schraudolph", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: \n1. The paper misses prior work on meta-learning for step-sizes. Two seminar works that should be discussion are IDBD (Sutton, 1992) and SMD [2].\n1. Additionally, Instead of using ReLU to make step-sizes positive in Meta-Sgd, the paper should use an exponential form for the step-size as used by IDBD (Sutton, 1992)  and SMD[2]. This assures that the step-sizes are always positive, and differentiable. I would expect the exponential form to fix most of the issues related to dead ReLU masks and result in a more robust system. \n\n\nSutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (pp. 171-176).\n[2] Local Gain Adaptation in Stochastic Gradient Descent∗ Nicol N. Schraudolph", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension to MAML that also meta-learns weight update masks, and analyses the resulting learning outcomes under a variety of settings. Both conventional few-shot learning and continual learning benchmarks are considered. Although some work has been done in this direction such as MT-Net, reviewers are generally positive  after rebuttal phase and find this to be worthwhile contribution. Overall I recommend acceptance.
Decision: Accept (Poster)