Learning where to learn: Gradient sparsity in meta and continual learning

Abstract: Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.


Reviews and author responses:

 We would like to share one last update on the exponential learning rate model. We reran from scratch our experiments with this model and tracked the time-evolution of the learning rates as meta-learning proceeded, cf. figure linked below on an anonymous image hosting online service (corresponding to 1-shot miniImageNet learning):

https://postimg.cc/8fm5ZGcg

At the optimal meta-learning rate, the learning rates approached zero slowly. This behavior is distinct from the fast discovery of zero values and dynamic fine-tuning (cf. mask reversion analysis, https://openreview.net/forum?id=CxefshFHEqh&noteId=oTPi8vDPz5) behavior exhibited by the linear straight-through update. This can be explained by the $\exp m$ factor that scales the meta-gradient, as per our previous post.  For higher meta-learning rates, we found training to be unstable. Even at the optimal meta-learning rate, there are outliers with very large learning rates, indicating a tendency for meta-optimization instabilities under the exponential parameterization.

For illustration purposes we provide below the learning rate time-evolution for a toy model where a single learning rate is subject to a sequence of negative updates under the thresholded-linear and the exponential models, the latter for two different meta-learning rates. This simple illustration shows the fast decrease (at the expense of potential instabilitites, when the learning is subject to positive updates, not shown in figure) followed by a slow-down which is characteristic of the exponential model:

https://postimg.cc/BP4ZLzgY

We further note that due to the $\exp m$ factor in the meta-gradient, when the exponential learning rate is at a small value, increasing away from it is difficult.

We would like to remark that we do not see these results as contradicting the benefits of meta-learned sparse gradient descent. At the optimal hyperparameter setting, the exponential learning rate model leads to a large number of small learning rates, and performance improves over vanilla MAML. What our experiments suggest is that first-order joint meta-learning of an initialization and a set of learning rates is more effective when using thresholded learning rates, and straight-through estimation techniques to deal with the threshold. On top of the benefits of gradient sparsity, we argue that the simplicity and effectiveness of this approach in finding sparse gradient descent learning algorithms is of interest to the community.

We hope that these analyses satisfy your concerns and that you find our paper worthy of acceptance.

 Dear reviewer HSzX,

Thank you very much for your message. We apologize for the overly simplified explanation provided in the reply above. Note that in the exponential learning rate model $\alpha = \exp m$, the first-order approximation of the loss meta-gradient $\frac{d L^\textrm{out}}{dm}(\phi_K) \approx - \exp (m) \circ \nabla_\phi L^\textrm{out}(\phi_K) \circ \sum_{k=0}^{K-1} \nabla_\phi L^\textrm{in}(\phi_k)$ is scaled by $\exp m$, which can lead to vanishing gradients. Due to this additional factor it is on the one hand difficult to reach effectively sparse gradients, and on the other hand difficult to revive 'dead' learning rates.

After reading your message we have inspected the histograms of the learning rates obtained after meta-learning, and we found that $m$ follows a unimodal distribution that is roughly Gaussian (except for the final output layer); the mean and standard deviations of the learning rates of the convolutional layer weights, in order of increasing depth, are as follows: $-5.9732 \pm 2.1541$, $-3.6711 \pm 2.1336$, $-3.0934 \pm 2.1108$ and $-0.9293 \pm 1.9758$. Therefore, as the reviewer correctly guessed and in line with the main hypothesis of our paper, meta-learning once again finds sparse gradient learning algorithms (approximately, in this case). We speculate that the gap in performance towards sparse-MAML and sparse-ReLU-MAML can be attributed to vanishing gradients under the exponential model. We note that our straight-through update (eq. 5), which is not affected by vanishing gradients, leads to a significant number of mask transitions in early phases of meta-learning (cf. mask reversion analysis in reply to Reviewer u5xJ, https://openreview.net/forum?id=CxefshFHEqh&noteId=oTPi8vDPz5), which can be the reason for its improved performance.

In case of acceptance, we will include these arguments and an analysis of the learning rate distributions in the final version of the paper. Thank you once again for the helpful comments and experiment suggestions, which were truly useful in improving our work. As before, we remain open for discussion and further analyses if your concerns are not yet fully addressed.

 Thanks for the response. 

The authors said the exponential form of mask did not work well because parameter freezing is essential for gradient modulation, but parameter freezing is exactly what the exponential form achieves. In-fact, the exponential form was designed with the goal to take geometric steps in the step-size parameter so the meta-learner can easily set the step-size to very small values (essentially freezing it) or very large values quickly, while staying differentiable. A step-size of 1e-10 is not different from a step-size of 0 in any meaningful way.

I have spent dozens of hours comparing step-size adaptation methods in my work and have always found exponential version to be a very strong baselines. I wonder what else could cause the poor performance of the exponential form in these experiments? Did the network learn very small step-sizes for certain parameters with the exponential form? 

Nonetheless, the authors did address my main concern and I'm increasing my score by one point. I think the paper can address the hypothesis "Parameter freezing is essential for gradient modulation" more clearly than it does, because the bad performance of the exponentiated version contradicts their hypothesis. 

The author proposes modulating learning using masking some learning updates. The value of the mask is generated using a simple LTU unit with zero threshold. The mask is learned using the first-order approximation to MAML (FOMAML). The authors use the straight-through estimator because LTU is not differentiable. 

The paper claims that (1) update sparsity emerges through this meta-learning process, (2) different layers have different level of sparsity, and the level of sparsity depends on the problem, (3) binary sparsity performs better than simply meta-learning the step-sizes, (4) sparsity improves the meta-testing performance on unseen task distributions and (5) sparsity improves continual learning performance for online meta-learning.   The paper is clearly written, and empirically tests the proposed approach on multiple environments and settings. The central idea --- meta-learning to modulate learning in different parts of the network --- is sound, and generalizes both MAML and ANIL.

It's not clear to me why a binary mask learned using a biased method (Straight-through estimator) would be better than simple meta-learning step-sizes. I would expect both to do well, and it's unclear to me how much of this difference would actually generalize to a bigger set of problems. 

One issue I have with the paper is that it ignores all prior work done on meta-learning step-sizes that wasn't done in the past 5 years. Two relevant pieces of work (more thoroughly discussed in Limitations sections) are IDBD (Sutton, 1992) and SMD [2]. There has been follow-up work to SMD as well that the authors should include in discussion. I'm happy to recommend this paper for acceptance once a discussion on prior work and an exponentiated version of step-size has been added as a baseline. 


Sutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (pp. 171-176).
[2] Local Gain Adaptation in Stochastic Gradient Descent∗ Nicol N. Schraudolph Limitations: 
1. The paper misses prior work on meta-learning for step-sizes. Two seminar works that should be discussion are IDBD (Sutton, 1992) and SMD [2].
1. Additionally, Instead of using ReLU to make step-sizes positive in Meta-Sgd, the paper should use an exponential form for the step-size as used by IDBD (Sutton, 1992)  and SMD[2]. This assures that the step-sizes are always positive, and differentiable. I would expect the exponential form to fix most of the issues related to dead ReLU masks and result in a more robust system. 


Sutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (pp. 171-176).
[2] Local Gain Adaptation in Stochastic Gradient Descent∗ Nicol N. Schraudolph

 Dear reviewers,

Thank you once again for your useful comments. The end of the rolling discussion period is coming close, and we see that most of you have not yet updated your reviews in response to our new results and proposed changes addressing your concerns. In case something is unclear, please do not hesitate in contacting us before the interactive discussion period ends.

The authors introduce a new version of the MAML algorithm which has learnable sparsity on the weights. Using a linear, first order approximation to the masking gradients, they present a tractable algorithm for meta-learning the sparsification of the gradients along with the rest of the parameters. They show small improvements on relevant fewshot learning and continual learning tasks.  I appreciated the exposition which described the sparse MAML algorithm; it gave a clear path to converting understanding of MAML to an understanding of sparse MAML, complete with an interpretation of the update steps. In fact, this interpretation helped me understand why the gradient approximation may be reasonable, and under what conditions it might fail.

In the fewshot and continual learning settings, the sparse MAML algorithm seems to generally perform better than vanilla MAML, and is competitive with other SOTA aproaches. In addition, the analysis of the learning dynamics of sparsification as well as the layer-wise analysis of the sparsification is interesting.

One concern is the fact that the final sparsification factors are not large (2x - 4x gradient reduction) - which to me suggests that the sparse MAML may be a useful final step after all other optimizations have been done, but will never provide qualitatively different results. Researchers may be better served by tuning architecture hyperparameters over tuning sparsity. A related concern is how the sparsifying method fares when pushed into a larger input dimension/dataset/model size regime. This concern was noted by the authors in their conclusion, but may limit the overall impact of the work.

One interesting thing to know would be the rate of reversion of the masks - that is, how often is a gradient turned on after it has been turned off? If reversion is rare, then it may have implications for the efficiency of the algorithm, as well as the general learning dynamics it induces.

Update: The authors addressed many of my concerns. They showed that the sparsification has to be learned simultaneously with the overall training. They also provided an interesting analysis of the rate of reversion of masks (most don't revert, but a non-trivial fraction do). Given these strong additions, I am changing my review from a 5 to a 6. Adequately addressed.

 Thanks for the followups!

It was good to get clarity that the sparsification needs to happen during learning. I think that result will be of general importance for any similar approaches in the future.

The results on ResNet-12 are encouraging as well. Thank you also for carrying out the experiment on the mask reversion; it is interesting that though most masks are "right the first time", mask reversion is non-trivial and important to the learning process.

I will review these responses, as well as the responses to the other authors and update my review score accordingly.

 Following your question on scalability to more complex architectures we pursued additional few-shot learning experiments on a ResNet-12 model, which have now concluded. We would like to draw your attention to our joint reply to all reviewers containing these new results.

 Following the reviewers' questions on the scalability of our methods, we ran additional miniImageNet 5-shot learning experiments on a ResNet-12 neural network, which have now concluded.  We summarize our findings in the table below. Our experiments confirm that gradient sparsity emerges on this more complex model accompanied by an increase in test accuracy. We compared vanilla MAML [8] to sparse-MAML, sparse-ReLU-MAML and ANIL [31]; all results were produced using the first-order gradient approximation. We found that this approximation combined with a 25-step-long inner-loop resulted in improved performance over using second-order derivatives and 5-step-long inner-loops (~68% 2nd-order vs. ~72% 1st-order MAML final test accuracy), while being significantly faster.

We found that sparse-ReLU-MAML, which combines gradient sparsity with learning rate modulation, results in the strongest increase in performance over vanilla MAML for this more complex architecture. Our results are competitive with the state-of-the-art for this model (see, e.g., Table 1 from ref. [Baik21]). As for the pattern of gradient sparsity arising in sparse-MAML and sparse-ReLU-MAML, it is qualitatively similar to the one obtained for the smaller convolutional neural network model. Both methods discover solutions with a very high level of parameter freezing in the earlier layers (~90%), which progressively decreases with depth, resulting in an overall level of sparsity of ~30% (note that the final layers of this model have a larger number of parameters).

| Algorithm | Final acc. (%)  | Sparsity (%)
| :------------- |:-------------:|:-------------:|
| MAML   | 72.08 +/- 0.55   | -
| ANIL   | 72.52 +/- 0.38   | - 
| sparse-MAML    | 73.11 +/- 0.45 | 30.37 +/- 1.66
| sparse-ReLU-MAML | 76.03 +/- 0.15 | 32.24 +/- 1.45

We will include 1-shot learning results in the final version of the paper.

[Baik21] Baik S, Oh J, Hong S, Lee KM (2021). Learning to forget for meta-learning via task-and-layer-wise attenuation, IEEE PAMI 2021, in press. URL: https://ieeexplore.ieee.org/document/9507366

 Thank you for the constructive criticism and interesting comments. We reply point-by-point to your questions and comments below:

---
> One concern is the fact that the final sparsification factors are not large (2x - 4x gradient reduction) - which to me suggests that the sparse MAML may be a useful final step after all other optimizations have been done

Following your concern, we ran an additional control where gradient masks are meta-learned only in a final phase, after vanilla MAML is concluded. First, we ran MAML for 300 epochs. Then, we froze the meta-learned initialization and only trained the gradient mask of sparse MAML for an additional 100 epochs, for two different sparsity initializations.

The results can be found in the table below and have been added to a new version of the paper. We find that meta-learning the gradient sparsity mask in this way (separately from the MAML weight initialization) does not result in improved performance. This new analysis indicates that joint optimization of both weights and gradient masks is needed to obtain the improved performance of sparse-MAML. In other words, sparse-MAML relies on discovering models for which only a fraction of the weights must be updated.

| Setup | Final acc. (%) | Final sparsity (%) | 
| :------------- |:-------------:|:-------------:|
| 1-shot - 0% initial sparsity    | 46.42 +/- 0.58  | 9 | 
| 1-shot - 50% initial sparsity    | 46.79 +/- 0.27  | 45 | 
| 5-shot - 0% initial sparsity    | 64.68 +/- 0.16  | 29 | 
| 5-shot - 50% initial sparsity    | 64.01 +/- 0.47  | 51 | 

---
> A related concern is how the sparsifying method fares when pushed into a larger input dimension/dataset/model size regime.

Following your (and reviewer bJTT’s) comment we ran additional experiments evaluating sparse-La-MAML using a convolutional neural network (the same used in the La-MAML paper [10]) on a Split-CIFAR-10 class-incremental learning problem (5 tasks of 2 classes each, each data point seen online once). We chose to study this challenging learning setting as the performance of current CL methods is far behind that of offline learning, even for simple convolutional neural networks. In particular, experience replay (ER) remains a strong baseline in class-incremental learning (cf. [Aljundi19]). We produced this baseline for our architecture, and compared sparse-La-MAML to it, performing for both methods a hyperparameter scan over replay batch size, the number of gradient updates per incoming batch, and learning rates. For both small (20 examples per class) and large (100 examples per class) replay buffer sizes, our method consistently outperforms ER, cf. table below. We observe a qualitatively distinct gradient sparsity pattern emerge in this setting, with a large fraction of frozen weights in the final fully-connected layer, which increase as more classes are learned, and significantly lower values of gradient sparsity for the remaining parameters. These results are now part of the main text. For the final version of the paper we will complete our results with GEM, MER, and standard La-MAML.

We note that scaling up MAML to more complex network architectures is still an open question in gradient-based meta-learning research that we consider beyond the scope of the current work. Sparse-MAML stands to benefit from forthcoming advances in this direction.

[Aljundi19] Aljundi, R, et al. (2019). Online continual learning with maximal interfered retrieval. NeurIPS.

---
> One interesting thing to know would be the rate of reversion of the masks - that is, how often is a gradient turned on after it has been turned off? If reversion is rare, then it may have implications for the efficiency of the algorithm, as well as the general learning dynamics it induces.

This additional analysis is a great suggestion. We ran the following mask reversion study on miniImageNet: We measured the amount of masks that change at least once and compared it to the amount of masks that reverse more than once in a time window of 100 update steps over the course of training. We find that a high fraction of gradient masks averaged over the network crosses the activation threshold once or more than once, in particular in early phases of training, cf. table below. This result has been included in a new version of the paper as a plot, showing the time evolution of the mask reversion rate. 

| Setup | % mask change >= 1 (first epoch/last epoch)  | % mask change > 1 (first epoch/last epoch) |
| :------------- |:-------------:|:-------------:|
| 1-shot     | 40%	/ <1%     | 	13%	/ <1%|
| 5-shot      | 27% / <1%  | 10% / <1%|

We would like to note that this is a strength of our straight-through mask estimation method. By contrast, gradient modulation methods which use rectified learning rates naively can never recover once the underlying parameter becomes negative, and soft rectifiers or logistic approximations to the mask may suffer from vanishing meta-gradient issues. 

---
We believe that our new results have improved our paper. Given that you agree, we would be very grateful for a reevaluation of our work. We remain open to any additional questions you may have during the rolling discussion period.


 Thank you for the encouraging review and for the useful feedback that we incorporated in a revised version of our paper. We reply to your suggestions point-by-point below.

---
> In Figure 1, performance is also robust across the different sparsity initializations?

In short, yes. In Figure 1, the left plot shows that sparse-MAML robustly improves over all sparsity initialization levels, when compared to standard MAML. We have now clarified the figure caption and reinforced this fact in the main text.

---
> Recent Continual Learning literature deploy CL model on modern deep neural networks, such as ResNet 18 or MobileNet v2. Even La-MAML validates their method on simple CNN, I strongly recommend validating the proposed method on modern nets.


Following your (and reviewer u5xJ’s) comment we ran additional experiments evaluating sparse-La-MAML using a convolutional neural network (the same used in the La-MAML paper [10]) on a Split-CIFAR-10 class-incremental learning problem (5 tasks of 2 classes each, each data point seen online once). We chose to study this challenging learning setting as the performance of current CL methods is far behind that of offline learning, even for simple convolutional neural networks. In particular, experience replay (ER) remains a strong baseline in class-incremental learning (cf. [Aljundi19]). We produced this baseline for our architecture, and compared sparse-La-MAML to it, performing for both methods a hyperparameter scan over replay batch size, the number of gradient updates per incoming batch, and learning rates. For both small (20 examples per class) and large (100 examples per class) replay buffer sizes, our method consistently outperforms ER, cf. table below. We observe a qualitatively distinct gradient sparsity pattern emerge in this setting, with a large fraction of frozen weights in the final fully-connected layer, which increase as more classes are learned, and significantly lower values of gradient sparsity for the remaining parameters. These results are now part of the main text. For the final version of the paper we will complete our results with GEM, MER, and standard La-MAML.

| Setup (replay buffer size) | Final acc. (%)  | 
| :------------- |:-------------:|
| Experience replay (20)   | 19.75 +/- 1.23   |
| sparse-La-MAML (20)   | 27.85 +/- 0.69  | 
| Experience replay (100) | 29.12 +/- 2.41   |
| sparse-La-MAML (100)    | 37.70 +/- 0.80 | 


We note that scaling up MAML to more complex network architectures is still an open question in gradient-based meta-learning research that we consider beyond the scope of the current work. Sparse-MAML stands to benefit from forthcoming advances in this direction.

[Aljundi19] Aljundi, R, et al. (2019). Online continual learning with maximal interfered retrieval. NeurIPS.

---
> In Figure 4, why does the sparsity decrease when layer sizes increase? As the network has additional neurons, it looks natural that there are more redundant/unnecessary updates (sparsity can be increasing).


Our mask meta-learning update (eq. 5) tends to disable learning for parameters whose outer- and inner-loop gradients have opposing signs. One explanation for the decreasing fraction of sparse gradients in larger models is that these architectures lead to reduced gradient conflicts. We note that the actual number of sparsified gradients is in fact still increasing with (although more slowly than) the number of parameters, as shown in the table below. We added this new analysis to the SM.

| Hidden size | 32 | 64| 128 | 256 |
| :------------- |:-------------:|:-------------:|:-------------:|:-------------:|
| Sparsity    | 52%  | 50% | 44% | 38% | 
| # trainable params    | 33k | 121k| 463k | 1.8Mio |
| # frozen params   | 17k | 61k | 203k |	688k |
| factor increase of frozen params    | -  | 3.5| 3.4 | 3.4 |

---
> Please describe the merits/drawbacks of meta-learning-based continual learning, compared to other CL directions (e.g., regularization-based, rehearsal-based, architecture-based).

The surge of meta-learning in continual learning can be explained by its ability to automatically discover the inductive biases that are appropriate for learning without forgetting. Previous works hypothesize that a particular inductive bias will mitigate catastrophic forgetting, e.g. keeping parameters from diverging too much from previous versions [Kirkpatrick17], and then develop a solution around that. Contrarily, meta-learning based approaches will learn inductive biases that are conducive for learning without interference in a data-driven way. For example, in [Javed19] sparsity emerges in the learned representations, a characteristic that has long been hypothesized as desirable in continual learning [French99].

Regularization-based methods are notoriously incapable of working in more realistic settings, such as those considered in our work, mostly because they are not equipped with a mechanism to perform cross-task discrimination or to recalibrate themselves on past tasks after some interference has occurred.  The same applies for dynamic architectures, which rely on task labels. This reliance can be bypassed with a task-inference module, which may however suffer from some forgetting itself.

Rehearsal-based methods do not suffer from the aforementioned weaknesses. Nevertheless, they scale poorly due to their reliance on always approximating an i.i.d. distribution at every update. The total runtime of these methods scales quadratically with the number of tasks.

The ambitious goal of meta-learning inductive biases that benefit continual learning directly from data may come at the cost of decreasing sample efficiency and increasing compute requirements. However, the latter is potentially offset by reducing the number of hyperparameter-search trials [Maclaurin15].

[Kirkpatrick17] Kirkpatrick, James, et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS.

[Javed19] Javed, K., & White, M. (2019). Meta-learning representations for continual learning. NeurIPS.

[French99] French, RM (1999). Catastrophic forgetting in connectionist networks. TCS.

[Maclaurin15] Maclaurin, D., Duvenaud, D., & Adams, R. (2015). Gradient-based hyperparameter optimization through reversible learning. ICML.

---
We would be very thankful for a reappreciation of our work given the experiments, clarifications and improvements to our paper detailed above. We remain available to answer any further questions that you may have.


 Thank you for your comments and for taking the time to review our work. Below we reply to your main concern by highlighting why we believe that the performance of sparse-MAML that we report (on standard benchmarks) should be considered high.

---
> My main concern is the seemingly weak performance of the Sparse-MAML in terms of its resulting model’s qualities. For instance, in Table 1, Sparse-MAML loses to MT-Net and Meta-SGD

We would like to stress that MT-Net and Meta-Curvature (MC) are computationally intensive state-of-the-art meta-learning methods, which introduce a large number of meta-parameters and use second-order derivatives in their meta-parameter updates. 

By contrast, our sparse-MAML(+) uses inexpensive first-order derivatives (readily computed by backpropagation), and only extends MAML with a binary gradient mask meta-parameter. The main result we wanted to convey in Section 3.3 is that despite its simplicity, sparse-MAML performs comparably to MT-Net and MC.

Meta-SGD and sparse-ReLU-MAML only add one meta-parameter (a learnable learning rate) per parameter, like our method. These methods are in fact always matched or outperformed by sparse-MAML(+), e.g., see 5-shot setting in Table 1, even though gradient sparsity is strictly less expressive than learning rate adjustment. Moreover, a significant fraction of learnable learning rates become near-zero (see Table S1). This suggests that the performance increase from methods like Meta-SGD stems mostly from gradient sparsity.

These findings show that learning by sparse gradient descent can improve over standard gradient descent, and explain or reproduce the performance of more sophisticated gradient modulation methods. We have now strengthened these points in the discussion of the results of Section 3.3.

Finally, we would like to point out that our findings contribute to an ongoing debate (see e.g. the ANIL paper of Raghu et al. [31]) on whether meta-learning methods such as MAML tend to discover frozen features that transfer well to new tasks, or initial weight configurations that still require some learning when a new task must be solved. Our method automatically discovers solutions with many frozen features that are shared over a family of tasks. Importantly, its performance improvements over both MAML (no freezing) and ANIL (entire body is frozen), suggest that some plasticity is still beneficial, if it occurs on the right subset of parameters. Our work shows that an appropriate subset can be discovered by simple first-order meta-learning methods.

---
> Sparse-MAML+ seems to be less sparse than Sparse-MAML

We are sorry for this error in the SM, this value was incorrectly inserted. The correct sparsity level for sparse-MAML+ is 78.05 +- 1.67%, in line with sparse-MAML. Many thanks for spotting this mistake.

---
We remain open to any additional suggestions or questions you may have that would make you reevaluate our work positively.


 Thank you for the insightful review and for pointing out an important family of methods that we were not aware of. We have expanded our discussion accordingly and ran the suggested experiment. We reply point-by-point below with a summary of findings and some discussion:

---
> It's not clear to me why a binary mask learned using a biased method (Straight-through estimator) would be better than simple meta-learning step-sizes. 

Meta-learning binary masks instead of step-sizes reinforces the inductive bias towards freezing weights. Our intuition is that determining which weights to update and which to freeze is the essential gradient modulation operation in few-shot and continual learning problems. Accordingly, we believe that the improvements we found over meta-learned step sizes stem from the reduced meta-learning search space. 

This can be perhaps most clearly seen when comparing sparse-MAML to our own sparse-ReLU-MAML control, where rectified learning rates are learned with first-order straight-through estimation, just as in sparse-MAML. In this model, the resulting learning rates tend to be sparse (cf. Table S1, inline with our intuition that freezing weights is key), and the performance lies between MAML and sparse-MAML. In any case, our results also motivate the use of meta-learned-step-size methods, as these may discover sparse forms of gradient descent too. We added a few sentences to the discussion section presenting this perspective.

We agree that the straight-through estimator may behave less well in other sets of experiments, as discussed in our limitations section. 

---
> The paper misses prior work on meta-learning for step-sizes. Two seminar works that should be discussion are IDBD (Sutton, 1992) and SMD [2].

The cited work is indeed seminal and highly relevant for our paper, thank you for the missing references. We have now included the following paragraph in the discussion:

The idea of meta-learning learning rates can be traced back to the seminal work of Sutton [Sutton92], who proposed to estimate learning rate meta-gradients online using forward-mode automatic differentiation, and to use consecutive batches of data to define inner- and outer-loop loss functions. This approach, known as stochastic meta-descent (SMD), was extended to nonlinear models by Schraudolph [Schraudolph99] using fast Hessian-vector product techniques. Using SMD to optimize neural network models is an ongoing area of research [41, Vivek17, Jacobsen19, Kearny19]. It is an interesting question whether gradient sparsity emerges when applying SMD to online learning problems that are not clearly structured in tasks, as considered here. Furthermore, this line of work suggests that it might be possible to obtain finer binary gradient mask updates in an online fashion using forward-mode automatic differentiation.

[Sutton92] Sutton, RS (1992) Adapting bias by gradient descent: An incremental version of delta-bar-delta. AAAI.

[Schraudolph99] Schraudolph, N (1999). Local gain adaptation in stochastic gradient descent. ICANN.

[Vivek17] Veeriah, V, Zhang, S and Sutton, RS (2017). Crossprop: Learning representations by stochastic meta-gradient descent in neural networks. ECML PKDD.

[Jacobsen19] Jacobsen, A, et al. (2019). Meta-descent for online, continual prediction. AAAI.

[Kearny19] Kearney, A, et al. (2019). Learning feature relevance through step size adaptation in temporal-difference learning. arXiv preprint arXiv:1903.03252

---
> Additionally, Instead of using ReLU to make step-sizes positive in Meta-Sgd, the paper should use an exponential form for the step-size as used by IDBD (Sutton, 1992) and SMD[2]. This assures that the step-sizes are always positive, and differentiable.

Thank you for suggesting this interesting additional experiment. We were able to run an analysis of the exponential learning rate model on miniImageNet, including the same hyperparameter scan conducted for sparse-MAML.
In the 1-shot learning setting, this model could not improve upon vanilla MAML (cf. table below), and in the 5-shot learning setting its performance lies in between MAML and sparse-MAML. This result is in line with our hypothesis that parameter freezing is the essential gradient modulation operation in few-shot learning, as a zero learning rate can be easily represented and learned with gradient methods in sparse-MAML and sparse-ReLU-MAML, but not in the exponential parameterization.

| Setup | Final acc. (%) |
| ------------- |:-------------:|
| 1-shot     | 48.38 +/- 0.25     | 
| 5-shot      | 65.21 +/- 0.62     | 

---
Given our new experimental results and expanded discussion we would be very grateful for a reassessment of our work. We are happy to answer any further questions you may have during the discussion period.


 We thank all reviewers for their constructive criticism and useful suggestions that have helped us improve our paper. We provide below a summary of the major changes made in response to the reviews:
- We included a new discussion paragraph relating our method to a line of research on learning rate meta-learning stemming from the work of Sutton (1992) and Schraudolph (1999).

- We performed few-shot learning experiments using an exponential parameterization of learning rates. Our experiments reveal that this model is not as effective in improving over MAML as binary gradient masking or rectified linear learning rates. These results support our hypothesis that gradient sparsity is key to improve learning performance, as it is easier to obtain zero learning rates in the latter models.

- We performed new class-incremental continual learning experiments on the Split-CIFAR-10 dataset. Our new results show that sparse-La-MAML consistently outperforms experience replay, a strong baseline in this challenging continual learning setting. To deal with class-incremental learning, our method discovers a solution with high levels of parameter freezing in the final output layer.

- We added a new discussion paragraph comparing meta-learning-based continual learning to other continual learning approaches.
We ran a new control where gradient masks are meta-learned separately, after a first vanilla MAML phase. We could not improve MAML in this way. This shows that our method relies on finding initializations that are appropriate for sparse gradient descent.

- We analyzed the rate of mask reversion and found that in early phases of meta-learning a significant number of masks repeatedly activate and deactivate, a feature that is supported by our straight-through mask update.

Given these new results, analyses and changes to the text we kindly ask the reviewers to reevaluate their scores, if they agree that our paper has improved. We remain available to respond to any further questions the reviewers may have in the upcoming discussion period.


The paper proposes a sparse meta gradient update method during MAML by adopting binary masks that modulate the gradient during the inner update. The masks depend on the angle-similarity between the averaged inner loss gradients and outer loss gradients, where two gradients are opposite directions, then mask to be zero, otherwise one. The method shows better performance compared to MAML and existing meta-learning baselines using the modulated learning rate strategies. Furthermore, the method can be combined with meta continual learning methods, such as La-MAML, and shows superior performances.  Overall, the paper is well written and easy to follow. 
The methodology is simple and reasonable, several ablation studies and analyses are helpful for a better understanding of the proposed method. Although the performance gain compared to recent baselines are not much impressive and sometimes looks marginal, I think this is good work. 

Questions and comments are as follows:

- In Figure 1, performance is also robust across the different sparsity initializations?

- Recent Continual Learning literature deploy CL model on modern deep neural networks, such as ResNet 18 or MobileNet v2. Even La-MAML validates their method on simple CNN, I strongly recommend validating the proposed method on modern nets.

- In Figure 4, why does the sparsity decrease when layer sizes increase? As the network has additional neurons, it looks natural that there are more redundant/unnecessary updates (sparsity can be $\uparrow$).

- Please describe the merits/drawbacks of meta-learning-based continual learning, compared to other CL directions (e.g., regularization-based, rehearsal-based, architecture-based). The authors addressed.

This paper proposes a strategy called Sparse-MAML, which extends the MAML algorithm to learn sparse updates.

The Sparse-MAML algorithm is applied in two settings: few-shot learning with MiniImageNet, and continual learning with MNIST rotations and permutations. The results show a mixed performance – some gains, some losses, compared to different baselines. The authors also provide many ablation studies and analysis which show that Sparse-MAML does exhibit the sparsity in the gradient updates of MAML.
  Disclaimer: I am not familiar with the benchmarks presented in the paper, namely miniImageNet and MNIST rotations/permutations. Therefore, I cannot appreciate the experimental results in the paper. I can only provide my assessment of the paper based on comparing the numbers provided therein, without much further insight.

On the methodology side, sparsity is a preferred property in many deep learning settings, so the motivation of learning sparse updates in this paper is reasonable. The formulation of Sparse-MAML, which uses a ReLU vector to shut off the gradient updates to various parameters is simple and not computationally intensive. On top of this, the paper is clearly written and easy to follow.

On the experimental side, the authors do provide analysis which confirms that Sparse-MAML indeed leads to sparser updates to the model parameters. My main concern is the seemingly weak performance of the Sparse-MAML in terms of its resulting model’s qualities. For instance, in Table 1, Sparse-MAML loses to MT-Net and Meta-SGD, and in Table 2, Sparse-MAML sometimes loses to ANIL and BOIL. While the performance of Sparse-MAML+ is sometimes better, Sparse-MAML+ seems to be less sparse than Sparse-MAML (as the authors explain that “gradient masks are generated from a low-dimensional Gaussian vector”). This raises the question whether sparsity should be enforced to the extent of Sparse-MAML.
 I cannot comment since I have almost no insight about the experiments in the paper.


Meta-review: This paper proposes an extension to MAML that also meta-learns weight update masks, and analyses the resulting learning outcomes under a variety of settings. Both conventional few-shot learning and continual learning benchmarks are considered. Although some work has been done in this direction such as MT-Net, reviewers are generally positive  after rebuttal phase and find this to be worthwhile contribution. Overall I recommend acceptance.

Decision: accept