Learning Continually by Spectral Regularization

Authors: Alex Lewandowski, Michał Bortkiewicz, Saurabh Kumar, András György, Dale Schuurmans, Mateusz Ostaszewski, Marlos C. Machado
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: plasticity, neural networks, spectral regularization, continual learning

Abstract:
Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek to mitigate this effect by sustaining good performance while maintaining network trainability. We develop a new technique for improving continual learning inspired by the observation that the singular values of the neural network parameters at initialization are an important factor for trainability during early phases of learning. From this perspective, we derive a new spectral regularizer for continual learning that better sustains these beneficial initialization properties throughout training. In particular, the regularizer keeps the maximum singular value of each layer close to one. Spectral regularization directly ensures that gradient diversity is maintained throughout training, which promotes continual trainability, while minimally interfering with performance in a single task. We present an experimental analysis that shows how the proposed spectral regularizer can sustain trainability and performance across a range of model architectures in continual supervised and reinforcement learning settings. Spectral regularization is less sensitive to hyperparameters while demonstrating better training in individual tasks, sustaining trainability as new tasks arrive, and achieving better generalization performance..

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730419145518

thread_type:
official_review

<thread_object>
creator: reviewer_f6f3

<content>
summary:
The paper introduces a new regularization to encourage neural networks to combat loss of plasticity phenomenon in a continual learning setting. The new approach is based on the observation that the maximum singular value of neural network layers grow over the course of training and avoiding this growth can help improve trainability of models in a continual learning setting. The proposed regularizer attempts to steer the maximum singular value of a layer to 1 (and 0 for the bias terms). The paper uses accepted datasets and tasks in continual learning literature to study the effectiveness of the proposed method over several baselines.

strengths:
- The paper solves the problem of loss of plasticity in continual learning by addressing trainability. 

- Section 3 and Section 4 in the paper are nicely written and is valuable to a general audience. Also, the paper provides at least two illustrative examples as well as some analysis to make the observation that the top singular value growth may cause training issues.

- The development of proposed solution via spectral regularization  is intuitive

weaknesses:
* The proposed method to keep top singular value small bears some similarities to the following paper:
  - https://arxiv.org/abs/2303.06296

* The experimental setup used in the paper is incomplete and not very convincing in order to judge the empirical results:
  - The models used are on the smaller side (ResNet-18 and ViT)
  - ViT model size is not specified in the paper 
  - The dataset used in Tiny-Imagenet which is not fully described. I believe the inputs are 64x64 based on my search but this detail is missing in the paper
While I am not an expert in continual learning area, there are recent papers that suggest that larger datasets (resolution and size) are available and used in continual learning studies

* The presentation of the experimental results and discussion section could be improved:
  - Figure 1 has legend only on the top left figure which makes it hard for the reader to follow how each method works across different continual learning scenarios
  - Figure 2 has a similar issue with legend
  - he paper does not define all the quantities plotted in Figure 3.
  - Since the paper focuses on trainability, additional graphs that show how trainability improves in CL setting might be useful in the presentation. Figure 3 provides a hint but in this case the shrink and perturb baseline shows better trainability as defined by singular value.

I'm unable to support acceptance at this point due questions on setup used in empirical analysis. The paper could benefit from an update that includes missing information that can give the reader an idea on exact metrics tracked, setup used, dataset details and other information.

questions:
In addition to the points noted in weakness section, I wonder if the following statement is too strong
```
Note that these datasets and architectures encompass all continual supervised learning experimental settings considered in the loss of plasticity literature.
```
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686006477

thread_type:
official_review

<thread_object>
creator: reviewer_ubbh

<content>
summary:
The paper addresses plasticity loss in continual learning by proposing a spectral regularization technique. The proposed regularization technique regularizes the network’s weights at each layer towards a spectral norm of 1. This method is motivated by a spectral analysis of (continual) learning and the observation that standard neural network weight initializations exhibit good trainability and generalization. The paper’s proposed method is validated on a series of continual learning benchmark problems where it is shown to be performant and robust to the choice of regularization strength.

strengths:
- Section 1 Introduction and Section 2 Problem Setting are well written with clarity and detail, effectively introducing the problem, the spectral regularization algorithm, and some intuition behind its formulation.
- The spectral analysis of continual learning well motivates the algorithm. 
- The experiments consider a wide swathe of existing continual learning algorithms, both ResNet-18 and Vision Transformer architectures, and a variety of datasets and non-stationarities. Over this set of experiments, spectral regularization attains competitive performance and is robust to its choice of hyperparameter.

weaknesses:
- Although spectral regularization is well motivated and the paper includes positive empirical results, one way to improve the paper would be to derive some theory illustrating the benefit of spectral regularization, and drawbacks of other regularization schemes, in continual learning.

questions:
- Is there any particular reason why Continual Backprop was not evaluated in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730830338876

thread_type:
official_review

<thread_object>
creator: reviewer_yumr

<content>
summary:
The paper considers the continual learning problem. In order to keep the stability of problem, the spectral regularizer is added to the objective function. Experimental results demonstrate the effective of the approach method.

strengths:
This paper provide the spectral regularize to the objective function for improving the stability of the problem.

weaknesses:
However, the added spectral regularize would bring in the cost of computation for the continual learning if facing large model and large data.

questions:
1 through the initialization is the key to the trainability, singular values of Jacobian and the condition numbers would induce the new selected parameters, e.g., rank
2 how to effectively deal with computation of the spectral norm, i.e. spectral regularization,
3 the performance of the spectral is not always better than that of vision transformer and resents
4 what’s more, the datasets are not enough, please refer to the latest approach,
5 for the figure 5, what’s the phenomenon of suddenly decrease for the y-axis.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730910564395

thread_type:
official_review

<thread_object>
creator: reviewer_vem9

<content>
summary:
The authors derive a new spectral regularizer for continual learning inspired by the observation that the singular values of the neural network parameters at initialization are an important factor for trainability during early phases of learning. Under such regularization they keep the maximum singular value of each layer close to one. 

Under such spectral regularization they claim that they can directly ensure gradient diversity throughout training, which promotes continual trainability, while minimally interfering with performance in a single task. 

To validate the utility of the proposed regularization and the claims they present empirical evaluation. The experimental results shows how the proposed spectral regularizer can sustain trainability and performance across a range of model architectures in continual supervised and reinforcement learning settings. They try to demonstrate better training in individual tasks, trainability as new tasks arrive, and better generalization performance.

strengths:
- The continual learning topic is important area and loss of plasticity is fundamental challenge.
 - The approach is interesting and the idea to regularize the maximum singular value of each layer close to one seems novel.
 - The presentation of the work is nice, while the motivation is appropriate.
 - The authors present results for several experiments, evaluating different properties.

weaknesses:
- It appears like a marginal improvement and novelty with respect to L2 regularization.
- It seems that there is a connection to L2 regularization, or L2 regularization towards initialization of parameters which is a sort of spectral regularization.
- Deeper understanding/analysis of the proposed and the one above seems important and might be beneficial, but I was not able to find it in the paper L2 norm vs spectral norm in this context.  
- In the experimental section, looking at the graph and the results it seems that the improvements are marginal, or not highlighted significantly.
- In the same experimental section, I was able to see only graphs and plots, was not able to see actual table or values, to be able to apricate the achieved results more.

questions:
How different the propose approach is from L2 regularization on the same set of parameters?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes spectral regularization scheme for continual learning tasks. Instead of simple L2 regularization, the proposed scheme aims to regularize the largest singular value of the weight matrices close to 1. While spectral regularization has been widely used for improving generalization capability of neural networks, directly applying it to continual learning seems to be new. A downside of it is its additional requirement of computation complexity to compute the regularization term. Empirical evaluation shows the proposed method is effective compared to other baselines for task-agnostic continual learning settings. Although the improvement is not a groundbreaking of a kind, the idea seems to be worth publishing in ICLR. So the decision is Accept. 

Additionally, https://www.sciencedirect.com/science/article/pii/S0167865524001909 seems to be a relevant work, so it would be good to include it in the final version and address the difference between their method.