Plastic Learning with Deep Fourier Features
Authors: Alex Lewandowski
Dale Schuurmans
Marlos C. Machado
Venue: ICLR 2025 Poster (2025)
Keywords: Fourier
plasticity
neural networks
continual learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Deep neural networks can struggle to learn continually in the face of non-stationarity, a
  phenomenon known as loss of plasticity.
  In this paper, we identify underlying principles that lead to plastic algorithms.
  We provide theoretical results showing that linear function approximation, as well as a special case of deep linear networks, do not suffer from loss of plasticity.
  We then propose deep Fourier features, which are the concatenation of a sine and cosine in every layer, and we show that this combination provides a dynamic balance between the trainability obtained through linearity and the effectiveness obtained through the nonlinearity of neural networks.
  Deep networks composed entirely of deep Fourier features are highly trainable and sustain their trainability over the course of learning.
  Our empirical results show that continual learning performance can be improved by replacing ReLU activations with deep Fourier features combined with regularization.
  These results hold for different continual learning scenarios (e.g., label noise, class incremental learning, pixel permutations)
  on all major supervised learning datasets used for continual learning research, such as CIFAR10, CIFAR100, and tiny-ImageNet.

Published as a conference paper at ICLR 2025
PLASTIC LEARNING WITH DEEP FOURIER FEATURES
Alex Lewandowski 222 Dale Schuurmans' 1,2,3,4
Marlos C. Machado
Deparment of Computing Science, University of Alberta, AAiii, & Google DeepMind, *Canada CIFAR Al Chair
ABSTRACT
Deep neural networks can struggle to learn continually in the face of non- stationarity, a phenomenon known as loss of plasticity. In this paper, we identify underlying principles that lead to plastic algorithms. We provide theoretical results showing that linear function approximation, as well as a special case of deep linear networks, do not suffer from loss of plasticity. We then propose deep Fourier fea- tures, which are the concatenation of a sine and cosine in every layer, and we show that this combination provides a dynamic balance between the trainability obtained through linearity and the effectiveness obtained through the nonlinearity of neural networks. Deep networks composed entirely of deep Fourier features are highly trainable and sustain their trainability over the course of learning. Our empirical results show that continual learning performance can be improved by replacing ReLU activations with deep Fourier features combined with regularization. These results hold for different continual learning scenarios (e.g., label noise, class incre- mental learning, pixel permutations) on all major supervised learning datasets used for continual learning research, such as CIFARIO, COFARI00, and tiny- -mageNe.
INTRODUCTION
Continual learning is a problem setting that moves past some of the rigid assumptions found in supervised, semi-supervised, and unsupervised learning (King) 1994| Thrun 1998). In particular, the continual learning setting involves learning from data sampled from a changing, non-stationary distribution rather than from a fixed distribution. A performant continual learning algorithm faces a trade-off due to its limited capacity: it should avoid forgetting what was previously learned while also being able adapt to new incoming data, an ability known as plasticity (Parisi al.] (211)). Current approaches that use neural networks for continual learning are not yet capable of making this trade-off due to catastrophic forgetting (Kirkpatrick et al., 2017) and loss of plasticity (Dohare)
et al.] 2021 Lyle et al.) 2023 Dohare et al.) 2024). The training of neural networks is in fact an active research area in the theory literature for supervised learning (Jacot et al. 2018 Yang et al. 2023
Kunin al. 2024), which suggests there is much left to be understood in training neural networks continually. Compared to the relatively well-understood problem setting of supervised learning, even the formalization of the continual learning problem is active research area (Kumar et al. [023a]
Abel et al. 2024] Liu et al. 2023). With these uncertainties surrounding current practice, we take step back to better understand the inductive biases used to build algorithms for continual learning.
One fundamental capability expected from a continual learning algorithm is its sustained ability update its predictions on new data. Recent work has identified the phenomenon of loss of plasticity in neural networks in which stochastic gradient-based training becomes less effective when faced with data from a changing, non-stationary distribution (Dohare et al., 2024). Several methods have been proposed to address the loss of plasticity in neural networks, with their success demonstrated empirically across both supervised and reinforcement learning (Ash and Adams) 2020 Lyle et al.,
2022 2023 Lee et al. 2024). Empirically, works have identified that the plasticity of neural networks is sensitive to different components of the training process, such as the activation function (Abbas ct al. 2023). However, little is known about what is required for learning with sustained plasticity.
The goal of this paper is to identify a basic continual learning algorithm that does not lose plasticity in both theory and practice, rather than mitigating the loss of plasticity in existing neural network architectures. Our focus is on loss of plasticity rather than catastrophic forgetting, because plasticity is needed to sustain continual learning from new data. In particular, we investigate the effect of the nonlinearity of neural networks on the loss of plasticity. While loss of plasticity is a well-documented phenomenon in neural networks, previous empirical observations suggest that linear function

---

Published as a conference paper at ICLR 2025
3
11
sin
sin
apprx)
apprx)
22
-1
-1
-3
-3
Figure 1: A neural network with deep Fourier features in every layer approximately embeds deep linear network. A single layer using deep Fourier features linearly combines the inputs, a, to compute the pre-activations, and each pre-activation is mapped to both a cos unit and sin unit (Leff). For each pre-activation, either the sin unit (Middle) or the cos unit (Right) is well-approximated by a linear function.
approximation capable of learning continually without suffering from loss of plasticity (Dohare et al.] [202T] 2024). In this paper, we prove that linear function approximation does not suffer from loss of plasticity and can sustain their learning ability on sequence of tasks. We then extend our analysis to special case of deep linear networks, which provide an interesting intermediate case between deep nonlinear networks and linear function approximation. This is because deep linear networks are linear in representation but nonlinear in gradient dynamics (Saxe et al., 2014). We provide theoretical and empirical evidence that general deep linear networks also do not suffer from loss of plasticity. The plasticity of deep linear networks is surprising compared to the loss of plasticity of deep nonlinear networks. This finding suggests that loss of plasticity is not necessarily caused by the nonlinear learning dynamics, but combination of nonlinear learning dynamics and nonlinear representations.
Given this seemingly natural advantage of linearity for continual learning, as well as its inherent limitation learning only linear representations, emulate the dynamics of deep linear networks to sustain plasticity. We hypothesize that, to effectively learn continually, the neural network must balance between introducing too much linearity and suffering from loss of deep representations and introducing too much nonlinearity and suffering from loss of plasticity. In fact, we show that previous work partially satisfies this hypothesis, such as the concatenated Re (Shang et al., 2016). leaky-Re activations (Xu et al., 2015), and residual connections (He et al., 2016), but they fail striking this balance. Our results build on previous work that identified issues of unit saturation (Abbas et al. 2023) and unit linearization (Lyle et al. 2024) as issues in continually training neural networks with common activation functions. In particular, we generalize both to the problem of low unit sign entropy, which indicates a lack of diversity in the activations as measured by the entropy of the sign of the hidden units. We show that linear networks have high unit sign entropy, meaning that the sign of hidden unit different inputs is positive approximately half the inputs. In contrast, deep nonlinear networks with most activation functions tend to have low unit sign entropy, which indicates saturation linearization.
explore how nonlinear networks can better
Periodic activation functions (Parascandolo et al. 2017). like the sinusoid function (sin), are a notable exception for having high unit sign entropy despite still suffering from loss of plasticity. Thus, in addition to unit sign entropy, we demonstrate that the network's activation function should be well- approximated by a linear function. We propose deep Fourier features as means of approximating linearity dynamically, with every pre-activation being connected to two units, one of which will always be -ppl-ipprox by linear function. In particular, deep Fourier features concatenate sine and cosine activation in each hidden layer. The resulting network is nonlinear while also approximately embedding a deep linear network using all of its parameters. Deep Fourier features differ from previous approaches that use Fourier features only in the input layer (Tancik et al. 2020 Li and Pathak 2021, Yang et al., 2022) or that use fixed Fourier feature basis (Rahimi and Recht 2007| Konidaris et al. 2011). We demonstrate that networks using these shallow Fourier features still exhibit a loss of plasticity. Only by using deep Fourier features in every layer is the network capable of sustaining and improving trainability in a continual learning setting. Using Imag- (Le) and Yang 2015), CIFAR1O, and CIFAR (Krizhevsky) 2009), we show that deep Fourier features can be used as drop-in replacement for improving trainability in commonly used neural network architectures. Furthermore, the trainability of deep Fourier features enables training with much larger regularization strength, leading to superior generalization performance.
2

---

Published as a conference paper at ICLR 2025
2 PROBLEM SETTING
We define a deep network, fo with a a parameter set, 0= (W,, by bi)(=r as a sequence of layers, in which each layer applies linear transformation followed by element-wise activation function, é in each hidden layer. The output of the network, fa) := hz(x), is defined recursively by [(2,,w] = o(=)), and, 21 W by where w is the width of the network, and hy = z. We refer to particular element of the hidden layer's output hy,;
fu = [ [m,=a] = [0(2,1).....
as a unit. The deep network is deep linear network when the activation function is the identity,
<(=) = z. Linear function approximation is equivalent to linear network with L =
The problem setting that we consider is continual supervised learning, where the learner does not have information about the task boundaries. Instead, at each iteration the learner has access to minibatch of observation-target pairs of size M, {Ii, y4 )i This minibatch is used to update the parameters 0 of neural network fo using variant of stochastic gradient descent. The learning problem is continual because the distribution from which the data is sampled, p(z, y). is changing. For simplicity, we assume this non-stationarity changes the distribution over the input-targe pairs every T iterations. The data is sampled from a single distribution for T steps, and we refer to this particular temporary stationary problem task, т. The distribution over observations and targets that defines a task 7 is denoted by Pr-
Loss of plasticity can refer two related phenomena: loss of generalization (Ash and Adams 2020 Dohare et al.] 2024) or loss of trainability (Dohare et al.) 2021; Lyle et al. (202)). We focus our theoretical analysis on the problem of loss of trainability, in which we evaluate the neural network at the end of each task using samples from the most recent task distribution, Pr, as is commonly done in previous work (Lyle al., 2023). Loss of trainability refers to the problem where the neural network is unable to sustain its initial performance on the first task to later tasks. Specifically, we denote the optimisation objective by (0) = TE [.)p.))>p [<((())) v)], for some loss function E, and task-specifie data distribution py- We use / to denote the iteration count of the learning algorithm, and thus the current task number can be written as T(t) = Lt/T].
TRAINABILITY AND LINEARITY
In this section, we show that, unlike nonlinear networks, linear networks do not suffer from loss of trainability. That is. if the number of iterations in each task is sufficiently large, linear network sustain trainability on every task in the sequence. We then show theoretically that a special case of deep linear networks also does not suffer from loss of trainability, and we empirically validate the theoretical findings in more general settings. These results provide a theoretical basis for previous work that uses linear baseline in loss of plasticity experiments.
3.1 TRAINABILITY OF LINEAR FUNCTION APPROXIMATION
We first prove that loss of trainability does not occur with linear function approximation, fo(x) = Wj We prove this by showing that linear function approximators can sustain learning sequence of tasks, with large enough number of iterations per task. In particular, the perfor- mance of the solution found on the --th task can be upper bounded on a quantity that is independent of the solution found on the first 7 tasks. Linear function approximation avoids loss of trainability be- cause the optimisation problem on each task is convex (Agrawal et al.) 202T; Boyd and Vandenberghe 2004), with a unique global optimum, #7. We now state the theorem, which we prove in Appendix B
Theorem 1. Let (1rT)) denote the linear weights learned the end of the 7-th task, with the corresponding unique global minimum for lask T being denoted by O=. Assuming the objective function is p-strongly convex, the suboptimality gap for gradient descent on the 7-th lask is
2D(1 axp)
(((0(TT)) - J,(07) aT((1 - (1- )
where each task lasts for T iteration, D is the assumed bound on the parameters the global minimum for every task, and a is the step-size.
Intuitively, this theorem states that if the problem is bounded and effectively strongly convex due to a finite number of iterations, then the optimisation dynamics are well-behaved for every task in the bounded set. In particular, this means that the error on each task can be upper bounded by quantity independent of the initialization found on previous tasks. Thus, given enough iterations, linear function approximation can learn continually without loss of trainability.
3

---

Published as a conference paper at ICLR 2025
3.2 TRAINABILITY OF DEEP LINEAR NETWORKS
We now provide evidence that, similar to linear function approximation, deep linear networks also do not suffer from loss of trainability. Unlike deep nonlinear networks, deep linear networks use linear activation functions in their hidden layers (Bernacchia et al., 2018 Ziyin et al., 2022). This means that a deep linear network can only represent linear functions. At the same time, its gradient update dynamics are nonlinear and non-convex, similar to deep nonlinear neural networks (Saxe et al. 2014). Our central claim here is that deep linear networks under gradient descent dynamics avoid parameter configurations that would lead to loss of trainability.
To simplify notation, without loss of generality, we combine the weights and biases into a single parameter for each layer in the deep linear network, 8 {81..... Og J. and fo(x) 0-0 81æ. We denote the product of weight matrices, or simply product matrix, as = 0, 01.0,_111 ... 81. which allows us write the deep linear network in terms of the product matrix: fo(x)
Ov.. The problem setup we use for the deep linear analysis follows previous work (Huh) 2020). and we provide additional technical details for optimisation dynamics of deep linear networks Appendix A.3
We now provide evidence to suggest that, despite deep linear networks being nonlinear in their gradient dynamics, they do not suffer from loss of trainability. We prove this for special case of deep diagonal linear networks, and provide empirical evidence to support this claim in general deep linear networks.
Theorem 2. Let fo(x) = 01 OLæ R be deep diagonal linear network where At Diag(Or, ,0,d). Then, deep diagonal linear network converges a sequence of tasks under the same conditions for convergence in a single task (i.e., the conditions in Arora et al. 2019).
Theorem states that deep diagonal linear network, special case of general deep linear networks, can converge to a solution on each task within a sequence of tasks. The proof. provided in Appendix
B shows that the minimum singular value of the product matrix stays greater than zero, minn (8) 0.
Hence, deep diagonal linear networks do not suffer from loss of trainability. This result provides further evidence suggesting that linearity might be effective inductive bias for learning continually.
While the analysis considers a special case of deep linear networks, namely deep diagonal networks, note that this is common setting for the analysis of deep linear networks more generally (Nacson) eet al. 2022: Even et al.) 2023). In particular, the analysis is motivated by the fact that, under certain conditions, the evolution of the deep linear network parameters can be analyzed through the independent singular mode dynamics (Braun et al. 2022). which simplifies the analysis of deep linear networks to deep diagonal linear networks.
3.3 EMPIRICAL EVIDENCE FOR TRAINABILITY OF GENERAL DEEP LINEAR NETWORKS
In the previous section, we proved that a special case of deep linear networks do not suffer from loss of trainability. We now provide additional empirical evidence that general deep linear net-
10
works not suffer from loss of trainability. To do so, we use linearly separable subset of the
Accuracy
0.6
Linear
MNIST dataset (LeCun et al., 1998), in which
ReLu
the labels of each image are randomized every 100 epochs. For this experiment, the data is linearly separable that even a linear baseline can fit the data if given enough iterations. While MNIST is simple classification problem, memorizing random labels highlights the dif- ficulties associated with maintaining trainability (see Lyle et al.) 2023; Kumar et al.] 2023b). We emphasize that the goal here is merely to validate that linear networks remain trainable in continual learning. We also provide results with traditional nonlinear neural networks on the same problem. showing that they suffer from loss of trainability in this simple problem. Later in Section we extend our investigation of loss of trainability to larger-scale benchmarks.
0.4
0.2
50
100
150
Number of tasks
Figure 2: Trainability on linearly separable task. The higher opacity corresponds to deeper networks, ranging from ( (1. 2, 4. 8, 16). Deep linear networks sustain trainability on new tasks, with some additional depth improving trainability. Nonlinear networks, using ReLU, suffer from loss of trainability at any depth even on this simple sequence of linearly separable problems.
4

---

Published as a conference paper at ICLR 2025
In Figure 2 we see that deep linear networks ranging from depth of to 16 can sustain trainability. Using a multi-layer perceptron with ReLU activations, deep nonlinear networks quickly reach a much higher accuracy on the first few tasks. However, due to loss of trainability, deep nonlinear networks of any depth eventually perform worse than the corresponding deep linear network. With additional epochs, the linear networks could achieve perfect accuracy on this task because it is linear separable. The number of epochs is comparatively low to showcase that, with some additional layers, a deep linear network is able improve its trainability as new tasks are encountered.
COMBINING LINEARITY AND NONLINEARITY
In the previous section, we provided empirical and theoretical evidence that linearity provides an effective inductive bias for learning continually by avoiding loss of trainability. However, linear methods are generally not as performant as deep nonlinear networks, meaning that their sustained performance can be inadequate on complex tasks. Even deep linear networks have only linear representational power, despite their nonlinear gradient dynamics. We now seek answer the following question:
How can the sustained trainability of linear methods be combined with the expressive power of learned nonlinear representations?
To answer this question, we first seek to better understand the effects of replacing linear activation functions with nonlinear ones in deep networks for continual learning. We observe that deep linear networks have diversity in their hidden units, which can be induced in nonlinear activation functions by adding linearity through a weighted linear component, an idea we refer to as a-linearization. To dynamically balance linearity and nonlinearity, we propose to use deep Fourier features for every layer in a network. We prove that such a network approximately embeds a deep linear network, a property we refer to adaptive linearity. We demonstrate that this adaptively-linees network is plastic, maintaining trainability even on non-linearly-separd problems.
4.1 ADDING LINEARITY TO NONLINEAR ACTIVATION FUNCTIONS
Deep nonlinear networks can learn expressive representations because of their nonlinear activation functions, but these nonlinearities can also lead to issues with trainability. Although several components of common network architectures incorporate linearity, the way in which linearity is used does not avoid loss of trainability. One example is the piecewise linearity of the ReLU activation function (Shang et al. 2016), ReLU(x) = max(0, x),, which is said to saturated if ReLU(x) = for most inputs x, preventing gradient propagation. While saturation is generally not problem for learning on a single distribution, it has been noted as problematic in learning from changing distributions, for example, in reinforcement learning (Abbas et al. 2023
potential solution to saturation is to use a non-saturating activation function. Two noteworthy examples of not-satura activation functions include periodic activation like sin (z) PPara = az + (1 - ax) Re (Xu et al., 2015). both of
candolo et al. 2017 and leaky-Reiu, which are zero on a of measure zero. Surprisingly, using 1 leaky-ReL. leads
related issue, "unit lincarization" (Lyle et al. 2024). in which the activation is only positive (or negative) for most inputs œ. Unlike saturated units, linearized units can provide non-zero gradients but render that unit effectively linear, limiting the expressive power of the learned representation. While unit lincarization seems to suggest that loss of trainability can occur due to linearity, it is important to note that a "linearized unit" is not the same as linear unit. This is because a linearized unit provides mostly positive (or negative) outputs, whereas a linear unit can output both positive and negative values.
We generalize the idea behind unit saturation and unit linearization to unit sign entropy, which is applicable to activation funetions beyond saturating and piecewise linear functions, such as periodic activation functions. Intuitively, it measures the diversity of the activations of hidden layer.
Definition (Unit Sign Entropy). The entropy, H. of the unit's sign, sgn(h(x)). on distribution of inputs to the network, p(x), is given by H (sgn(h(z)) = Ep(z) (sgn(h(z))
The maximum value of unit sign entropy is 1, which occurs when the unit is positive on half the inputs. Conversely, a low sign entropy is associated with the aforementioned issues of saturation and lincarization. For example, low sign entropy for deep network using Le activations means that the unit is almost always positive (P (sgn(h(zz) = 1) = 1, meaning it is linearized) or negative (P (s))()(x)) = 1) meaning it is saturated).
5

---

Published as a conference paper at ICLR 2025
With unit sign entropy. investigate how the leak parameter for the leaky-ReIO activation function influences training as pure linearity (a = 1) is traded-off for pure nonlinearity (a = 0). The idea of mixing linearity and nonlinearity can also be generalized to an arbitrary activation function, which we refer to as the a-linearization of an activation function.
Definition 2 (a-linearization). The a-linearization of an activation activationfumcciinn function @ is denoted by O (z) = or (1 (x)(())).
A natural hypothesis is that, as increases from 0 to 1, and the network becomes more linear, loss of trainability is mitigated. We emphasize that the a-linearization is pri- marily to gain insights from empirical investigation and it not a solution to loss of trainability. This is because any benefits of a-linearization depend on tuning a, and even optimal tuning can lead to overly linear representations and slow training compared to nonlinear networks.
0.8
Accoracy
0.6
o-rel ta = 0)
dn
0.4
Deep Lincar
25
50
75
100
Number Tasks
Empirical Evidence for a-linear Plasticity To under- stand the trainability issues introduced by nonlinearity, we present a case-study using a n and ReLU with different values of the linearization parameter, a. The same exper- iment setup is used from Section 3.3 Referring to the results in Figure 3 we see that both ReLU and sin acti- vation functions are able to sustain trainability for larger values of a. This verifies the hypothesis: a larger a pro- vides more linearity to the network, allowing it to sustain trainability. Despite sustaining trainability, a larger a can lead to overly linear representations, evidenced by worse performance and slower training speed on the first few tasks compared to nonlinear networks (a 0). For a-ReLU, also verify the hypothesis that the unit sign entropy increases for larger values of a (inset plot). The fact that the periodic sin activation function has high unit sign entropy despite losing trainability is particularly interesting, which we explore in the next section.
Figure 3: Trainability on a linearly separable task with a-lincarization Darker opacity lines correspond to higher values of Unit sign entropy increases as a increases (inset), leading sustained trainability for a-relu.
4.2 ADAPTIVE-LINEARTT BY CONCATENATING SINUSOID ACTIVATION FUNCTIONS
Using the insight that linearity promotes unit sign entropy, we explore an alternative approach to sustain trainability. In particular, found that linearity sustain trainability but requires tuning a, and even optimal tuning can lead to slow learning from overly linear representations. Our approach is motivated by concatenated ReLU activations (Shang al. 2016 Abbas al., 2023), (R)) = [U Re]U(-2)], which avoids the problems from saturated units, but does not avoid the problem of low unit sign entropy. In particular, we propose using a pair of activations functions such that one activation function is always approximately linear, with a bounded error.
One way to dynamically balance the linearities and nonlinearities of a network using periodic activation functions. This is because, due to their periodicity, the properties of the activation function can re-occur as the magnitude of the preactivations grows rather than staying constant, linear, or saturating. But, as we saw in Figure 3 single periodic activation function like sin is not enough. Instead, we propose to use deep Fourier features, meaning that every layer in the network uses Fourier features. This is a notable departure from previous work which considers only shallow Fourier features in the first layer (Rahimi and Recht 2007 Tancik et al., 2020). In particular, each unit is concatenation of a sinusoid basis of two elements, Fouri e ((2) = sin(z))cos({)) Each pre-activation is mapped to both sin(z) and cos(z), which requires that a layer with deep Fourier features have half the output width to accomodate the concatenation
The advantage of this approach is that a network with deep Fourier features maintains approximate linearity in all of its parameters. Moreover, deep Fourier features are closed under differentation, meaning that the activations and their gradients provide a basis for representing periodic functions..
Proposition 1. For any z, there exists a linear function, [x(x) = a(=) + b(z), such that either:
sin(z) - [x)] < c, or cos(2)) - Lg (x)) & c, for /2=" /22 and all x E [2 7/1, + 7/4].
For fixed width, network with deep Fourier features has approximately half the number of parameters.

---

Published as a conference paper at ICLR 2025
An intuitive description of this is provided in Figure I The advantage of using two sinusoids over just a single sinusoid is that whenever cos(2) is near a critical point, 4/d= c2)(2) = 0, we have that sin(2) z 2, meaning that (dssin(z)) == 1 (and vice-versa). The argument follows from analysis of the Taylor series remainder, showing that the Taylor series of half the units in a deep Fourier layer can be approximated by linear function, with small error of v2x2/25 x 0.05. While found that two sinusoids is sufficient, the approximation error can be further improved by concatenating additional sinusoids, at the expense of reducing the effective width of the layer.
Because each pre-activation is connected to a unit that is approximately linear, we can conclude that a deep network comprised of deep Fourier features approximately embeds deep linear network. Corollary 1. A network parameterized by 0, with deep Fourier features, approximately embeds a deep linear network parameterized by 0 with bounded error.
Notice that piecewise linear activations also embed deep linear network, but these embedded deep linear networks do not use the same parameter set. For example, the deep linear network embedded by a ReLU network does not depend on any of the parameters used to compute a ReLU unit that is zero. Although the leaky-ReLD function involves every parameter, the deep linear network vanishes because the leak parameter is small, a < 1, and hence the embedded deep linear network multiplied by a small constant, L, where L is the depth of the network.
CReLL
FourienRReL Deep Fourler
sish (a 0.5)
1.0
Accuracy
0.6
0.2
20
40
60
80
Number of tasks
Empirical Evidence for Nonlinear Plasticity We Figure 4:
Trainability on non linearly-separable task. Deep Fourier features improve and sustain their train- ability when other networks cannot.
now consider similar experimental setup from Sections 3.3 and 4.1 except we make the problem non linearly- separable by considering random label assignments the entire dataset. Each task is more difficult because it involves memorizing more labels, and the effect of the non-stationarity is also stronger due to randomization of more datapoints. As a result, the deep linear network can no longer fit a single task well. Referring to Figure the a-linear activation functions can sustain and even improve their trainability, albeit very slowly. See also unit sign entropy in Figure 9 Appendix D.]] In contrast, using deep Fourier features within the network enables the network easily memorize all the labels for 100 tasks. Deep Fourier features surpass the trainability of the other nonlinear baselines at initialization, CReLU and shallow Fourier features followed by ReLU This is surprising, because deep nonlinear networks at initialization are often a gold-standard for trainability.
5 EXPERIMENTS
Our experiments demonstrate the benefits of the adaptive linearity provided by deep Fourier features. While trainability was the primary focus behind our theoretical results and empirical case studies, we show that these findings generalize to other problems in continual learning. In particular, demonstrate that networks composed of deep Fourier features are capable of being strongly regularized leading to improved generalization performance on diminishing levels of label noise, and in class-incremental learning. The main results we present are on all of the major continual supervised learning settings considered in the plasticity literature. They build on the standard ResNet- 18 architecture, widely used in practice (He et al. 2016).
Datasets and Non-stationarittes Our experiments use the common image classification datasets for continual learning, namely iiny-ImageNet (Le and Yang 2015). CIFARIO. and CIFAR100 (Krizhessky) 2009). We augment these datasets with commonly used non-stationarities to create continual learning problems, with the non-stationarity creating sequence of tasks from the dataset. Specifically, we follow recent work that introduced the diminishing label noise problem (Lee et al.) 2024), which is inspired by the warm-starting problem: We start with half the data being corrupted by label noise and reduce the noise to clean labels over 10 tasks. Additionally, for the datasets with a larger number of classes, tiny- Imy-ImageNet and CIFAR100 100, we also consider the class- incremenall setting: the first task involves only five classes, and five new classes are added to the existing pool of classes at the beginning of each task (Van de Ven et al. 2022). Other results and more details on datasets and non-stationarities considered can be found in Appendix C
7

---

Published as a conference paper at ICLR 2025
thty ImageNet
CITARLDO
CIFARLO
Leerot
0.33
0.72
Ocep fouttor
Teee Arcurncy
0:32
Teet Arcurncy
0.65
0.22
0.22
0.25
0.33
0:12
0:22
0.52
0.12
200
1000
220
K00
500
0.41
235
203
420
Epochs
Epochs
Number Epochs
Figure 5: Training ResNet-18 continually with diminishing label noise. Deep Fourier features are particularly performant on complex tasks like uny-ImageNet. Despite networks with deep Fourier features having approximately half the number of parameters, they surpass the baselines in CIFARIO0 and are on mn-par with spectral regularization on CIFARIO.
Architecture and Baselines We compare ResNet- 18 using only deep Fourier features against a standard ResNet- 18 with ReLU activations. The network with deep Fourier features has fewer parameters because it uses concatenation of two different activation functions, halving the effective width compared to the network with ReLU activations. This provides an advantage to the nonlinear baseline. We also include all prominent baselines that have previously been proposed mitigate loss of plasticity in the field: L2 regularization towards zero, L.2 regularizatio
towards the initialization (Kumar et al., (023b), spectral regularization (Lewandowski et al. 2024). Concatenated ReLU (Shang) iet al.] [0166] Abbas et al. 2023). Dormant Neuron Recycling (ReDO, Sooar et al. 2023), Shrink and Perturb (Ash and Adams, 2020). and Streaming Elastic Weight Consolidation (S-EWC, Kirkpatrick
iet al. 2017 Elsayed and Mahmood|21 2024).
5.1
MAIN RESULTS
Our results demonstrate that deep Fourier features, combined with regularization, are effective at continual learning. In these set of experiments, we consider the problem of sustaining test accuracy on a sequence of tasks. In addition to requiring trainability, methods must also sustain generalization.
Diminishing Label Noise In Figure 5 we can clearly see the benefits of deep Fourier features in the diminishing label noise setting. At the end of training on ten tasks with diminishing levels of label noise, the network with deep Fourier features was always among the methods with the highest test accuracy on the the uncorrupted test set. On the first of ten tasks, deep Fourier features could occasionally overfit to the corrupted labels leading to initially low test accuracy. However, as the label noise diminished on future tasks, the network with deep Fourier features was able to continue to learn to correct its previous woolly-enneralizing predictions. In contrast, the improvements achieved by the other methods that we considered was oftentimes marginal compared to the baseline ReLU network. Two exceptions are: (i) networks with CReLU activations, which underperformed relative to the baseline network, and (ii) Shrink and Perturb, which was the best-performing baseline method for diminishing label noise. Interestingly, the performance benefit of deep Fourier features is most prominent on more complex datasets, like tiny-ImageNe In Appendix D.2 we provide an ablation of the architecture, where we use Wide Residual Network (Zagoruyko and Komodakis, 2016) and
vary the width scale.
Ling-ImagoNet
CIFAR1D0
Maas--nnrrmenttl Learning Deep Fourier features are also effective in the insremnalr setting, where later tasks involve training on a larger subset of the classes, following the experiment described (Dohare et al.) 2024). The network is evaluated at the end of each task on the entire test set. As the network is trained on later tasks, its test set performance increases
0.30
0.2
0.30
Tust Accuracy
0
0.35
0.30
2ea 10 S100
0.08
Serok
Spectrettgg
Number tasks
Number tasks
Figure 6: Class incremental learning results on tmny-emagenet (Left) and CIFAR-100 (Right). On both datasets, deep Fourier features substantially improve over most baselines.
8

---

Published as a conference paper at ICLR 2025
because it has access to a larger subset of the training data. In Figure 6 features largely outperform the baselines in this setting, particularly on tiny- min--Im in which the first forty tasks involve training on growing subset of the dataset and the last forty "tasks" involve training to convergence on the full dataset. We use quotation marks to characterize the last forty tasks because they are, in fact, single task, as the data distribution stops changing after the first forty tasks. We call them "tasks" because of the number of iterations in which they are trained. Not only are deep Fourier features quicker learn on earlier continual learning tasks, but they are also able to improve their generalization performance by subsequently training on the full dataset. On CIFAR 100, the difference between methods is not as prominent, but we can see that deep Fourier features are still among the top-performing methods. The large performance difference on tiny Imn-lmmeeNet can be attributed to the fact that it a harder problem compared to CIFARIO 10 and CIFAR 100, with higher resolution images, more classes and more datapoints.
that Deep Fourier
5.2 SENSITIVITY ANALYSIS
In the previous sections, we found that deep Fourier features used in combination with spectral regularization leads to strong generalization performance. However, the theoretical analysis and case-studies that we presented earlier concerned trainability. We now present a sensitivity result to un- derstand the relationship between trainabil- ity and generalization. Using ResNet- 18 with di different activation functions, we var- ied the regularization strength between no regularization (left) and high regularization (right). In Figure 5.22 can that deep Fourier features indeed have a high degree of trainability, sustaining higher trainabil- ity at every level of regularization strength. However, without any regularization, deep Fourier features have a tendency to overfit. Over-fitting is a known issue for shallow
12
Iran Accuracy
Tiat
1P
27*
Reguarization Strength
Reguarization Strength
Figure 7:
Sensitivity analysis on works with deep Fourier features are highly trainable, but have a tendency to overfit without regularization, leading high training accuracy but low test accuracy. Due to deep Fourier features heing highly trainable, they are able to train with much higher regularization strengths leading to ultimately better generalization.
Net-
Fourier features (e.g.. when using Fourier features only for the input layer, Mavor-Parkee et al. 22024) and this can be attributed to their spectral bias of learning figh-freuency features (Tancik al.. 2020). However, deep Fourier features are able to use their high trainability to learn effectively even when highly regularized. Thus, while high trainability does not always lead to high generalization, the trainability provided by deep Fourier features can be used in combination with regularization to improve continual learning performance. Hyperparameter sensitivity results are presented on other We also provide an in-depth sensitivity study smaller-scale MLPs in datasets in Appendix D.6 Appendix D.8
6 CONCLUSION
In this paper, we proved that linear function approximation and special case of deep linearity are effective inductive biases for learning continually without loss of trainability. This surprising finding for deep linear networks suggests that nonlinearity of representations, rather than nonlinearity of gradient dynamics, contributes to loss of plasticity. We then investigated the issues that arise from using nonlinear activation functions, namely the problem of low unit sign entropy, which indicates a lack of diversity in the activations as measured by the entropy of the sign of the hidden units. Motivated by the effectiveness of linearity in sustaining trainability, we proposed deep Fourier features to approximately embed deep linear network inside a deep nonlinear network. We found that deep Fourier features dynamically balance the trainability afforded by linearity and the effectiveness of nonlinearity, thus providing effective inductive bias for learning continually. Experimentally, we demonstrated that networks with deep Fourier features provided benefits for continual learning across every dataset we considered. We found that networks with deep Fourier features were effective plastic learners because their trainability enabled training with a much larger regularization strength, leading superior generalization performance.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The research supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR Al Chair Program, Alberta Innovates, and the Digital Research Alliance Canada.
REFERENCES
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning, In Conference Lifelong Learning Agents.
Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, P., and Singh, (2024). A definition of continual reinforcement learning. Advances in Neural Information Processing Systems.
Agrawal, A., Barratt, S., and Boyd, S. (2021). Learning convex optimization models. Journal of Automatica Sinica, 8(5):1355-1364
Arora, S., Cohen, N., Golowich, N., and Hu, W. (2019). A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations.
Arora, S., Cohen, N., and Hazan, E. (2018). On the optimization of deep networks: Implicit acceleration by overparameterization. International Conference on Machine Learning.
Ash, J. T. and Adams, R. P. (2020). On Warm-Starting Neural Network Training. In Advances in Neural Information Processing Systems.
Ba, I.,, Kiros, R., and Hinton,
(2016). Layer normalization. CoRR, Abs/1160.004400v/
Bah, B., Rauhut, H., Terstiege, U., and Westdickenberg, M. (2022). Learning deep linear neural networks: Riemannian gradient flows and convergence global minimizers. Information and Inference: A Journal of the IMA.
Bernacchia, A., Lengyel, M., and Hennequin, G. (2018). Exact natural gradient in deep linear networks and its application to the nonlinear case. Advances in Neural Information Processing Systems.
Boyd, S. and Vandenberghe, L. (2004). Convex optimization, Cambridge university press.
Braun, L., Dominé, C., Fitzgerald, J., and Saxe, A. (2022). Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems.
Chou, H.-H., Gieshoff, C., Maly, J., and Rauhut, H. (2024). Gradient descent for deep matrix factorization:
Dynamics and implicit bias towards low rank. Applied and Computational Harmonic Analysis.
Cohen, G., Afshar, S., Tapson. J., and Van Schaik, A. (2017). Emnist: Extending mnist to handwritten letters. In International Joint Conference on Neural Networks (ICNNN).
Dohare, S., Hernandez-Garvia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. (2024). Loss of plasticity in deep continual learning. Nature, 632(8006))
Dohare, S., Sutton, R. S., and Mahmood, A. R. (2021). Continual backprop: Stochastic gradient descent with persistent randomness. CORR, 0al/22188.063254:
Elsayed, M. and Mahmood, A. R. (2024). Addressing loss of plasticity and catastrophic forgetting in continual learning. In International Conference Learning Representations.
Even, M.. Pesme, S., Gunasekar, S., and Flammarion, N. (2023). (s)GD over diagonal linear networks: Implicit bias, large stepsizes and edge of stability. In Advances in Neural Information Processing Systems.
Garrigos, G. and Gower, R. M. (2023). Handbook of Convergence Theorems for (Stochastic) Gradient Methods. CoRR, 0ts//301.1123355
10

---

Published as a conference paper at ICLR 2025
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In International Conference Computer Vision.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition.
Huh, D. (2020). Curvature-corrected learning dynamics in deep neural networks. In International Conference on Machine Learning.
loffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Leurning.
Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems.
Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. In International Conference Learning Representations.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A., Milan, K., Quan,
J.. Ramalho, T., Grabska-Barwwing A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3211-3526.
Kleinman, M., Achille, A., and Soatto, S. (2024). Critical learning periods emerge even deep linear networks. In International Conference on Learning Representutions.
Konidaris, G., Osentoski, S., and Thomas, P. (2011). Value function approximation in reinforcement learning using the fourier basis. In AAAI Conference on Artificial Intelligence.
Krizhevsky, A. (2009). Learning multiple layers features from tiny images. Technical report, University of Toronto.
Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, J., Liu, Y., and Van Roy, B. (2023a). Continual Learning as Computationally Constrained Reinforcement Learning. CoRR, abs/2307.04p3 0.4455.
Kumar, S., Marklund, H., and Roy, B. V. (2023b). Maintaining plasticity via regenerative regulariza- tion. CoRR, abs/2308. 11958v1.
Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A., and Ganguli, (2024). Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. CoRR, 00s/22406.061581..
Le, Y. and Yang, X. (2015). Tiny imagenet visual recognition challenge.
LeCun, Y., Cortes, C., and Burges, C. (1998). MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun http://aann.lecuu.ccooeeeee wwtwwwwww.w./ee///isss.tt.t
Lee, H., Cho, H., Kim, H., Kim, D., Min, D., Choo, J., and Lyle, C. (2024). Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In International Conference on Machine Learning.
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, (2019). Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems.
Lewandowski, A., Kumar, S., Schuurmans, D., György, A., and Machado, M. (2024). Learning Continually by Spectral Regularization. CoRR, abs/2406.068 01w.400006661111
Li, A. C. and Pathak, D. (2021). Functional regularization for reinforcement learning via learned fourier features. In Advances in Neural Information Processing Systems.
Liu, Y.. Kuang, X., and Roy, B. V. (2023). A Definition con-Stationary Bandits. CoRR, abs/2302. 12202v2.
11

---

Published as a conference paper at ICLR 2025
Lyle, C., Rowland, M., and Dabney, W. (2022). Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations.
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. (2024).
Disentangling the Causes of Plasticity Loss in Neural Networks. CoRR, abs/2402.18762v1.
Lyle, C., Zheng, Z., Nikishin, E., Avila Pires, B., Pascanu, R., and Dabney, W. (2023). Understanding plasticity in neural networks. In International Conference on Machine Learning.
Mavor-Parker,, A. N., Sargent, M. J., Barry, C.,, Griffin, L., and Lyle, C. (2024). Frequency and Gener- alisation of Periodic Activation Functions in Reinforcement Learning. CoRR, 0ess/2407.067566
Nacson, M. S., Ravichandran, K., Srebro, N., and Soudry, D. (2022). Implicit bias of the step size in linear diagonal neural networks. In International Conference Machine Learning.
Parascandolo, G., Huttumen, H., and Virtanen, T. (2017). Taming the waves: sine as activation function in deep neural networks.
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C.,, and Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71.
Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in Neural Information Processing Systems.
Ring, M. B. (1994). Continual learning in reinforcement environments. The University of Texas at Austin.
Saxe, A., McClelland, J., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Represenatations.
Shang,
W., Sohn, K., Almeida, D., and Lee, H. (2016). Understanding and improving convolutional neural networks via concatenated rectified linear units. In International Conference Machine Learning.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning.
Tancik, M., Srinivasan, P., Mildenhall, B., -ridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoor- thi, R., Barron, J., and Ng. R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems.
Thrun, S. (1998). Lifelong learning algorithms. In Learning to Learn, pages 181-209. Springer.
Van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. (2022). Three types of incremental learning. Nature Machine Intelligence, 4(12): 185-11977
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747.
Xu, B., Wang, N., Chen, T., and Li. M. (2015). Empirical Evaluation of Rectified Activations in Convolutional Network. CORR, abs/1 505.0083333..
Yang, G., Ajay, A., and Agrawal, P. (2022). Overcoming the spectral bias of neural value approxima- tion. In International Conference on Learning Representations.
Yang, G., Simon, B., and Bernstein, J. (2023). A Spectral Condition for Feature Learning. CoRR, abs/2310.1781144
Zagoruyko, S. and Komodakis, N. (2016). Wide Residual Networks. CoRR, abs/1605.07 146v4.
Ziyin, L., Li, B., and Meng. X. (2022). Exact solutions of deep linear network. Advances in Neural Information Processing Systems.
12

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL DETAILS
A.1 ASSUMPTIONS FOR TRAINABILTTY OF LINEAR FUNCTION APPROXIMATION
We assume that the parameters at the global optimum for every task are bounded: ||0-|2 < D. This is true for regression problems if the observations and targets are bounded. In classification tasks, the global optimum can be at infinity because activation functions such as the sigmoid and the softmax are maximized at infinity. In this case, we constrain the parameter set, {0: ||01|2 < D}. and project the optimum onto this set.
In addition to convexity, we assume that the objective function is j--strongly convex, (0) μI, where V² denotes the Hessian. Note that neither squared nor cross-entropy loss are j-strongly in general. However, this assumption is satisfied in continual learning problems with finite number of iterations. For regression, denote the observations for task T as X, E mathbbR where N is the sample size. Then the Hessian is the outer products of the data matrix, V. X, E Rddd Thus, the squared loss is strongly-convex the data is full rank. This is satisfied in high dimensional image classification problems, which is what we consider.
For classification, the Hessian involves an additional diagonal matrix of the predictions for each datapoint.
ViJccass -
X, (0)=
Raxd
where D Diag(Pi, ... PN) and p; (1) (fo(x:)) If the prediction becomes sufficiently confident, offa (x;) = 1, then there can be rank deficiency in the Hessian. However, because each task is only budgeted finite number of iterations this bounds the predictions away from 1.
A.2
RELATED WORK REGARDING TRAINABILITY OF DEEP LINEAR NETWORKS
Some authors have suggested deep linear networks suffer from a related issue, namely that critical learning periods also occur for deep linear networks (Kleinman et al. 2024). Unlike the focus on loss of trainability in this work where the entire network is trained, these critical learning periods are due winner-take-all dynamics due manufactured defects in one half of the linear network, for which the other half compensates.
Finally, we note that some previous work have found that gradient dynamics have low rank bias for deep linear networks (Chou et al. (0024). One important assumption that these works make is that the neural network weights are initialized identically across layers, 0, = a02 Our analysis assumes that the initialization uses small random values, such as those used in practice with common neural network initialization schemes (Glorot and Bengio, 2010: He et 2015).
A.3 DETAILS FOR DEEP LINEAR SETUP
The gradient of the loss function with respect the parameters of deep linear network can be written in terms of the gradient with respect to the product matrix 0 (Bah et al. 2022):
V e ((0) =
0%. (()0) 0 A..
(1)
where the term (0) is the gradient of the loss with respect to the product matrix, treating it as if it was linear function approximation. The gradient is nonlinear because of the coupling between the gradient of the parameter at one layer and the value of the parameters of the other layers. Nevertheless, the gradient dynamies of the individual parameters can be combined to yield the dynamics of the product matrix (Arora et al. 2018).
VaJ = - Po)
The dynamics involve a preconditioner. Ps. that accelerates optimisation (Arora et al. 2018). which we empirically demonstrate in Section 3.3 On the left-hand side of the equation. we use V (nJ(O) to denote the combined dynamics of the gradients for each layer on the dynamics of the product matrix This means that the effective gradient dynamics of the deep network is related to the dynamics of
Note use V because V J(O)) is gradient for any function of 0; see discussion by Arora et al. 2018
13

---

Published as a conference paper at ICLR 2025
linear function approximation with a precondition. While the dynamics nonlinear and non-convex, the overall dynamics are remarkably similar to that of linear function approximation, which is convex.
To simplify notation, without loss of generality, consider deep linear network without the bias terms, 0 = [ } and fz( = 010 01z. We denote the product of weight matrices, or simply product matrix, f = 0g
01, which allows us to write the deep linear network in terms of the product matrix: fa (z) = dx. The problem setup we use for the deep linear analysis follows previous work (Huh) 2020), and we provide additional details in Appendix A.3 We consider
the squared error, J- (+) = E --p- [lly 0x:l] 2"
and we assume that the observations are whitened
to simplify the analysis,
Σg = E pr = I. focusing on the case where the targets y are changing during continual learning. Then we can write the squared error as
(( Tr |
where 4, = 0 - is the distance to the optimal linear predictor, 0 = Ege,r = Eu, [yx JEa-
The convergence of gradient descent for general deep linear networks requires assumption on the deficiency margin, which is used to ensure that the solution found by a deep linear network, in terms of the product matrix, is full rank (Arora et al./ (211)). That is, the deep linear network converges if the minimum singular value of the product matrix stays positive, Tmin (0) > 0.
We now show that diagonal linear network maintains a positive minimum singular value under continual learning. This is a simplified setting for analysis, where we assume that the weight matrices are diagonal and thus the input, hidden, and output dimension are all equal. Let fe (zx) be a diagonal linear network, defined by a set of diagonal weight matrices, 0₂ = Diag (f,,,, .1,a). The output of the diagonal linear network is the product of the diagonal matrices, fo (2) = 0-0 - O2 x. Then the product matrix is also a diagonal matrix, whose diagonals are the products of the parameters of each layer, 0 = Diag([I or...... II 0r,a) == Di. Ba). The minimum singular value of diagonal matrix is the minimum of its absolute values, minn (0) min, (0.). Thus, we must show that the minimum absolute value of the product matrix is never zero
Lemma 1. Consider a deep diagonal linear network, fo(x) 0,0 ............... 0,x and 0 = Diag(0, 1r...... ,0,d). Then, under gradient descent dynamics, 0/ = 0/% itf of 0 for l' # L.
The proof of this proposition, and the next, can be found in Appendix B This first proposition states that two parameters that are initialized to different values, such as by a random initialization, will never have the same value under gradient descent. Conversely, if the parameters are initialized identically, then they will stay the same value under gradient descent. This means that, in particular, two parameters will never be simultaneously zero
Lemma 2. Denote deep diagonal lineur nelwork Sa(x) = Diag(0 Da)æ where 0, Ih' fr,5, Then, under gradient descent dynamics, g.1) = o((+1) 0 iff two (or more) components are zero, of o(() 0, for L' +l.
While the analysis considers special case of deep linear networks, namely deep diagonal networks, we note that this is a common setting for the analysis of deep linear networks more generally (Nacson) et al. 2022. Even et al., 2023). In particular, the analysis is motivated by the fact that, under certain conditions, the evolution of the deep linear network parameters can be analyzed through the indepen- dent singular mode dynamics (Saxe al., 2014), which simplify the analysis of deep linear networks to deep diagonal linear networks. The target function being Iearned, y (x) = 0 x. is represented in terms of the singular- -aauue value decomposition, A* U S* Σ5= 8uu,u We also assume that the neural network has fixed hidden dimension, so that 01 E Rdxd,.,, 92 E Rd xd 0f<i<< E Rddd and we apply the singular value decomposition to the function approximator's parameters, O2 U.S.V E dout xda, To simplify the product of weight matrices, we assume Vi+1 = U, Vi2 = V
and Un = U The simplifying result is that the squared error loss can be expressed entirely in terms of the singular values, Ily x I 6 x ||S* - II S.ll, which is equivalent to our analysis of the deep diagonal network, the matrix of singular values is diagonal matrix. These decoupled learning dynamics are closely approximated by networks with small random weights and they persist under gradient flows (Huh) 2020).
14

---

Published as a conference paper at ICLR 2025
A.4 PSEUDOCODE FOR DEEP FOURIER FEATURE LAYER
Algorithm Deep Fourier Feature Layer
function DERRFOOURIIR,,,ATUURRRC,, W, b)
2:
f- Wr + b
b Calculate pre-activation
3: d1 - sin(z)
b Apply sine activation
4: 0g cos(2
Apply cosine activation
5: output t [a1
b Concatenate activations
6: return output
7:
end function
B PROOFS
Proof of Theorem 7 We first present the result for two tasks and we then generalize it to an arbitary number of tasks. Let the linear weights learned on the first task be B(T), with the corresponding unique global minimum denoted by 07. The solution found on the first task is used an initialization on the second task, which will end ((22", with the corresponding unique global minimum denoted by 05. We start from the known suboptimality gap for gradient descent on the second task (Garrigos) and Gower 2023):
103 10--0'1)1-
J2(0()T)) (2(
aT
We upper bound the distance from the initialization on the second task, ((T), to the optimum, 05. by
|02- - 112 < 110g 112 + 110 - 1 < 110 - 011 + (1 aa)) 10 0oll? (4)
Where the last inequality uses the assumption that the objective function u-strongly convex. We upper bound the suboptimality gap the second task by quantity independent of ((T):
(2(()) (2() |10 11
|02 0111 + (1 a 10; 00
aT
oT
which implies that the parameter value learned on the previous task does not influence training on the new task beyond dependence on the initial distance. This is true for an arbitrary number of tasks:
(1- TI) 11.
2D - at4)
J -
- (07)
(6)
oT
aT (1 au))²)
where we denote Oj- 0g. The last inequality follows from our assumption that the distance between
the task solutions, 10% - 0 2 < 2D, is bounded and using geometric sum in (1 - a4))
Proof of Lemma 7
We first prove the lemma the forward direction:
Assuming that 0/2
0, for l' + I, we will show that 0/t-
(t-1))
Writing the gradient update for 0/0 with a fixed step-size a, we have that
00/ = (t-1) aV J(0)
(7)
o (1-1) aV f(((f(()) v) v) o, fo(x)
(8)
0b- V (.(()())
11-1)
(9)
3.2
0 (0-1) - aV fo v)
(10)
j.8
(11)
15

---

Published as a conference paper at ICLR 2025
Similarly, the gradient update for Bp is
0 = 0/. - -VV
II 1
9)
(12)
(13)
Using our assumption that f = = we set the two updates equal to eachother:
- & (f)()) (2),
II
o - 01 aV (o v) II 0 x.
(14)
1+1
We can simplify both sides of the equations, where the LHS
0 aV a ((o(())))))))
II - x
(15)
((-11)
(t-1) 1 1 a ((f((()) II (t-1)
(16)
//- 1
Similarly, the RHS of the equation is
((--1)
(1 9)
II
(17)
Notice that both expressions in the parenthesis on the LHS and RHS are equal. Thus, /0-11) The reverse direction follows directly by following the above argument in reverse.
0(1-1)
Proof of Lemma
We first prove the lemma the forward direction:
Assuming that ((41) 0 = 0, we will show that 0 = 0 = 0.
We proceed by contradiction, and assume that only a single component is zero, that is 0/ = 0 and
(()) L. * 0 for l + l', We will show that the gradient update will ensure that gl++1) * 0
First, consider the update to 8/:)
oft = 0/9 (V v)
II
(18)
J
-_V v)
(t-1
(19)
Because assumed that 0/2 * 0 for * 1', we have that IT 0(t-11) 1) # 0. Thus g((+ #
Next consider the update e(0
(t+1)
0/00 a ((fo()))
9)
111-1)
(20)
1,
3.
jz
(1)
(21)
16

---

Published as a conference paper at ICLR 2025
Where the last line follows from the fact that 11 00 because 0/%, 0.
Thus, we have shown that 0(4-1) + for all and hence, 0(++1 of 0 which is a contradiction.
+ The reverse direction follows from the assumption directly. If two components are both equal to zero, 0 0/ = 0, then every sub-product is zero, Il (((-1) and so the entire product, J,
((t-1) II J
Proof of Theorem 2 We now show that a diagonal linear network maintains a positive minimum singular value under continual learning. This is simplified setting for analysis, where we assume that the weight matrices are diagonal and thus the input, hidden, and output dimension are all equal. Let (z) be a diagonal linear network, defined by a set of diagonal weight matrices, 0r = Diag(Or.1,.... O1.a). The output of the diagonal linear network is the product of the diagonal matrices, fo(z) = 0 Oz.. Then the product matrix is also a diagonal matrix, whose diagonals are the products of the parameters of each layer, 0- Diag [1 01.1,.... II 01,a) == Diag(O 0a). The minimum singular value of diagonal matrix is the minimum of its absolute values, miin (0) min; |0.|- Thus, we must show that the minimum absolute value of the product matrix is never zero
This follows immediately from Lemma I and Lemma 2 Taken together, these two lemmas state that with a random initialization and under gradient dynamics, a diagonal linear network will not have more than one parameter equal zero. This means that the minimum singular value of the product matrix will never be zero. Thus, we have shown that a diagonal linear network trained with gradient descent, if initialized appropriately, will be able to converge on any given task in sequence.
Proof of Proposition 7 We prove this by considering the remainder of Taylor series on the given interval. Due to periodicity of sin(z) and cos(2), we can consider E n, x] without loss of generality. We can further consider two cases, either 2 E [-TT, -3//4] U [ 7/4, 7/a] U [37/4, r] or E (-3r/4, -//4] [m/4,, 3x/a]. In the first case, is near a critical point of cos (z) and in the second case is near critical point of sin(z).
We focus on a particular subcase, where 2 E [-7/4, 7/4,, which close to a critical point of cos(2). but far from critical point of sin(h) (the other cases follow similar argument).
Because we know that x [-<//4 by Taylor's theorem follows that si) = 2 R1,,, where R1) sin (c) 22 is the 1st degree Taylor remainder centered at a for some E [-//4, 7/4]. 2
In the case of a sinusoid, this can be upperbounded, [[],v(=)| = | sin(c) 2$| < 3V2 (7/4)², using the fact that |z| < 7/1 and (in(c) < 1/v2.
Thus, when (z) is close to critical point, sin(z) is approximately linear. A similar argument holds for the other case, when sin(=) is close to critical point, cos(=) is approximately linear. In this other case, the error incurred is the same.
Proof of Corollary 7 We prove this claim using induction.
Base case: We want to show that a single layer that outputs Fourier features embeds deep linear network. Using Proposition |
there exists one unit for each pre-activation that is approximately linear. Because each pre-activation is used an approximately- unit, the single layer approximately embeds a deep linear network using all of its parameters.
Induction step: Assume a deep Fourier network with depth prove that adding an additional deep Fourier layer retains the embedded deep linear network. There are two cases to consider, corresponding to the units of the additional deep Fourier layer which are approximately linear and the other units that not approximately-
embeds a deep linear network, we
Case (approximately-linee) units): For the additional deep Fourier layer, the set of approximately- linear units already embeds deep linear network. Because linearity is closed under composition, the composition of the additional deep Fourier layer and the deep Fourier network with depth L 1 simply adds an additional linear layer to the embedded deep linear network, increasing its depth to L.
17

---

Published as a conference paper at ICLR 2025
Case 2 (other units): For the units that not well-approximatee by linear function, treat them as they were separate inputs to the deep Fourier network with depth L - 1. The network's parameters associated with those inputs are, by the inductive hypothesis, already embedded in the deep linear network.
Note that case embeds the parameters of the additional deep Fourier layer into the deep Fourier network. Case states that the parameters of the network associated with the nonlinear units of the additional deep Fourier layer already embedded in the deep Fourier network by construction.
Thus, a neural network composed of deep Fourier layers embeds a deep linear network.
18

---

Published as a conference paper at ICLR 2025
C EMPIRICAL DETAILS
All of our experiments use 10 seeds and we report the standard error of the mean in the figures. The optimiser used for all experiments was Adam, and after sweep each of the datasets over [0.005, 0.001, 0.0005), we found that a = 0.0005 was most performant.
We used the Adam optimizer (Kingma and Ba 2015 for all experiments, settling the default learning rate of 0.001 after evaluating [0.005, 0.001, 0.0005]. Results are presented with standard error of the mean, indicated by shaded regions, based on 10 random seeds.
Dataset specifications and non-stationarity conditions:
For MNIST, Fashion MNIST and EMNIST: we use a random sample of 25600 of the observations and batch size of 256 (unless otherwise indicated, such the linearly separable experiment).
For CIFARIO and CIFARII00: Full 50000 images for training, 1000 test images for validation, rest for testing. The batch size used was 250. Labelnoise non-stationarity: 60 epochs, 10 tasks. Class incremental learning: 6000 iterations per task, 80 tasks. Note that the datasets different tasks in the class incremental setting can have different sizes, and so epochs are not comparable.
tiny- ImageNet: All 100000 images training. 10000 for validation, 10000 for testing as per predetermined split. The batch size used was 250. Labelnoise experiment non-stationarity: 80 epochs per task, 10 tasks total. Class incremental learning: 10000 iterations per task, 80 tasks. Note that the datasets different tasks in the class incremental setting can have different sizes, and so epochs are not comparable.
Neural Network Architectures For ting-emageNet, CIFARIO, CIFAR100, and SVHN2: We utilized standard ResNet-18 18 with batch normalization and a standard tiny Vision Transformer. The smaller datasets use an MLP with different widths and depths, specified in the scaling section.
Experiment Metrics All figures reporting accuracy evaluate the accuracy on the distribution given by the current task. Figures 2, 3, and 7 (top) report the training accuracy on the current task. Figures the current task. Figure 5 shows the accuracy at the end of each
5 and 6 report the test accuracy epoch, whereas Figure 6 shows the accuracy at the end of each task (due to too many tasks). The accuracy reported in Figure is the final accuracy the end of the last task.
19

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
D.I ADDITIONAL DEEP LINEAR NETWORK RESULTS
Min. Sing. Val. Product Matrix (log)
10
Linear ReLU
50
100
150
Number tasks
Figure 8: Minimum singular value of the product matrix for deep general linear network on linearly-separable task.
CReLU
Fourier+ReLU
ReLU
o-ReLU 0.5)
a:sin (a 0.5)
Deep Fourier
Accuracy
0.6
0.4
0.2
80
Number of tasks
Figure 9: Average unit sign entropy non linearly-separabee task. Deep Fourier features and other perioidic activation functions have high average unit sign entropy compared to piecewise linear activations like ReLU and leaky-ReLU.
D.2 ABLATING ARCHITECTURE WITH WIDE RESIDUAL NETWORKS
Wide Resket-18 (0 5x width)
Wide ResNet-18 (2 5x width)
Wide ResNet-11 (2x didh))
3.90
Goytiiua Baciaroo
0.50
Condinuua Sacipnno
0.30
Cordinaa Ramiaanoo
Toat Aucurory
Spectra Reg
Specrra
Speetra
2.15
Toat
2.10
0.10
0.10
5.0
6900
5.09
0.00
2000
600
9903
Figure 10: Investigating Wide Residual Networks with different width scales on tiny-ImageNNe with label noise.
20

---

Published as a conference paper at ICLR 2025
D.3
INVESTIGATING SINGLE TASK PERFORMANCE
ResNet-18 with Regularization
ResNet-18 with Regularization
0.30
0.5
0.25
Train Accuracy
Accuracy
0.4
0.20
CReLU
CReLU
ReLU
ReLlJ
eaky ReLU
leaky ReLU
sin
Deep Fourier
0.05
Deep Fourler
100
150
50
100
150
Number of iterations (x100)
Number Iterations (x100)
ResNet 18 without Regularization
ResNet 18 without Regu arization
0.8
Train Accuracy
Accuracy
0.20
0.6
0.4
CRcLU ReLU eaky-ReLU
CRcLU
0.10
RaLU
leaky-ReLU
Fourier
Deep
100
150
50
100
150
Number of Iterations (x100)
Number of Iterations (x100)
Investigating single task performance of ResNet-18 using different activation functions on tiny-ImageNet.
Figure 11:
D.4
CONTINUAL IMAGENET RESULTS
1.000
0.85
0.80
Test Accuracy
Accuracy
0.995
0.990
0.70
Deep Fourier ReLu
Deep Founie
ReLli
Spectra/Rag
Spectra Reg
0.60
100
300
400
500
100
300
400
500
Number of tasks
Number of tasks
Investigating sustained performance many tasks using ResNet-18 on Continual ImageNet. Note that loss of trainability does not occur on this problem, whereas loss of generalization does occur.
Figure 12:
21

---

Published as a conference paper at ICLR 2025
D.5
COMPARING CONTINUAL BACKPROP TO RECYCLING DORMANT NEURONS
tiny-ImageNet
CIFAR100
0.40
Backprop
0.25
Deep
Test Test Accuracy
Test Accuracy
0.20
0.30
0.15
0.25
0.10
0.20
Continual Backprop
0.05
Deep Fourier
ReDO
0.15
0
10
20
30
40
50
10
15
Number of tasks
Number tasks
Recycling dormant neurons and continual backprop are both weight reinitialization methods that perform similarly.
Figure 13:
D.6 ADDITIONAL SENSITIVITY RESULTS
tiny-ImageNet
CIFAR100
CIFAR10
09
Train AcCUrSCl
5-3
8-4
0.6
12
18
0.4
Jary kcUl
Text Accuracy
Doep fourer
12
Regularization Strangth
Regularization Strength
Reguiartzation Strength
Figure 14: Sensitivity analysis on Imny-ImageNet, CIFARIO, and CIFARI00. Networks with deep Fourier features highly trainable, but have a tendency to overfit without regularization, leading to high training accuracy but low test accuracy. Due to deep Fourier features being highly trainable, they able train with much higher regularization strengths leading ultimately better generalization.
22

---

Published as a conference paper at ICLR 2025
D.7 FORGETTING RESULTS
tiny Inyaimage
CIFARIDO
CEAR10
Ontnn Accurocy (Cument Tack)
Online IntouroyyCuument (Cument Tuaki
Onfire Accuracy Task)
2.20
0.50
Deep Toureer
2200
5.13
0-0
0.20
033
0.00
0.00
20
60
80
20
40
60
Tasks
Number
Number
1.20
877
Ter: Accuraty (Prolouss Task)
Accuracy (Previous Thak)
1.30
(Proiouss Task)
1.25
0.10
0.
2.33
2.05
2.10
20
40
60
50
20
40
60
Number Tasks
Number
Number
That Accuracy (Rist Tuod)
Text Accuucay litszt
Accuracy (Fist
0.025
0.07
0.000
8.
60
60
40
20
60
20
40
60
Taslss
Number
Number
Avg. Previous Task Perfomance
0.20
Avg Prestous Perfomance
Performanc
6.35
0:33
0:20
0.5
0.13
0.23
Regularization Strength (EWC)
Regularization Strength (EWC)
Regularization Strength (EWC)
Forgetting on online label-permuted úiy-IImageNel CIFARIO, and CIFARI00. All networks capable of continual online learning within task, indicating that they maintain plasticity and succeed in avoiding catastrophic forgetting data early within a single task. However, deep Fourier features particularly capable of maintaining performance previous tasks.
Figure 15:
D.8 ADDITIONAL TRAINABILITY RESULTS USING DEEP FOURIER FEATURES
These additional experiments validate the benefits of deep Fourier features as a means of improving trainability. The experiments use the following datasets for continual supervised learning: MNIST (LeCun al. 1998), Fashion MNIST (Xiao al. 2017). and EMNIST (Cohen al. (007)). We focus primarily on the problem of trainability, and thus consider random label non-stationarity. in which the labels are randomly assigned to each observation and must be memorized on each task. This type of non-stationarity is particular difficulty in sustaining trainability in continual learning (Lyle et al. 2023 Kumar et al.) 2023b). We compare our network with deep Fourier feature against a corresponding feed-forward neural network with ReLU activations with the same depth. Because deep Fourier features use concatenation of two different activation functions, it has half the width of the ReLU network and less parameters, which provides advantage to the ReLU baseline.
23

---

Published as a conference paper at ICLR 2025
D.8.1 DEEP FOURIER FEATURES ARE HIGHLY TRAINABLE
The main result of this appendix is presented in Figure 16 Across different datasets, deep Fourier feature networks are highly trainable, either achieving high accuracy and maintaining it on easier tasks, such as MNIST, or improving their trainability on new tasks, such as on Fashion MNIST. In contrast, the ReLU network suffered from loss of trainability in each of the problems that we studied. This is not surprising, as loss of trainability is a well-documented issue for ReLU networks without some additional method designed to mitigate it (Dohare et al. 202 Lyle et al. 2022 Kumar et al. 2023b Elsayed and Mahmood 2024).
20 Epocns, MMST
20 Epocns, EMNST
20 Epochs, FashionMNSST
Acturacy
0.
Acturary
2.0
6:2
30
6.0
10
20
6.0
10
26
40
Number tasks
Number of tasks
Number of tasks
40 Epochs. MN ST
40 Epochs, EMNSST
40 Epochs, FashionMNIST
0.3
Accurecy
0.4
Accurary
6.6
Touter
0.3
0.2
20
30
10
20
600
10
20
40
Number tasts
Number of tasks
Number of tasks
Figure 16: Trainability across different datasets and epochs per tasks. ReLU networks lose their trainability, whereas networks with deep Fourier features improve and sustain their trainability
D.8.2 METHODS FOR IMPROVING TRAINABILITY
Given that ReLU network is unable to maintain its trainability in isolation, investigate whether recently proposed methods for mitigating loss of trainability are able to make up for the difference in performance between network with deep Fourier features and network with ReLU activations. We investigate two categories of mitigators for loss of plasticity: (i) regularization and (ii) normalization layers.
20 Epochs, L2 Towards Init
40 Epochs, L2 Towards Init
1.0
1.0
0.8
0.6
02
02
Deep
ReLü
0.0
0.0
10
10
10
1002
Regularization Strength
Regularization Strength
Figure 17: Hyperparameter Sensitivity Analysis. Deep Fourier features seem to not benefit from regularization for trainability. While ReLU networks are more trainable with regularization, their performance is still worse than the deep Fourier feature network. Note that the experiments in Section 5.2 indicate that deep Fourier features do benefit from regularization for generalization.
24

---

Published as a conference paper at ICLR 2025
Regularization Loss of plasticity occurs in ReIU networks when they not regularized. Thus, we compare the performance of the ReLU network and the deep Fourier feature network with varying regularization strengths. In particular, we the recently proposed L2 regularization towards the initialization (Kumar et al.) 2023b), because it addresses the issue of sensitivity towards zero common to 1.2 regularization towards zero. In Figure [7] find that regularization does improve the trainability of ReLU networks, validating previous empirical findings. However, we found that deep Fourier feature networks do not benefit substantially from regularization. That is, deep Fourier feature network with a smaller regularization strength always outperformed the ReLU network.
Layer Normalization Training deep neural networks typically involve normalization layers, either Batch Normalization (loffe and Szegedy 2015) Layer Normalization (Ba et al. 2016). Recently, it was demonstrated that layer normalization is an effective mitigator for loss trainability (Lyle et al. 2024). We investigate whether trainability can be improved with the addition of normalization layers, for both the ReLU and deep Fourier feature network. In Figure [8] we found that layer normalization increases performance but that loss of trainability can still occur with ReLU network. In addition to Layer Normalization, we also tried a linear version of LayerNorm which uses a stop-gradient on the standard deviation maintain linearity, which improved training speed in some instances.
20 Epochs. MNSTT
20 Epochs, EMNSST
20 Epochs, Fashion MN
3.5
1.4
Accuricy
0.6
0.4
6.2
0.0
10
68
Figure 18: Comparison of trainability with Layer Normalization. ReLU networks are more trainable with Layer Normalization, but deep Fourier feature networks learn faster and achieve better accuracy, particularly with linearized Layer Norm.
25

---

Published as a conference paper at ICLR 2025
D.8.3
SCALING PROPERTIES OF DEEP FOURIER FEATURE NETWORKS
Average End of Tbak Accuracy
20 Epochs, MNIST
20 Epochs, EMNIST
Toik Accuracy
20 Epochs, Fashion MNIST
Avorage End of Tack Accurocy
6-
0.4
Average
0.2
200
320
420
200
300
50c
Width
Width
Width
Average End of Taak Accuricy
20 Epochs, MNIST
Averoge End of Tock Accuracy
20 Epochs, EMNIST
Taak Accuracy
20 Epochs, Fashion MNIST
6.8
06-
0.6
0.4
Average
02
Depth
Depth
Depth
Figure 19: Scaling Neural Network Width and Depth. (Top) Due to the concatenation used by the activation function in deep Fourier feature networks, they scale particularly well with width. (Bottom) Deeper Fourire features also lead to improved average end of task performance.
Width Scaling Another source of linearity recently proposed is an increasing width of the neural network, causing their parameter dynamics evolves as linear models in the limit (Lee et al. 2019). We investigate whether an increase in width can close the gap between the trainability of the ReLU network and the deep Fourier feature network. In Figure 19 (Top), we found that deep Fourier feature networks scale particularly well with width, whereas width seems to have little effect on the trainability of ReLU networks. Thus, results suggest that increasing the width of neural network does not necessarily impact its trainability, at least not to the width values we considered.
Depth Scaling Neural networks in supervised learning tend to scale with depth, allowing them to learn more complex predictions. We investigate whether the depth scaling of deep Fourier feature networks also leads similar improvements continual learning. In Figure I (Bottom), we found that deep Fourier feature networks do improve with additional depth, but the degree of improvement was not as pronounced as scaling the width.
26


Reviews:
[{"review_id": "tGJ8RGfDSb", "reviewer": "Reviewer_rXxi", "summary": "Continuous learning involves a model learning from changing, non-stationary distributions. The model should avoid forgetting previously learned information while adapting to new incoming data. The paper first demonstrates that linear networks adapt more easily to new data than non-linear models. However, non-linearity is crucial for capturing features and achieving good performance in supervised settings. To balance linearity and non-linearity, the paper proposes using an activation function that combines cosine and sine functions, which is called deep fourier features.", "questions": "1. The theory in Section 3.1 suggests that linear networks can adapt to new data, but does not address resistance to forgetting. Could the authors provide empirical results to assess the anti-forgetting capability of their approach? This would help clarify whether the method supports both critical aspects of continuous learning.\n\n2. Can the authors explain how the Fourier activation function, defined as $Fourier(z)=[sin(z),cos(z)]$, is applied in practice? For instance, if a pre-activation vector is given as $[z_1, z_2, ..., z_n]$, how would each unit in the layer be processed? Should each pre-activation value use both sine and cosine, or is there a selection mechanism between the two? Pseudo-code or a mathematical expression would be highly helpful.\n\n3. Why did the authors choose cosine and sine for the activation function over other periodic functions or piecewise linear functions within defined intervals (e.g., periodic $y = x$ replacing $ y = sin(x)$, periodic $y = 1-|x|$ replacing $ y = cos(x)$)? A justification of this choice would provide greater insight into the effectiveness of these functions for balancing linearity and non-linearity in continuous learning.\n\nAddressing these questions could strengthen the paper’s contribution and may support a higher rating score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The proposed activation funtion may replace the ReLU and serve as the comman practice when building continuous learning model.", "weaknesses": "1. While the authors establish that linear networks can more readily adapt to new data (Sec 3.1), they do not examine whether these networks are resistant to catastrophic forgetting. Without measuring anti-forgetting properties, it’s unclear if the proposed approach truly meets both requirements of continuous learning (adaptability and retention).\n\n2. The proposed activation function, $Fourier(z)=[sin(z),cos(z)]$, lacks clear implementation details, which may hinder practical use and replication. For instance, it’s ambiguous how the function applies to each unit in a layer and whether each pre-activation value is mapped to both sine and cosine or alternates between the two.\n\n3. While using sine and cosine for the activation function is interesting, the rationale behind this choice is not fully explained. Periodic linear functions might also be a candidate for balancing linearity and non-linearity, so further justification or comparative analysis would strengthen the argument for cosine and sine.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mnTY8Z5SEs", "reviewer": "Reviewer_GT8H", "summary": "The paper studies the problem of plasticity loss in deep neural network under continual learning. The authors suggest that the loss of plasticity or trainability is caused by the nonlinearity in activation functions.  The authors prove that linear layer and a diagonal linear network dont suffer from a loss of plasticity, showing empirically that deep linear network also dont suffer from that   The authors then suggest that a perfect mix of linearity and nonlinearity  is a potential solution to the plasticity loss while maintain the expressiveness of deep neural network. \nThe authors then suggest deep Fourier features where sin and cosine activation functions are combined. The authors showed in the experiments that such activation function maintain trainability but needs regularization to prevent overfitting.", "questions": "Why loss of trainability is used, does it differ from the loss of plasticity defined in https://www.nature.com/articles/s41586-024-07711-7 and if not why using different terms?\n\nWhat is the performance compared to ReLU or LeakyReLU activations on offline setting (training on full dataset with no continual learning)?\n\n\nLn 076-079 are not quite clear\n\nCould you define unit sign entropy earlier, or make it clear when using it in the intro.\n\nln 107 is unclear.\n\nWhat is the suboptimality gap in line 144\n\nIn ln 305, what if \\alpha was trainable parameter?\n\nCould you add continual backpropagation as a baseline https://www.nature.com/articles/s41586-024-07711-7\n\nPlease add a section to describe the metrics used. \n\nIt is unclear to me why the setup of diminishing noise is considered as the first experiment.\n\nDoes the trainability still persist on longer tasks, e.g., the ImageNet sequence of   https://www.nature.com/articles/s41586-024-07711-7\n\nI am welling to increase my score upon addressing my questions, thanks!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well written. \n\nA discussion and analysis of an important problem. \n\nShowing that linear network don't suffer plasticity loss and a mix between linear and non linear can retrain trainability. \n\nTheoretical evidence and empirical evidence support the paper contribution.", "weaknesses": "While I enjoyed reading the paper and its flow, the presentation can still be improved substantially especially in the figures and equations. \nFor example, equations are not numbered which makes it hard to refer back to specific equation. Figures are not well explained, and colours are really hard to distinguish among different baselines. \n\nWhile a new activation function is introduced, a proper analysis of its behaviour and the performance of the network should be demonstrate here we only see results on continual learning and later figure out that the results were with a regularization.\n\nIt is not clear which metrics are reported, e.g., test accuracy measured by what? performance on the latest task or all tasks?\nForgetting is never discussed. \n\nOnly one network is studied, what about other architectures?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAqqomgSDB", "reviewer": "Reviewer_TwPR", "summary": "This work investigates loss of plasticity in different continual learning scenarios, which hinders networks' ability to adapt to new knowledge over time. The authors introduce Deep Fourier Features as a new method to enhance plasticity, which integrate sine and cosine functions as activation components in each layer of the network. These features concatenate sine and cosine functions in each network layer, allowing for a balance between linearity (supporting plasticity) and nonlinearity (supporting representational power). Through theoretical and empirical results, the paper demonstrates that using Deep Fourier Features can significantly improve continual learning performance across different tasks and datasets, outperforming traditional ReLU-based networks.", "questions": "plz see Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper presents a novelty method by integrating Fourier Features into deep networks, offering a new view on balancing linearity and nonlinearity to sustain plasticity.\n+ The authors provide a comprehensive theoretical basis for their Deep Fourier Feature, proving that linear function approximations and certain linear networks maintain plasticity.\n+ The experiments were conducted on benchmark datasets under various continual learning scenarios.", "weaknesses": "+ Although they compare the proposed method with ReLU-based architectures, it could include comparisons with other advanced techniques in continual learning to validate the advantages further.\n+ There is limited discussion on the computational and memory costs associated with the model compared to standard activation functions.\n+ The authors claim that the proposed method focuses on the loss of plasticity, however, there seems to be something confusing about the experiments on label noise and pixel permutations (especially label noise).\n+ The manuscript could benefit from more discussion of the overfitting problem.\n+ The manuscript could benefit from more practical insights or guidelines for implementing deep Fourier features.\n+ More continual learning methods related to plasticity should be included, like\n\n[1] Architecture matters in continual learning \n\n[2] Revisiting Neural Networks for Continual Learning: An Architectural Perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Z5OwVNE3S", "reviewer": "Reviewer_TWNH", "summary": "This manuscript targets continual learning and the phenomenon of loss of plasticity. The authors propose a theoretical analysis to solve this issue, thereby proposing deep Fourier features that employ sin and cos as activation functions.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I find that the manuscript is well-written, including motivation, problem statement, and theoretical analysis. The underlying theory is quite solid and sufficiently described with mathematical details in the appendix.\n\nThe use of sin and cos looks plausible to me. Indeed, sigmoid activation function is known to exibit vanishing gradient problem. However, when using sin and cos simultaneously, at least one of the two functions would exhibit an alive gradient. In consideration of this issue, the combination of sin and cos would provide much stable behavior compared with the use of a single one.", "weaknesses": "Nevertheless, the most critical issue for this manuscript is the small-scale experiments. Nowadays, experiments on MNIST/CIFAR/tiny-ImageNet with ResNet-18 are too weak to validate the proposed methods. They have only small scale sizes, which is quite deviated from the practical scenario. I believe that the ICLR paper should have a technical bar at least for ImageNet experiments. Note that experiments with ImageNet would require a large number of GPUs and training time (2~3 days per training). I found that you mentioned limited GPU sources, which may be inappropriate to conduct ImageNet experiments. Although my comment may be unfair for your experimental environment, conducting extensive validation with sufficient datasets is critical in our research field, so please understand my concern. Readers want credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjfHuttbun", "reviewer": "Reviewer_tK5u", "summary": "This paper studies the loss of plasticity in commonly faced by deep neural networks in continual learning settings: As the training data continues to shift, eventually most deep networks start to lose their ability to adapt to the new distributions. The authors observe empirically and provide theoretical analysis showing that deep linear networks do not suffer this same problem. However, deep linear networks do not have the same representation power as nonlinear ones, so the authors instead propose imbuing nonlinear networks with more linear characteristics through deep Fourier features. The paper goes on to explain why alternative activation functions fail to embed a similarly linear network, and empirically demonstrate how deep Fourier features are able to preserve model trainability in two continual learning settings (Diminishing Label Noise, a variant of Class-Incremental Learning), for CIFAR-10, CIFAR-100, and tiny-ImageNet.", "questions": "Q1: Line 117: “The problem setting that we consider is continual supervised learning without task boundaries.” The description that follows sounds like there are task boundaries every $T$ iterations? Is the “without” a typo?\n\nQ2: Figure 4: Any reason not to also include plain ReLU in this figure? I presume it does badly.\n\nQ3: Figure 5: Deep Fourier features have a very stark advantage over the other approaches for tiny-ImageNet, but the sizable gap disappears on less complex datasets like CIFAR 10 and 100. This happens in Figure 6 too. Why do you suspect this is?\n\nQ4: In both the experimental settings, it doesn’t seem like any sort of remembering of past tasks is necessary. For diminishing noise, the label space remains the same, and the new data distributions just contain better information; for the class-incremental setting, the new data distributions contain all past distributions as a subset. What about just re-initializing the model and training from scratch? It would be good to have this naïve baseline also plotted in Figure 5 + 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. A well-motivated, principled perspective: I’ve been reading (and reviewing) papers on continual learning for 7 years now, and most of the papers I’ve seen after the first papers that established the popular continual learning paradigms (e.g. replay, regularization, parameter isolation/expansion) have continued to fall into those buckets. Generally, many of the submissions I see are hacky and heuristics driven, consisting of various remixes of previously explored ideas. I thus found this paper to be a breath of fresh air, examining some of the problems of continual learning from a more theoretical perspective, building upon past works on plasticity, activations functions, Fourier features, and more to examine what exactly about nonlinear deep networks leads to lower plasticity for future tasks, as well as how to mitigate. I want to see more papers trying to solve continual learning from this kind of approach.\n\nS2. Insights on plasticity: Continual learning papers tend to focus on the catastrophic forgetting problem, forgetting that being able to continually learn is also part of the problem. This paper doesn’t originate the observation of the loss of trainability in nonlinear deep networks (previous works cited), so it shouldn’t be attributed to the authors, but I would like to see more activity from the continual learning on this topic. I feel this paper could make a good contribution in this sense.\n\nS3. Connections with other common activations: I appreciate the framing of other common activation types like ReLU and leaky ReLU from the perspective of this paper’s analysis (especially lines 361-369). These were helpful for understanding why these standard practices might lose trainability, and why the proposed Fourier feature approach might succeed.\n\nS4. Writing: The paper is written very clearly and are organized well. The small empirical experiments in Figures 2 and 4 very compactly provides good empirical evidence of trainability over long task sequences for linear and non-linear models. Thanks for including.", "weaknesses": "W1. What about catastrophic forgetting?/Why not re-init?: A discussion about catastrophic forgetting--the most prominent challenge of continual learning--is largely absent from the paper. To be fair, trainability/plasticity is also often absent from papers focusing on catastrophic forgetting, and it’s good to have a deeper exploration on trainability given the lesser attention to it. However, I still feel a discussion and empirical investigation of deep Fourier features response catastrophic forgetting behavior is necessary; focusing on just trainability doesn’t make much sense to me, because if trainability is a major problem, then worst case a model can just be re-initialized and retrained from scratch. See Q4.  If the problem can be solved by just re-initialization, then the contribution here is interesting but not practical. This is currently my biggest concern.\n\nW2. Strong Assumptions: While I understand that strong assumptions are commonplace in many theoretical treatments of deep learning, the restrictiveness of some of the assumptions here still are worth calling out. In particular, Theorem 2 assumes a deep diagonal linear network. Despite the cited precedents for such an assumption, this is pretty far from the reality of my deep networks in practice. How much do these insights shed here transfer to the more general deep linear network setting? \n\n\nW3. Double width vs half representation: The proposed method involves concatenating sin(z) and cos(z). This either involves double the width of the network (increases parameters and computation time) or halves the effective width (reduced model capacity). For a fixed size head-to-head comparison then, models parameterized this way may be at a sizable disadvantage. The experiments however are fair in this regard however, as they choose to incur the half effective penalty rather than unfairly double model width.\n\nW4. Page Length: The page limit of ICLR now permits papers up to 10 pages, but 9 pages is highly encouraged; the recommendation for using the tenth page is more for including larger and more detailed figures. This paper extends past 9.5 pages, but I personally do not feel the visual content justifies going over the 9-page recommendation. I would recommend the authors try to get down to 9 pages (e.g. moving more content to the Appendix), or else for fairness reasons I believe this paper should be evaluated at a higher bar.\n\nMiscellaneous:\n- Figure 3: Having a graph inset in another like is non-standard and can be a little confusing. It seems like this may be better suited as two separate subplots.\n- Linear 273: “function” => “functions”\n- Line 419: Class incremental learning definition: This is not the traditional class incremental learning setup usually shown in most continual learning papers. Notably, it seems like addressing catastrophic forgetting is completely optional, as new tasks still contain the classes from the previous tasks. I would suggest naming this something else to distinguish it from the more common (and harder) setting.\n- Line 465: extra space before footnote\n- Figure 6 (right) caption: It doesn’t appear that deep Fourier features “substantially improve over most baselines”. In fact, they seem pretty middle of the pack?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)