Atlas: Universal Function Approximator For Memory Retention
Authors: Heinrich van Deventer
Anna Sergeevna Bosman
Venue: NeurIPS 2022 Submitted (2022)
Keywords: universal function approximation
artificial neural networks
splines
catastrophic forgetting
continual learning

Abstract: Artificial neural networks (ANNs), despite their universal function approximation capability and practical success, are subject to catastrophic forgetting. Catastrophic forgetting refers to the abrupt unlearning of a previous task when a new task is learned. It is an emergent phenomenon that plagues ANNs and hinders continual learning. Existing universal function approximation theorems for ANNs guarantee function approximation ability but seldom touch on the model details and do not predict catastrophic forgetting. This paper presents a novel universal approximation theorem for multi-variable functions using only single-variable functions and exponential functions. Furthermore, we present ATLAS—a novel ANN architecture based on the exponential approximation theorem and B-splines. It is shown that ATLAS is a universal function approximator capable of memory retention and, therefore, continual learning. The memory retention of ATLAS is imperfect, with some off-target effects during continual learning, but it is well-behaved and predictable. An efficient implementation of ATLAS is provided. Experiments were conducted to evaluate both the function approximation and memory retention capabilities of ATLAS.

ATLAS: Universal Function Approximator for Memory Retention
Anonymous Author(s)
Affiliation Address email
Abstract
Artificial neural networks (ANNs), despite their universal function approximation capability and practical success, are subject catastrophic forgetting. Catastrophic forgetting refers to the abrupt unlearning of a previous task when a new task is learned. is an emergent phenomenon that plagues ANNs and hinders continual learning. Existing universal function approximation theorems for ANNs guarantee function approximation ability, but seldom touch on the model details and do not predict catastrophic forgetting. This paper presents novel universal approximation theorem for multi-variable functions using only single-variable functions and exponential functions. Furthermore, we present ATLAS novel ANN architecture based on the exponential approximation theorem and B-splines. It shown that ATLAS is a universal function approximator capable of memory retention and, therefore, continual learning. The memory retention of ATLÁS imperfect, with some off-target effects during continual learning, but it is well-behaved and predictable. An efficient implementation of ATLAS is provided. Experiments are conducted to evaluate both the function approximation and memory retention capabilities of ATLAS.
10
12
14
16
Introduction
17
18
is an emergent phenomenon where a machine learning model such artificial neural network (ANN) learns interfere with the model's performance on previously learned tasks. Catastrophic forgetting is also called catastrophic interference [19]. If ANN cannot effectively learn many tasks, it has limited
Catastrophic forgetting 7 13 23
task, and the subsequent parameter updates
20
22 utility the context continual learning 9 12]. Catastrophic forgetting is like learning to pick
23 up cup, but simultaneously forgetting how breathe. Even linear functions susceptible
24
catastrophic forgetting, as illustrated in Figure
1.25
0.75
0.75
First Tosk Second Task
1.00
Figure 1: A linear function is susceptible to catastrophic forgetting.
Submitted to Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

25
The simple example of a linear regression model being susceptible to catastrophic forgetting might be due to the won-linearity of the target function, noise, or parameter sharing across the imput. Parameter sharing is avoidable with piece-wise defined functions such as splines [27] ANNs can be explained
26
27
28 in many ways; a useful analogy to compare ANNS to very large lookup tables that store information. Removing and updating values has off-target effects throughout the table or ANN
29
30
Universal function approximation theorems are a cornerstone of machine learning, and prove that ANNs can approximate any given continuous target function [0] 15) under certain assumptions. The theorems do not specify how to find an ANN with sufficient performance for problems in practice. Gradient descent optimisation is the convention for finding/training neural networks, but other optimisation and learning procedures exist [22]. ATLAS models trained with gradient descent methods exhibit desirable properties. However, other optimisation techniques like evolutionary algorithms may not elicit the same properties.
31
32
33
34
35
36
37
This paper introduces ATLAS a novel universal function approximator based on B-splines that has some intrinsic memory retention, even in the absence of other training and regularisation techniques. ATLAS has well-behaved parameter gradients that are sparse, bounded and orthogonal between input points that are far enough from each other. The accompanying representation and universal approximation theorems are also provided.
38
39
40
41
2
Relevant Studies
42
43 It is conjectured that overlapping representations in ANNs lead to catastrophic forgetting [12
44 Catastrophic forgetting occurs when parameters necessary for one task change while training to meet the objectives of another task 20] The least desirable strategy to mitigate catastrophic forgetting
46
is retraining a model over all tasks. Regularisation techniques like elastic weight consolidation (EWC) have also been employed [4.. Data augmentation approaches such as rehearsal and pseudo-rehearsal
47
48
have also been employed 23 Other ideas from optimal control theory in combination with dynamic
49
programming have also been applied counteract catastrophic forgetting, with a cost functional similar in form to the action integral from physics and Lagrangian mechanics [16].
50
Orthogonal Gradient Descent (OGD) is training augmentation modifies the gradient updates of subsequent tasks to be orthogonal to previous tasks 6 One can describe data in terms of distribution defined over the input space, target values, and time (the order of data or tasks that are presented during training). OGD attempts to make gradient updates orthogonal to each other over time. ATLAS, in contrast, possesses distal orthogonality, meaning that two inputs are far enough from each other in the input space, then corresponding gradient updates will be orthogonal. A corollary of this is that if the data distribution between tasks shifts in the input space, then the subsequent gradient updates will tend to be orthogonal. ATLAS does not use external 59 memory like OGD. Extensions of OGD include PCA-OGD, which compresses gradient updates into principal components to reduce memory requirements (4. The Neural Tangent Kernel (NTK) overlap matrices, as discussed by Doan al. (4) could be useful tool for analysing ATLAS models.
optimisation technique that
52
54
56
57
58
80
61
62
The survey by Delange et al. 3 gives an extensive overview of continual learning to address catastrophic forgetting. ATLAS model that implements parameter isolation, because of its use of piece-wise defined splines. Particularly relevant to ATLAS is the work on scale of initialisation and extreme memorisation 21I. Increasing the density of basis functions in ATLAS can lead to better memorisation, and increases the scale of some parameters in ATLAS which may affect generalisation.
64
66
67
Pi-sigma neural networks use nodes that compute products instead of sums 26 Pi-sigma neural networks have some similarities with the global structure of ATLAS. B-splines, which form the basis of ATLAS, have been applied for machine learning (5). Scardapane et al. (25 investigated trainable activation functions parameterised by splines. Uniform cubic B-splines have basis functions that are translates of one another (2). Uniform cubic B-splines have been tested for memory retention, and
68
70
71
72
ATLAS is an improvement on existing spline models (7).
73 B B-splines, and by extension ATLAS, can be trained to fit lower frequency components, expanded and
74 trained again until a network is found with sufficient accuracy and generalisation, similar to other techniques 17 18 It is not necessary to expand the capacity of an ATLAS model to learn new
76 tasks, as with some other approaches (24). ATLAS does in practice demonstrate something akin to "graceful forgetting" as discussed in Golkar et al. [8].

---

78
3
Notation
Vector quantities like x are clearly indicated with bar arrow for legibility. Parameters, inputs, 80 functions etc. without a bar or arrow are scalar quantities like S(z). Some scalar quantities with indices the scalar components of vector like x. scalar parameters in the model like 8,. The 81
gradient operator that acts on a scalar function like xxx(x) yields vector-valued function V (xA(x) is typical of multi-variable calculus.
82
Exponential Representation Theorem
84
Any continuous multi-variable function compact space be uniformly approximated with multi variable polynomials by the Stone- Weierstrass Theorem. Let I denote an index set of tuples of natural numbers including zero such that ij No for all N with (n) E I and as E IR. Multi-variable polynomials can be represented as:
86
88
agay 22 xx = y(I1,,,, = Σ iEI
....
Eailii-a
iEr
89
Each monomial term 0, -; 2 is a product of single-variable functions each variable. It is desirable to rewrite products as sums using exponentials and logarithms.
91 Lemma 1. For any a₂ IR, there exists 7/8 > and 3, O, such that: a, -
Theorem 1 (Exponential representation theorem). Any multi-variable polynomial function y(X)
92
o variables over the positive orthant, can be exactly represented by continuous single-variable in the form:
94
functions gi.z (Zz) and ha (zz)
v(x) Σ exp exp(Ex) En
1915) exp(E)
95 Proof.
Consider any monomial term a, a;'1i=1a)) with a; I, then by Lemma there exist strictly positive numbers > 0 and B, > O, such that:
0: II 1 - B.I1-1133
exxp log (1;11 exp (oog (3,1)
exxp log(1)) + Σj=1 log ) exp
exp (log(B)) Σz log
x3)
97 The argument each exponential function is sum of single-variable functions and constants. 98 Without loss of generality, a set of single-variable functions can be defined such that:
o. -ル = exp(2)=n
39 Since this holds for any a; 17 and E I, follows that:
xxx) Σ exp 19., (x,)))
100
This result is fundamental the paper. Since every continuous function can be approximated with polynomials, it follows that every continuous function can be approximated with positive and negative exponential functions. Single-variable function approximators are pivotal and must be reconsidered. Universal function approximation can also be proven with the sub-algebra formulation of the Stone- Weierstrass theorem, but it's not delightful and simple as the first constructive proof given above.
102
105
105

---

Single-Variable Function Approximation
107
108
Splines are piece-wise defined single-variable functions over some interval. Each sub-interval of a spline is most often locally given by low degree polynomial, even though the global structure
109
110 a low degree polynomial. B-splines are polynomial splines that are defined in a way that resembles
other basis function formulations [2]. Each single-variable function in ATLAS is approximated
112
with uniform cubic B-spline basis functions, shown in Figure 2 B-splines can approximate any
113
single-variable function, similar using the Fourier basis. With uniform B-splines, each basis
114
function is scaled so that the unit interval is uniformly partitioned, as in Figure
60
0.3
(a) p
(c) p
Figure 2: If uniformly spaced B- B-splines used, then each basis function has the same shape. This makes possible to use the same activation function by scaling and translating the inputs. This is also true for different densities of uniform cubic B-splines.
115 The activation function to implement B-splines is given by:
0<r<1
3(x - 1)33 +3r 1) 1)
1<r<2
S(x)
3(x -2)3 -6(x 220--2)-22 +4]
2<r<3
6 (4 3²
35r<4
0
otherwise
116
The choice was made to use uniform cubic B-splines due to their excellent performance and robustness to catastrophic forgetting, illustrated in Figure 3 Using uniform B-splines instead of arbitrary sub- interval partitions (also called knots in literature) makes optimisation easier. Optimising partitions is non-linear, but optimising only coefficient (also called control points) is linear and thus convex.
117
118
119
0.9
Inactive Basi Function
0.9
iractive Basis Function
Artive Basis Function
Act Rosis Function
0.6
0.6
0.3
0.3
0.2
0.2
0.0
-0.5
0.0
0.5
1.0
1.5
-0.20 -0.05
0.10
0.25
0.00
0.55
0.70
0.85
1.00
3.15
(a) The activation is sparse and bounded.
(b) Distal orthogonality.
Figure 3: Single-variable function
Each basis function is multiplied by parameter and summed together: The total number of basis functions typically fixed. Cubic B-splines are 3rd order polynomials, and thus require minimum o 1 control points or basis functions.
121

---

123
Instead of considering arbitrary densities of uniform cubic B-splines, we look at powers of two times the minimum number of basis functions, called p-density B-spline functions.
124
125
Definition (p-density B-spline function). A p-density B-spline function is uniform cubic B-spline
function with 2 basis functions:
126
2042
20+2
20+2
f(x) = Σ (2S;(z)= = Σ 0,8(wxx +b))
Σ 0, -3)x +4
1=1
127
Consider the problem of expanding a single-variable function approximator with more basis functions to increase its expressive power. Using the Fourier basis makes it trivially easy by adding higher frequency sines and cosines with coefficients initialised to zero. It trickier to achieve something similar with uniform cubic B-splines. There are algorithms for creating
129
splines from existing splines with knot insertion, but the intermediate steps result non-uniform knots and splines. A simple and practical compromise that we propose is to use mixtures of different p-density B-spline functions, as illustrated in Figure
131
132
133
134
Definition 2 (mixed-density B-spline functionn. A mixed-density B-spline function is a single- variable function approximator that is obtained by summing together different p-density B-spline functions. Only the maximum p-density B-spline function has trainable parameters, the others are constant. Mixed-density B- B-spline functions are of the form:
136
20+2
f(z) Σ> Sp.(p.))) P=
Only the maximum p-density B-spline has trainable coefficients. All lower density p- density B-spline have frozen and constant coefficients. The maximum r = p-density B-spline has trainable coefficients with gradient updates that are orthogonal if the distance between two inputs is large enough.
139
141
142
Similar to increasing the expressiveness of a Fourier basis function approximator by adding higher frequency terms, one can add larger density cubic B-spline functions. Analytically, we can initialise all the new scalar parameters 0,+1, O, Vi N such that:
143
2212
2212
f(x)
Σ P=0
0 Sp.Sp.((z)
= Σ5>0 Sp.
145 is therefore possible create minimal model with r 0 initialised zero, and train the model until convergence. Then one can create a new model with r = 1. by subsuming the previous model's parameters, and train this more expressive model until convergence. This process of training and expansion can be continued indefinitely, and is shown in Figure
146
147
148
Dora
Data
Data
400
-0.
Figure 4: Doubling densities of basis functions before and after training.
6
ATLAS
149
ATLAS is named for carrying the burden of all it must remember, after the Titan god Atlas in Greek mythology who was tasked with holding the weight of the world. ATLAS is also an acronym for AddiTive exponentiaL Additive Splines.
151
5

---

153 Definition 3 (ATLAS), ATLAS is a function approximator of n variables, with mixed-densig
154
B-spline functions/ fy fz(z)), (T, and (ı,5(z;) in the form:
" A(x) = Σ f(0,)) 2=1
M Σ
exp exp(=j (z,)) exp exp
k2
155
ATLAS is equivalently given by the compact notation:
M Σ
A(*) F(X)
exp(Gx(X))
122 exp(Hx(()
12
The absolutely convergent series of scale factors h was chosen for numerical stability and to ensure the model is absolutely convergent. Another feature is that the series scale factors also breaks the symmetry that would otherwise exist if all mixed-density B-spline functions were initialised zero. Initialising all the parameters to be zero is a departure from the conventional approach of random initialisation. The number of exponential terms can be increased without changing the output of the model. We can choose to initialise Gm+1 (x) = 0 and H M+ (X) = 0, such that the model capacity can be increased at will.
157
158
159
160
161
162
163
ATLAS is a universal function approximator with some inherent memory retention. possesses three properties atypical of most universal function approximators:
165
1. The activity within ATLAS is sparse most neural units are zero and inactive.
2. The gradient vector with respect trainable parameters is bounded regardless of the size and capacity of the model, so training is numerically stable for many possible training peperpparameters
167
168
169
3. Inputs that are sufficiently far from each other have orthogonal representations.
170
The proofs of the three properties follows from the single-variable case, the assumption of bounded single-variable functions and parameters, and the absolutely convergent k scale factors.
171
172
Property (Sparsity). For any ((A) C Re and bounded trainable parameters Og with index
173
sei O the gradient vector of trainable parameters (for ATLAS) is sparse:
V f8,433)
= - dHamming
(A 00;
x))00
< 4n(2 + 1)
iEe
174 Remark. For a fixed number variables n, the model has a total of n22 + 1) trainable parameters. The gradient vector has maximum of 4n (2M+1) non-zero entries, which is independent of r. Recall that only the maximum density (p = r) cubic B-spline function has trainable parameters. The fraction of trainable basis functions that are active is at most 2-'. Sparsity entails efficient implementation, and suggests possible memory retention and robustness to catastrophic forgetting.
176
178
179
Property 2 (Gradient flow attenuation), For any X E D(A) C. R" and bounded trainable parameters O with index set θ: if all the mixed-density B-spline functions are bounded, then the gradient vector of trainable parameters for ATLAS is bounded:
180
181
5 A(x)
=
Σ
0A a0,
x
<
ic
Remark. For fixed number of variables n, the model has total of n2 + 1) trainable parameters. The factor of - inside the expression for ATLAS necessary to ensure the sum is convergent in the limit of infinitely many exponential terms M - co. Only the maximum density
182
183
184
185
(p- = r) cubic B-spline function has trainable parameters, so that the gradient vector is bounded in the limit of arbitrarily large densities + co. Smaller densities cannot be trainable, otherwise this property does not hold. The bounded gradient vector implies that ATLAS numerically stable during training. regardless of its size parameter count.
187

---

189
Property 3 (Distal orthogonality). For any x, E D(A) C R and bounded trainable parameters 8, for an ATLAS model A(x):
190
min (1 >
(Xx, (
==
191
Remark. Two points that sufficiently differ in each input variable have orthogonal parameter gradients.
192
Distal orthogonality means ATLAS reasonably robust to catastrophic forgetting, without other regularisation and training techniques. However, memory retention can still potentially be improved when used in conjunction with other techniques.
193
194
195
ATLAS can be implemented with ID convolution, reshaping, embedding, multiplication and dense layers. The same basis functions have to be computed for each input variable, hence ID convolutions. By correctly scaling, shifting, and rounding inputs one can compute only the non-zero basis functions with embedding layers. The number of basis functions are chosen from powers of two for convenience, with the maximum density B-spline function having exactly À = 4 x 2" basis functions. Summing over all densities the total number of all basis functions in each input variable is at most 2X, because
198
197
198
199
200
201
a geometric series was used. For every output dimension p. there are 2M exponentials. Each
exponential has n single variable functions, with at most 2 cubic B-spline basis functions each.
203
ATLAS models have time complexity OpMM log A), and O(pMnd) space complexity.
Methodology
204
The 1-.2- and 8-dimensional models were considered for evaluation, in combination with a chosen width for the update region in Task 2 from 0.1 to 0.9 in 0.1 increments. 30 trials were performed for each combination of model dimension and update region width. Mean Absolute Error (MAF) loss function, the Adam optimiser, and mini batch sizes of 100 are used throughout all experiments.
206
207
208
209
At the beginning of each trial (for given dimension and update region width) random learning rate was sampled uniformly between 10 and 0.01 + 10-6 A random noise level was sampled from an exponential distribution with scale parameter equal to one. The Task I target function is constructed from 1000 Euclidean radial basis functions (RBFs) with locations chosen uniformly over the entire input domain, with RBF scale parameters sampled independently from an exponential distribution (scale parameter equal to 10). The weights of each radial basis function are sampled from a normal distribution with mean zero and standard deviation equal to one. The Task 2 target function is exactly the same as the Task target function - except for square-like region with width equal to update region width. The location of the update region is chosen uniformly random, and such that it is completely inside the domain of the model. The updated region masks the Task target function and instead replaces the values inside it with another function that is sampled from the same distribution
210
212
214
216
218
219
220
as the Task target function, but independently from the Task target function.
After the generation of the target functions 10000 data points are sampled for training, validation, and 222
test sets for Task 1 and Task 2. To simulate the effect of learning unrelated tasks, the training data for Task 2 is only sampled from update region- with no training data outside of it being presented again, by contrast the validation and test sets for Task 2 were sampled over the entire input domain. Gaussian noise with standard deviation equal the randomly chosen noise level is added to all training data. An ATLAS model (M- = 10 positive and M = 10 negative exponential functions, maximum basis 4) with guaranteed distal orthogonality is trained and evaluated on Task 1 and Task 2. Then a modified ATLAS model (M = 10 positive and M = 10 negative exponential functions, maximum basis function density r trainable lower density basis functions) without
224
225
226
function density
228
229
230
guaranteed distal orthogonality is trained and evaluated on Task I and Task 2 using the same data sets as previously mentioned model. The final test errors for Task 2 are presented. A randomly selected trial of the 2-dimensional case is shown for visual inspection. The experiments presented in the main body of the paper were performed on Google Colab and the relevant code is provided.
231
232
233
234
8
Results
As shown in Figure the effect of distal orthogonality is clear and crisp boundaries that limit the effect of Task 2 on the memory of Task 1. Without distal orthogonality there are more off-target effects that can be visualised.
296
7

---

1st Target
1st Output
2nd Target
2nd Output
Offilagget
10
10
0
10
10
(a) Guaranteed distal orthogonality, Off off-ttrget effects deviate from Task target.
1st Target
1st Output
2nd Target
2nd Output
OffTarget
-10
(b) No guaranteed distal orthogonality, Ofr-tt effects deviate from Task 2 target.
Figure 5: A randomly chosen trial is presented for visual inspection.
1-Dimensieral
Dimensionsl
Dimmensional
Toak Ve voetiion MOKE
Traningg
Epochs Training
Figure Distal orthogonality guaranteed: All validation MAE curves for Task 1.
J-Dimensional
2-Bimensionsl
Bellmannioon
Took 2 Veliciation MAC
Epochs of Traiiing
Epochs of Training
Epochs of Training
Figure 7: No distal orthogonality: Task 2 validation MAE with update region width 8 0.1.
1-Dimensional
2-Dimensional
& Disionaall
Took Valdaation MAN
Epochs Traning
Epochs
Epochs Training
Figure 8: Distal orthogonality guaranteed: Task validation MAE with update region width 0.1.
The effect of distal orthogonality the averaged MAE for various trials for 1-,2- and 8-dimensional problems are presented as scatter plots of the averaged MAE over 30 trials for different update region widths as shown in Figure The expected tfr-target enror depends the dimension of the problem and the width of the updated regions.
239
241
Analytical results to the expected off-target error require simplification, but reasonable assumption the absence of other evidence is that each input dimension has equal contribution on the unit hyper-cube. Assume for fixed input dimension and some region of width 0 < 8 < 1 where the
243

---

Distal Orthogonaity
Distal Orthegonality
Distal orthegenality Ne Distal Orthoponality
No Distal Orthogonality
No Distal orthogorality
Was
(a) The 1-dimensional model.
(b) The 2-dimensional model.
(c) The 8-dimensional model.
Figure 9: The effect of distal orthogonality on the final test error on task 2 for the 1-22- and 8- dimensional input.
245 target function is changed such that |Y| 1 is one larger than it was originally. The expected
off off- target error depends on i the number of input variables inside the updated region: Eg n- To correctly account for all permutations with the same magnitude of change:
246
11 01 8)
p(ex)
248
One can calculate expected change values:
E ELP(EN) Rd
n2
k
8" 8 (1 8)*h
n
249
However if one assumes that the target function inside the updated region of width ö is correct, with probability 8" of sampling from the entire input-domain, then the expected off-target error should be:
250
Expected off target error = 8 8
251
This seems consistent with some of the experimental results, but further investigation is needed.
Conclusion
252
The main contribution of the paper is theoretical and technical. A representation theorem is presented that outlines how to approximate multi-variable functions with single-variable functions (splines and exponential functions). ATLAS approximates all arbitrary single-variable functions with mixtures of B-spline functions. ATLAS is constructed in such way that the gradient vector with respect to trainalle parameters is bounded, regardless of how large an ATLAS model is. The activation of units in ATLAS is sparse, and allowed for an efficient implementation that only computes non-zero activation values with the aid of embedding layers. The gradient update vector with respect to trainable parameters is orthogonal for different inputs as long as the inputs are sufficiently different from each other.
254
255
256
257
258
259
251
262
For every output dimension p in an ATLAS model, there are 2M exponentials. Fach exponential has single variable functions, with at most 2 cubic B-spline basis functions each. ATLAS models have time complexity O(pMn log A), and (AM space complexity.
263
254
ATLAS was shown to exhibit some memory retention, without the assistance of other techniques. This is a good indication of the potential for combining it with other techniques and models for continual learning. The chosen experiments demonstrated the theoretically derived predictions and contrasted two models, incuding a variant of ATLAS without distal orthogonality guarantees.
266
288
As far societal impacts are concerned: It is possible that ATLAS could allow for the creation of more powerful machine learning algorithms, that require less resources to train and deploy. Further testing is needed to make any concrete claim.
270

---

272 References
273 [1] M. A. Bennani, T. Doan, and M. Sugiyama. Generalisation guarantees for continual learn- ing with orthogonal gradient descent, 2021. URL attp::://oeemeevievv.tettttrrrre hecuSLbL_ vC
275
[2] K. Branson. A practical review of uniform b-splines, 2004.
277
[3]
M. Delange, R. Aljundi, M. Masana, S. Parisot. X. Jia, Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2021. doi: 10.1 tpami.2021.00 200.544444. UtLpttts:://oot org/ 1109%22tppmmmmmmmmmmmm 2021. 3057446
278
279
280
[4]
T. Doan, M. Abbana Bennani, B. Mazoure, G. Rabusseau, and P. Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In A. Banerjee and K. Fukumizu, edi- tors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1072-1080, PMLR, 13-15 Apr 2021. URL http::tttpppppt httpps://proceedingg. mlr. www.ssprea///eee sitas/vvv3/// h13///1110//0//// htmll
282
283
284
285
286
[5] A. S. Douzette, B-splines in machine learning. Master's thesis, Department of Mathematics, University of Oslo, 2017.
288
66] M. Farajtahar, N. Azizan, A. Mott, and A. Li. Orthogonal gradient descent for continual learning. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3762-3773, PMLR, 26-28 Aug 2020. Utthhttss:/pppooeeeeee
289
290
292
mlr. preesss/v100//faaaaataaaa222 html.
293
[7]
R. M. French. Catastrophic f'orgetiing in connectionist networks. Trends in cognitive sciences, 3(4::228 135, 1999.
294
295
88) S. Golkar, M. Kagan, and K. Cho. Continual learning via neural pruning, 2019. URL https 04476
7/arxiv.
297
[9] R. Hadsell, D. Rao, A. A. Rusu, and R. Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-1440 2020.
299
[10] B. Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7(10)9992, 2019.
300
[11] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366, 1989.
302
[12] P. Kaushik, A. Gain, A. Kortylewski, and A. Yuille. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. CoRR, abs/2102. 11343, 2021. URL aattps://arx:t
304
org/ abs///as..22../ 11343.
306
[13] R. Kemker, M. McClure, A. Abitino, Hayes, and C. Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
307
308
309
[14] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska- Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 2017. ISSN 0027-8424. doi: 10.1073/pnas. 1611835114. URL https:ts:: pnas. org/ wwwotoneetntttnoe/eooooo
310
311
312
313
[15] A. Kratsios. The universal approximation property: Characterization, construction, representa- tion, and existence. Annals of Mathematics and Artificial Intelligence, 89(5):435-469, 2021. doi:
315
316
317
[16] R. Krishnan and P. Balaprakash. Meta continual learning via dynamic programming, 2020.
URL htthstt::: : /ttp/v..ivv../////a//eee//a.all/ org/ abo///bb////.../////// 02219
10

---

319
[17] S. H. Lane. M. Flax, D. Handelman, and J. Gelfand. Multi-layer perceptrons with b-spline receptive field functions. In Advances in Neural Information Processing Systems, pages 684 692, 1991.
320
322
[18] A. C. Li and D. Pathak. Functional reguuaiizaaon for reinfo reinforcemeet learning via learned fourier features, 2021. URL attepes//arxiiv.orrr/bbb/2122222 03257
323
324
[19] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165, Elsevier, 1989.
326
327
[20]
K. McRac and P. A. Hetherington. Catastrophic interference is eliminated in pretrained networks. In Proceedings of the 15h Annual Conference of the Cognitive Science Society, pages 723-728, 1993.
329
330
[21] H. Mchta, A. Cutkosky, and B. Neyshabur. Extreme memorization via scale of initialization, 2020. Utthttss:://rrtt org/ aa//////////ae /000 1000.13363.
[22] A. Meulemans, F. Carzaniga, J. Suykens, J. Sacramento, and B. F. Grewe. A theoretical framework for target propagation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and
333
334
H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20024 20036. Curran Associates, Inc., 2020. URL httpt::::///poocceiigg..oo newriss meuriip.cc/ppper 2020/file/e7 (0lo///eeeeett0c0000666669333333
335
336
pdf
337
[23]
A. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
338
339
[24] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu,
340
R. Pascanu, and R. Hadsell. Progressive neural networks, 2016. https:///aax org/
abs/1606. 04671
342
[25] S. Scardapane, M. Scarpiniti, D. Comminiello, and A. Uncini. Learning activation functions from data using cubic spline interpolation. In Italian Workshop on Neural Nets, pages 73-83. Springer, 2017.
343
344
[26] Y. Shin and J. Ghosh. The pi-sigma network: an efficient higher-order neural network for pattern classification and function approximation. on Neural Networks, volume i, pages 13-18 vol.1, 1991. doi: 10. 10911IINNN.199911555111
346
SEONN-9/-Seattie International Joint Conference
347
[27] H. van Deventer, P. J. van Rensburg, and A. Bosman. Kasam: Spline additive models for function approximation, 2022. URL http:::ttts///iiiiee org/ abs/2205.06377
349
Checklist
351
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
353
contributions and scope? [Yes]
354
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
357
358
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
360
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
382
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
363
364
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
11

---

386
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
367
368
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [TODO]
370
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
371
(a) If your work uses existing assets, did you cite the creators? [TODO]
372
(b) Did you mention the license of the assets? [TODO]
(c) Did you include any new assets either in the supplemental material URL? [TODO]
374
375
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
377
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
379
5.
If you used crowdsourcing or conducted research with human subjects...
380
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
382
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
383
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
385
12


Reviews:
[{"review_id": "n4tpz_Hp8uC", "reviewer": "Reviewer_LHt4", "summary": "In this work, the authors propose a theoretical model that is generally well-suited for continual learning and exhibits less forgetting. In particular, they present a novel universal approximator theorem with multi-variable functions using single-variable functions and also exponential functions. They conduct a set of experiments to back their theoretical findings.", "questions": "*Authors should make a comparison with [1-3] and show how current theoretical results proposed in this manuscript compares and also comment on the generalization guarantee of the theorem. \n\n* Line 27, the authors have stated that ANNs have issues such as vanishing and exploding gradients, whereas this is not true. This is not an issue with ANNs but learning approach which is backpropagation of error (BP or BPTT), recently people have investigated local learning approaches such as Local representation alignment (LRA) and Difference target propagation (DTP and DTP-sigma), where they show models can be trained from zero weight initialization. Finally, ANNs by design are not well suited for continual learning, but additional memory (Gradient episodic memory, mnemonics), regularization-based approaches such as elastic weight consolidation (EWC), and sparsity helps them handle these tasks. I would advise the authors to revise the intro and avoid any partially correct claims.\nAuthors have reported they use test splits as the validation set to evaluate model performance on the test set. The reason we have a validation set is to approximate a distribution that we might see in the future, therefore we try all permutations and combinations to extract the best model. A common practice is to test your model once on test splits. Constantly changing hyper-parameters by checking performance on the test set is invalid. Hence I would request authors redo experiments with separate splits. – Since using a 0.01 learning rate with Adam is an odd choice. Additionally, there is no hyperparameter optimization and results with K-trials. One should conduct results across trials and report average performance with standard error. Hence I am not convinced by the experiments\n\n* A second ablation study with various noise levels is needed. How does model performance change when the standard deviation for Gaussian noise is varied from 0.1 to some other constant?\n\n* For task two why are training sets sampled over a domain [0.45-0.55], what are these magic numbers, please provide reasoning on how should one derive such ranges?\n\n* Task 1 is trained for 30 epochs, whereas Task 2 is trained for 6 epochs, why is this and how did you arrive to such conclusions, given there are no stopping criteria and using only 6 epochs is an odd choice.\n\n* I liked the theoretical finding on the sparsity of atlas which states trainable parameters for the atlas is sparse, this would help in understanding why models such as Hard attention to task (HAT) and Sparse neural coding network (SNCN) work well on continual learning benchmarks without any memory component. What is the theoretical bound or upper bound for the total number of neurons required that will guarantee generalization and less forgetting? Can the authors comment on this?\n\n* [1] https://proceedings.mlr.press/v130/doan21a.html\n* [2] https://arxiv.org/abs/2008.02219\n* [3] https://openreview.net/forum?id=hecuSLbL_vC\n", "limitations": "Highlighted above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* Approach and theorems are novel.\n* Paper is well-written with fixable errors\n* Results are okay\n\n# Weakness\n* Ablation study is missing\n* No baseline models to compare model performance\n* Experiment section is missing few details", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zw8sHISLdYm", "reviewer": "Reviewer_qV8i", "summary": "This paper introduces Atlas, a NN architecture based on mixed-density B-spline functions. These function satisfy a universal approximation theorem (proved in the paper), and have several useful properties, such as sparse and bounded gradients. The goal of this paper is to use such networks for continual learning problems.", "questions": "- What is meant by robust, and how does Atlas achieve such robustness while other NN models do not? Is there a quantitative metric associated to this robustness?\n\n- How does Atlas compare with baseline models and continual learning methods, e.g. in terms of new task performance, memory (since Atlas is an expanding model)? In general, as similar model expansion and weight fixing ideas have been implemented in other NNs, it is not clear how one may benefit from Atlas. Are there problems Atlas can solve, but other NNs can not solve?\n\n- Can Atlas be applied to some of the benchmark datasets for CL, for example permuted MNIST is fairly common, but more challeneging benchmarks also exist.\n\n- How is information shared across tasks in Atlas? When the model is expanded, does the model rely heavily on previously learned information, or simply learn new tasks from scratch in the expanded parts of the model? How does the expanded learning on subsequent tasks compare to learning that new tasks from scratch?\n", "limitations": "Yes, some limitations are briefly mentioned in the conclusions. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The examination of this class of functions in interesting, and the authors explain these models and their properties clearly\n\n- Several interesting properties of these functions are clearly proved in the text\n\nWeaknesses:\n- ATLAS is discussed as having memory retention, but this seems to come mainly from training one B-spline at a time and fixing previous splines after training. As such the memory retention appears to come from fixing model parameters, rather than something intrinsic to the models. Similar ideas were used in other continual learning papers, e.g. continual learning via neural pruning [1903.04476]\n\n- B-splines are discussed as being \"robust\", but what is meant by robust is not clearly defined. If this refers to memory retention, then as stated in the previous comment, this memory retention seems to come from parameter fixing during training. Why would one need B-splines in this case?\n\n- To address continual learning, e.g. sequentially learning tasks as is discussed in experiments, the ability to learn new tasks seems to come from expanding the model. Is this correct? This can also be done with standard NN models, e.g. as in Progressive Neural Networks [1606.04671]\n\n- The experiments are only examined on 2D problems, but there are several reasonably standard benchmarks for sequential task learning. Moreover, no comparison to baseline models is shown.  This would seem especially relevant, as the Atlas models do not seem to robustly fit several tasks.\n\n- There is no discussion or examination of information sharing across tasks, e.g. ideas such as forward or backward utilization or transfer of information across tasks. This would be helpful in understanding how the model could robustly use previously learned information across tasks.\n\n- the related work section does not discuss the breadth of work on continual learning, including work on models with similar ideas implemented (such as the two mentioned above)\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BU6haAAgtlx", "reviewer": "Reviewer_9ayK", "summary": "This work presents Atlas, a new universal function approximator based on B-splines. Theoretically, Atlas has nice properties, such as sparse gradients, bounded gradients, and distal orthogonality. Experimentally, Atlas is robust to catastrophic forgetting, indicating its potential to be applied in continual learning.", "questions": "In line 90, it is mentioned that \"The minimum number of cubic B-spline basis functions is four.\" In line 140, it is written that \"It is worth recalling that at most four basis functions are active for uniform cubic B-spline functions.\" Is there a contradiction here? Why four basis functions?", "limitations": "Some limitations are already listed in the Conclusion. Given that there is still space in the main paper, the authors are encouraged to finish some of them.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As far as I know, the approach presented in this work is novel. This new function approximator has good theoretical properties with sound proofs. The experiments also show its potential in reducing catastrophic forgetting and continual learning.\n\nHowever, this work can also be improved in many ways.\n\nFirst, related works are not adequately cited. In continual learning, more strategies to mitigate catastrophic forgetting can be found in this [survey](https://arxiv.org/abs/1909.08383). Some methods with dynamic architectures should be cited and compared.\nA short introduction of B-splines would be appreciated, given that B-splines form the basis of Atlas. As for memory retention, there are also some related works in deep learning, such as https://arxiv.org/abs/2008.13363 and http://arxiv.org/abs/2112.03257.\n\nThe notations used in the paper are not well-defined or explained. For example, in Definition 1, what are function $S_i(x)$ and $S(x)$? What is a uniform cubic B-spline function? Is $\\theta_i$ a vector or a real value? There are similar issues in Definition 3 and Property 1-3. Providing proof sketches in the main paper for Theorem 2 and Property 1-3 would be much appreciated.\n\nThere is a lack of comparison of Atlas with traditional artificial neural networks for approximating the swiss-roll target function.\nIt will also be better to see the results of testing Altas in the classical continual learning setting on some benchmarks, such as MNIST.\nMoreover, the experimental results are not well-explained. In Figure 6, why does the output difference have a cross shape?\nI would also like to know the exact number of parameters used in the experiments for Atlas.\n\nOverall, this work is interesting but more like a work in progress. And it can be improved significantly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject