Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors propose a theoretical model that is generally well-suited for continual learning and exhibits less forgetting. In particular, they present a novel universal approximator theorem with multi-variable functions using single-variable functions and also exponential functions. They conduct a set of experiments to back their theoretical findings.
strengths_and_weaknesses: # Strengths
* Approach and theorems are novel.
* Paper is well-written with fixable errors
* Results are okay

# Weakness
* Ablation study is missing
* No baseline models to compare model performance
* Experiment section is missing few details
questions: *Authors should make a comparison with [1-3] and show how current theoretical results proposed in this manuscript compares and also comment on the generalization guarantee of the theorem. 

* Line 27, the authors have stated that ANNs have issues such as vanishing and exploding gradients, whereas this is not true. This is not an issue with ANNs but learning approach which is backpropagation of error (BP or BPTT), recently people have investigated local learning approaches such as Local representation alignment (LRA) and Difference target propagation (DTP and DTP-sigma), where they show models can be trained from zero weight initialization. Finally, ANNs by design are not well suited for continual learning, but additional memory (Gradient episodic memory, mnemonics), regularization-based approaches such as elastic weight consolidation (EWC), and sparsity helps them handle these tasks. I would advise the authors to revise the intro and avoid any partially correct claims.
Authors have reported they use test splits as the validation set to evaluate model performance on the test set. The reason we have a validation set is to approximate a distribution that we might see in the future, therefore we try all permutations and combinations to extract the best model. A common practice is to test your model once on test splits. Constantly changing hyper-parameters by checking performance on the test set is invalid. Hence I would request authors redo experiments with separate splits. – Since using a 0.01 learning rate with Adam is an odd choice. Additionally, there is no hyperparameter optimization and results with K-trials. One should conduct results across trials and report average performance with standard error. Hence I am not convinced by the experiments

* A second ablation study with various noise levels is needed. How does model performance change when the standard deviation for Gaussian noise is varied from 0.1 to some other constant?

* For task two why are training sets sampled over a domain [0.45-0.55], what are these magic numbers, please provide reasoning on how should one derive such ranges?

* Task 1 is trained for 30 epochs, whereas Task 2 is trained for 6 epochs, why is this and how did you arrive to such conclusions, given there are no stopping criteria and using only 6 epochs is an odd choice.

* I liked the theoretical finding on the sparsity of atlas which states trainable parameters for the atlas is sparse, this would help in understanding why models such as Hard attention to task (HAT) and Sparse neural coding network (SNCN) work well on continual learning benchmarks without any memory component. What is the theoretical bound or upper bound for the total number of neurons required that will guarantee generalization and less forgetting? Can the authors comment on this?

* [1] https://proceedings.mlr.press/v130/doan21a.html
* [2] https://arxiv.org/abs/2008.02219
* [3] https://openreview.net/forum?id=hecuSLbL_vC

limitations: Highlighted above
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper introduces Atlas, a NN architecture based on mixed-density B-spline functions. These function satisfy a universal approximation theorem (proved in the paper), and have several useful properties, such as sparse and bounded gradients. The goal of this paper is to use such networks for continual learning problems.
strengths_and_weaknesses: Strengths:
- The examination of this class of functions in interesting, and the authors explain these models and their properties clearly

- Several interesting properties of these functions are clearly proved in the text

Weaknesses:
- ATLAS is discussed as having memory retention, but this seems to come mainly from training one B-spline at a time and fixing previous splines after training. As such the memory retention appears to come from fixing model parameters, rather than something intrinsic to the models. Similar ideas were used in other continual learning papers, e.g. continual learning via neural pruning [1903.04476]

- B-splines are discussed as being "robust", but what is meant by robust is not clearly defined. If this refers to memory retention, then as stated in the previous comment, this memory retention seems to come from parameter fixing during training. Why would one need B-splines in this case?

- To address continual learning, e.g. sequentially learning tasks as is discussed in experiments, the ability to learn new tasks seems to come from expanding the model. Is this correct? This can also be done with standard NN models, e.g. as in Progressive Neural Networks [1606.04671]

- The experiments are only examined on 2D problems, but there are several reasonably standard benchmarks for sequential task learning. Moreover, no comparison to baseline models is shown.  This would seem especially relevant, as the Atlas models do not seem to robustly fit several tasks.

- There is no discussion or examination of information sharing across tasks, e.g. ideas such as forward or backward utilization or transfer of information across tasks. This would be helpful in understanding how the model could robustly use previously learned information across tasks.

- the related work section does not discuss the breadth of work on continual learning, including work on models with similar ideas implemented (such as the two mentioned above)

 
questions: - What is meant by robust, and how does Atlas achieve such robustness while other NN models do not? Is there a quantitative metric associated to this robustness?

- How does Atlas compare with baseline models and continual learning methods, e.g. in terms of new task performance, memory (since Atlas is an expanding model)? In general, as similar model expansion and weight fixing ideas have been implemented in other NNs, it is not clear how one may benefit from Atlas. Are there problems Atlas can solve, but other NNs can not solve?

- Can Atlas be applied to some of the benchmark datasets for CL, for example permuted MNIST is fairly common, but more challeneging benchmarks also exist.

- How is information shared across tasks in Atlas? When the model is expanded, does the model rely heavily on previously learned information, or simply learn new tasks from scratch in the expanded parts of the model? How does the expanded learning on subsequent tasks compare to learning that new tasks from scratch?

limitations: Yes, some limitations are briefly mentioned in the conclusions.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work presents Atlas, a new universal function approximator based on B-splines. Theoretically, Atlas has nice properties, such as sparse gradients, bounded gradients, and distal orthogonality. Experimentally, Atlas is robust to catastrophic forgetting, indicating its potential to be applied in continual learning.
strengths_and_weaknesses: As far as I know, the approach presented in this work is novel. This new function approximator has good theoretical properties with sound proofs. The experiments also show its potential in reducing catastrophic forgetting and continual learning.

However, this work can also be improved in many ways.

First, related works are not adequately cited. In continual learning, more strategies to mitigate catastrophic forgetting can be found in this [survey](https://arxiv.org/abs/1909.08383). Some methods with dynamic architectures should be cited and compared.
A short introduction of B-splines would be appreciated, given that B-splines form the basis of Atlas. As for memory retention, there are also some related works in deep learning, such as https://arxiv.org/abs/2008.13363 and http://arxiv.org/abs/2112.03257.

The notations used in the paper are not well-defined or explained. For example, in Definition 1, what are function $S_i(x)$ and $S(x)$? What is a uniform cubic B-spline function? Is $\theta_i$ a vector or a real value? There are similar issues in Definition 3 and Property 1-3. Providing proof sketches in the main paper for Theorem 2 and Property 1-3 would be much appreciated.

There is a lack of comparison of Atlas with traditional artificial neural networks for approximating the swiss-roll target function.
It will also be better to see the results of testing Altas in the classical continual learning setting on some benchmarks, such as MNIST.
Moreover, the experimental results are not well-explained. In Figure 6, why does the output difference have a cross shape?
I would also like to know the exact number of parameters used in the experiments for Atlas.

Overall, this work is interesting but more like a work in progress. And it can be improved significantly.
questions: In line 90, it is mentioned that "The minimum number of cubic B-spline basis functions is four." In line 140, it is written that "It is worth recalling that at most four basis functions are active for uniform cubic B-spline functions." Is there a contradiction here? Why four basis functions?
limitations: Some limitations are already listed in the Conclusion. Given that there is still space in the main paper, the authors are encouraged to finish some of them.
review_title: null

Meta-review: metareview: The submission proposes a novel type of neural network based on B-splines and exponential functions designed to reduce catastrophic forgetting, proves universal approximation results, and provides some experimental results. The reviewers find the submission interesting, but believe that the submission could be improved significantly in a number of respects, including better practices in use of the test dataset, comparison against baselines, and distinguishing between properties of the network class and properties of the learning method. Accordingly, I cannot recommend the present paper for acceptance.
confidence: Certain
Decision: Reject