Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper propose two new NN architectures, namely TNN, and SQANN. They claim that these networks are resistant to catastrophic forgetting, are interpretable, and are highly accurate.
main_review: The proposed methodology constructs the neural network according to the ordering of the input data and leaves a fingerprint for each data point in each neuron. No optimization is used in constructing the network. This is confusing and limits the applicability of the neural network.

The authors write that “a neuron in SQANN corresponds exactly to a data sample as SQANN stores its “fingerprints” as neurons’ nuclei”. In this case, how will the neural network be interpretable if it is trained on millions of data units? Will the proposed neural network still have millions of neurons and learned parameters? This questions interpretability. Also, how will the neural network conduct prediction for unseen test data?

The proposed methodology is very limited in dealing with high dimensional data. It is not clear how high dimensions can be incorporated in this modeling. This is an important issue that should not be overlooked. 

Under the Subsection Generalizability to test dataset, are the authors pointing to using test data in training?

The experimentation is also limited and not convincing.
summary_of_the_review: There are several key issues about this paper that are not well-supported and are not clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes two new neural networks (NN) construction schemes that aim at better interpretability, in the sense that (1) the NN should always memorize the training data; (2) the NN can roughly tell if a new test data has any similarity to any data in the training sample. The authors also provide approximation error bounds on both training and test data under certain assumptions. Some numerical examples are shown to support their theory.
main_review: Strengths:

The idea of making neurons reflect training data seems new to me. The authors also tried hard to give readers enough intuition to understand what they are trying to achieve in this paper.

Weaknesses:

1. Although the authors tried to guide readers by intuitive explanation in the paper, they did this at the expense of sacrificing too much rigor and clarity in their statement of the theoretical results. For instance, Theorem 1 is too loose as a theorem. One might write a theorem like this in the introduction but not like this in the main text. Further, what does "using TNN constructed with $D \cup A$" mean in the statement of Proposition 2, given $D$ being the training and $A$ being a subset of the test data? As another example, the word "synapse" does not mean much to a machine learning person who knows nothing about neurology.

2. As new as TNN and SQANN look, I highly doubt if constructions alike are really helpful for interpretability issues. The type of interpretability that TNN and SQANN try to garner, at least to me, is the ability to memorize the training data and then tell if a new test data has a similar counterpart in the training data. I understand interpretability is not a well-defined mathematical concept so what it really means depends on the audience. But I do think the authors should write what "interpretable NN" means in this paper upfront as there are too many different types of interpretability people are talking about.

3. In the kind of interpretability that TNN and SQANN are pursuing, TNN only works for input in $\mathbb{R}$. But with input in $\mathbb{R}$, no matter how complicated the neural networks are, interpretability is not a big issue. 

4. One has to constantly refer to the appendix even to understand how SQANN is constructed, the main algorithm of this paper.

5. No running-time analysis or empirical evidence is provided. The test examples are relatively simple (small data and low dimension). The main benefit of neural networks is the high prediction accuracy and scalability to high-dimension and big datasets. With this premise, interpretability then becomes important. Without showing that these new constructions can achieve state-of-the-art prediction accuracy in high-dimensional datasets, I do not see the importance of interpretability.

Further questions:

1. Is it necessary to write a whole paragraph on "Generalization to n-dimensions" for TNN? Maybe it is better to write one sentence and leave all the appendix as the authors did not think their solution is satisfactory in the TNN part.

2. If one uses SQANN in a one-dimensional setting, how does it compare to TNN? Does it also enjoy the properties that TNN has?

Some minor comments:

(1) It would be extremely helpful if the authors could create hyperlinks to all the sections/equations/figures that they refer to.

(2) In the statement of Lemma 1, "First layer of SQANN" should be "The first layer of SQANN".

(3) The appendix contains many typos and needs further proofreading.
summary_of_the_review: I do not doubt the theorems in this paper have any issues. But I do have trouble understanding the motivation of this work, as I have detailed in my weakness section. Without showing a learning algorithm achieves state-of-the-art prediction performance, I do not get why interpretability is of concern. As a result, I do not see how this work is a significant contribution to the ML community and would not recommend the paper to be published by this year's ICLR.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper presents a method for constructing neural networks where the architecture and the weights are derived analytically from the structure of ordered inputs instead of iterative training to provide arbitrary accuracy on training data.  Authors make a number of unconventional arguments about their architecture being resistant to catastrophic forgetting, and generalisation ability of the proposed model.
main_review: A very similar method for "constructing" neural network was presented by Szymanski.etal "Deep, super-narrow neural network is a universal classifier".  As far as I can tell, the requirement of enforcing an ordering of training point is equivalent to  Szymanski's projection of the data onto a line...which in effect order the points...and the manner of dividing the points into classes by stacking classifies side by side along the projection seems similar.  That paper shows that the choice of ordering of the inputs has a massive effect on generalisation.  So my question is - how is the choice made in the method proposed in this paper?  Authors claim that ordering of inputs gives rise to interpretability (though it is not clear to me how exactly that works)...but if the ordering is arbitrarily chosen by the user, then this gives nothing more than what the user already knows about the data (when making that ordering choice).

As far as I can tell, the catastrophic forgetting resistance claim boils down to ability of the model to perform correct approximation on a training dataset supplemented with new data points.  Is this understanding corrct?  Doesn't that imply that some points (presumably at least the two before and after the new point in the sequence) need to be known for the adjustment?  Does this involve adding more architectural elements (presumably at least one more neuron) to the architecture?  Catastrophic forgetting is about ability of a model to learn a new task, often out of distribution of the first task, without having access to training data of the previous task.  Not sure how the arguments presented in this paper relate to catastrophic forgetting.

I am very confused by the claim of generlisation of the proposed model.  My reading of it is that authors show that the error on a test point is bounded, but not by an arbitrarily small number, and the suggestion is to incorporate the test points that have high error into the train data.  Is this high level assessment of the presented proof correct?  If that's so, then wouldn't the model "generalise" best if all test points were incorporated into the training set....and wouldn't every model out there "generalise" by this standard by simply including the points with test error into the training data?


summary_of_the_review: The proposed model of progressive constructive of neural networks does not seem to be novel and has been shown to provide universal approximation as well as have limits, which the authors of this method do not seem to be aware of.  While the resulting architecture is capable of arbitrary accuracy of approximation on the training data, I don't see that it would generalise or cope with catastrophic forgetting, at least according to the typical notions of what constitutes generalisation and catastrophic forgetting in machine learning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Summary: 
1. This paper proposes two neural networks by construction, i.e., Triangularly-constructed Neural Network (TNN) and Semi-Quantized Activation Neural Network (SQANN). 
2. These two neural networks are universal approximators, which is proven by construction. 
3. These two neural networks are resistant to catastrophic forgetting. 
4. For TNN, strongly activated neurons and half-activated neurons can be identified. 
5. For SQANN, users can identify the samples that are likely out of distribution. 

Contributions claimed by authors: 
1. TNN and SQANN are proposed. 
2. The universal approximation is proven, using the construction of TNN and SQANN. 
3. Resistance to catastrophic forgetting is proven. 
4. SQANN can identify out-of-distribution samples.

main_review: Strengths: 
1. This paper provides two new proof of the universal approximation theorem by construction. To my knowledge, the constructions are novel and interesting. 

Weakness: 
1. The writing should be improved for proofreading.
1.1. Some figures are far away from the paragraphs describing them. For example, the double selective activation is defined on page 5, while the illustration of this activation appears on page 3 (Fig 1(A)). 
1.2. Mathematic formula should be more professional. In the 7th line of section 2, x^(k) are n-dimensional vectors and the relation "<" is not defined for them. Texts in math formula should not be oblique, such as "otherwise", "argmax", and "if". 
2. The claimed contributions are minor or not supported. 
2.1. Interpretable constructions for the universal approximation is not novel. Chapter 4 of [1] provides a visual (and can be proven rigorously) proof of the universal approximation, which indicates that neural networks can approximate functions using "almost" piecewise constant functions. 
2.2. The proof of the universal approximation using TNN only holds for 1-dimensional inputs, as admitted by authors in the paragraph begins with "generalization to n-dimensions" at the end of page 4. 
2.3. The claimed “universal approximation” is not exactly the one machine learning cares. This paper only considers fitting training samples, while the universal approximation in machine learning considers approximating a target function on a compact set. 
2.4. The resistance to catastrophic forgetting is not supported by theorems. Authors claim that this contribution is supported by Proposition 1 on page 4, but Proposition 1 has nothing to do with catastrophic forgetting. Catastrophic forgetting is " the tendency for knowledge of previously learned task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated"[2], which needs two sequential tasks. Although D and DUA are two training sets, Proposition 1 has nothing to do with catastrophic forgetting in the following sense: i) DUA contains D as a subset, thus the knowledge in the previous task (D) also exists in the current task and forgetting does not exist obviously; ii) the statement of Proposition 1 is trivial since we can always choose A=D’ and the conclusion in Proposition 1 is the same as Theorem 1. 
2.5. The generalization is not the one machine learning cares about. The paragraph begins with “generalizability to test dataset” on page 4 claims that “they (test data) can be incorporated into the training dataset to create a better model with the above-said error upper-bound”. In machine learning, the label of test data should not be used for training. 

[1] Nielsen, M. A. (2015). Neural networks and deep learning (Vol. 25). San Francisco, CA: Determination press. 
[2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.

summary_of_the_review: The contributions are not novel or even not supported. 
1. Interpretable universal approximation is not novel. It is written in a book in 2015. 
2. The resistance to catastrophic forgetting is not supported by theorems. Settings of the claimed theorems are different from that of catastrophic forgetting. 
3. The universal approximation and generalization are not the ones machine learning cares about.
review_title: null

Meta-review: This paper propose two new neural network (NN) architectures, namely TNN, and SQANN. The paper claims that these networks are resistant to catastrophic forgetting, are interpretable, and are highly accurate. While the reviewers agree that the idea of making neurons reflect training data is novel, some concerns remain post rebuttal. Most of the reviewers opine that the statements of theorems are unclear, confusing, and hard to interpret (even after the rebuttal and update), thus making it hard to appreciate the contributions of this work. Given this, we are unable to recommend this paper for acceptance at this time. We hope the authors find reviewer feedback useful.
Decision: Reject