Two Instances of Interpretable Neural Network for Universal Approximations
Authors: Erico Tjoa
Cuntai Guan
Venue: ICLR 2022 Submitted (2022)
Keywords: Explainable Artificial Intelligence
Neural Network
Universal Approximation
Universal Approximator

Abstract: This paper proposes two bottom-up interpretable neural network (NN) constructions for universal approximation, namely Triangularly-constructed NN (TNN) and Semi-Quantized Activation NN (SQANN). The notable properties are (1) resistance to catastrophic forgetting (2) existence of proof for arbitrarily high accuracies on training dataset (3) for an input x, users can identify specific samples of training data whose activation "fingerprints" are similar to that of x's activations. Users can also identify samples that are out of distribution.

Under review as a conference paper at ICLR 2022
TWo INSTANCES OF INTERPRETABLE NEURAL NET- WORK FOR UNIVERSAL APPROXIMATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes two bottom-up interpretable neural network (NN) construc- tions for universal approximation, namely Triangulaly-consmmnteed NN (TNN) and Semi-Quantized Activation NN (SQANN). The notable properties are (1) re- sistance catastrophic forgetting (2) existence of proof for arbitrarily high accu- racies on training dataset (3) for an imput users can identify specific samples of training data whose activation "fingerprints" are similar to that of x's activations. Users can also identify samples that are out of distribution.
INTRODUCTION
Artificial neural networks (NN) have recently seen successful applications in many fields. Modern deep neural network (DNN) architecture, usually trained through the backpropagation mechanism, has been called a black- box because of its lack of interpretability. To tackle this issue, various studies have been performed to understand how a NN works; see the following surveys Arrieta et al. (2020); Gilpin et al. (201 8); Tjoa & Guan (2020); Wiegreffe & Marasović (2021). This paper primarily proposes two interpretable models, namely triangularly-consstuuedd NN (TNN) and Semi-Quantized Activation NN (SQANN). Both possess the following three notable properties:
(1) Resistance to catastrophic forgetting. (2) Mathematical proofs for arbitrarily high accuracy on training datasets; experimentally demonstrable with python code and simple common datasets (see supp. materials). (3) Detection of out- of-distributton samples through weak activations.
Concept disambiguation. Several concepts have multiple possible definitions. We clarify the defi- nitions used in this paper.
1. Interpretability. We consider only fine-grained interpretation, i.e. we look at the meaning of each single neuron or its activation our models.
2. Universal approximation. Readers might be familiar with universal approximation of functions on certain conditions, e.g. compact sets. Our models can be more general, e.g. user can freely choose the interpolation function between 2 activations based on knowledge of the local manifold. The function can even be pathological. See appendix 1.2.
3. Catastrophic forgetting: the tendency for knowledge of previously learned dataset to be abruptly lost as information relevant to a new dataset incorporated. This definition a slightly nuanced version of Kirkpatrick et al. (2017). Hence, our models' resistance to catastrophic forgetting is the following. Given training dataset accurately modeled by an architecture M, a new dataset D' (especially new, out of distribution dataset) can be incorporated into M without losing accuracy D. See appendix A.2.2 (analogy to biological system included).
Related works and interpretability issues. Recent remarkable studies universal approximators include the Deep Narrow Network by Kidger & Lyons (2020), DeepONet universal approximation for operators by Lu al. (2021) and the Broad Learning System by Chen al. (2019); Hanin (2019); Park et al. 1) Johnson (2019). While insightful, they do not directly address the eXplainable Artificial Intelligence (XAI) issue, especially the blackbox property of the DNN. Similarly, a number of classical papers provide theoretical insights for NN as universal approximators, but interpretabil- ity, transparency and fairness issues are not their main focus. The universal approximation theorem by Cybenko (1989) asserts that NN with a single hidden layer can approximate any function to arbitrarily small error under common conditions, proven by asserting the density of that set of NN

---

Under review as a conference paper at ICLR 2022
in the function space using classic mathematical theorems. In particular, its theorem 1 uses an ab- stract proof by contradiction. From the proof. it is not easy to observe the internal mechanism of a NN in a straight-forwan Network) might inherit the blackbox property. Bottom-up constructions for function approximation using NN then emerged, though they also lack the interpretability (see appendix A.3 for more related works). Also consider a demonstration in Nielsen (2015) that could help improve our understanding of universal approximation.
manner, consequently modern works that depend on it (e.g. Deep Narrow
Outline. This paper is arranged as the following. Section 2 shows explicit TNN construction, related results, including a pencil-and-paper example for pedagogical purpose. Likewise, section 3 shows SQANN construction, statements regarding SQANN, another pencil-and- paper example, then experimental results of its application, before we conclude the paper with limitations and future works. Python codes and clearer figures are fully available in supp. materials (also see appendix).
2 BRAAGUUARLYY-CONSRRUA
NN (TNN)
TNN is the prototype NN for our interpretable universal approximator. SQANN (next section) partially borrows the concept from TNN which will be useful as an easy and manageable illustration to deliver the following ideas: (1) organized activations of neurons and (2) the retrieval of values as the outputs. The model is [2) = a² (Wr + where æ E [0, 1], a, b E IR and W E BN, where we sigmoid function for simplicity. We start with a simple scalar function y f(x) E R for x E [0, 1]. thus TNN's s interpretability can be illustrated very clearly.
Assumption: Linear Ordering. It is constructed ples ((x(()) (*) E R" x J such that ((NN) < (NN-1) < .... < x(1) and y(8) = (()(()), the true function that TNN will approximate. The interpretability comes from the linear ordering property where higher value of x ( 1) will activate more neurons while lower values will activate less neurons as shown in fig I(A). Then a values will be retrieved in a continuous way through dot product, eventually used to compute the output for prediction. In time series, such ECG (Electrocardiogram), signals can be approximated point-wise (although it is still preferable to have noise model during preprocessing prevent overfitting the noise). Meaningful interpretation can be given, for example, by mapping PQRST segments from ECG to specific neurons within TNN, giving some neurons specific meaning and thus interpretability. For more remarks and definition of formal linear ordering etc, see appendix A.4.
linearly ordered dataset containing N sam-
(A)
y00
yM
(B)
Jak--2
oW 00 b)
m
0
dWWx bb)
<WWMmis b)
x(i)
x(a
200
Figure 1: (A) Triangular construction is built by prioritizing interpretability of a neural network. As (4) decreases in "strength", the neurons are "turned off" correspondingly. (B) Activations of neurons for (M))) (k+1) and their mid-point Imid, Not only will neuron activation be half at the mid-point, the output ymid, = (x) + ((++1)) also half the sum of its neighbours'.
Ordered activation. We would like (1) activate all neurons, while (N) activates only 1 neu- ron. In other words, ideally (1) + b) [1, I, .,, 1, 1J"", followed by (2) + b) I,, 1, 0j and on until (N) + 0,, ... Ofz again, refer fig. 1(A). With this concept. we seek to achieve interpretability by successive activations of neurons depending on the "intensity" of the input, with x(1)) being the most intense. In general, the above can be written as
o( (k) = (W) + b) [1.....] 1. O..... T
N N- k-1
2

---

Under review as a conference paper at ICLR 2022
which is approximately achieved for lo .....,N at large (and exactly if - oo) with
v a,j ] N - k
(Wx(k) + b)5 +a S -
For more remarks, see appendix A.4 subsection ordered activation.
computing weights W, b, a. How then do we compute W, to achieve the ordered uctivation? Consider first (Wx(2) +Wx())+bxx = -a and (Wx(11) +b) = a and solve them. This yields WN= = 2a/ (1) and by = W Wyz(1) where A x0)) x 222), Iterating through k, i.e. solving (W (k+ + b) -+++1 = -a and (Wz(M) b) - we obtain Wx-k+1 2a/A
TNN construction:
and by
a W WN- where (kk) [()) (1+ We can rewrite the indices so that Wx = 2 (N (N+ and 6y
+++ -
WNr(N-K+H whenever convenient. For A (A), we need dummy x (N++)) value or we can directly choose its value, e-g. E.WIAA (k)), The effect is illustrated by the value near æ 0 in fig, 2(A1-3) and should not pose any problem; the chosen dummy value will only affect the shape at the left end of the graph.
We compute using the property of equation (I). From fig, I(A), this means ideally y()) aa: + b); and
until g( a1o(W=(()) + +)1, Putting them together y = jy(), ..... ........)jj. g(M)jT we get y = Aa where A is an upper-left triangular matrix and the inverse A- exists. Thus, a A a matrix such and zeroes otherwise, which facilitates a convenient computation. The triangular construction is complete:
EN 0:.0 (Wat x11)) + b); for + oc, and similarly y(2)
that A. 1 along the diagonal, A
TNN(() = a" (Wx+
(3)
While Nielsen (2015) provides only visual demonstration, the following result shows rigorous proof on universal approximation at work (python code also available).
Theorem TNN achieves arbitrarily high accuracy on the training dataset. Proof: see appendix 1. Also see example results in fig. 2.
(A1)
(A2)
(A3)
(81)
1.0
(B2)
(B3)
06
0.2
A1C
8,00
450
8.78
2.50
65 3.0
00
Figure 2: (A1-3) Three triangular constructions (orange plots) using different values of a. Higher a results in more step-wise plots and more constant values around the data samples (blue points). (B1) Plots of NNs approximated using triangular construction (smooth green plots) over scatter plots of the corresponding true data (green open circles). The parameters are as the following A 1,A
-1, C 0 for all, (B2) A = 0.1, B 20, C = 1 (B3) A = 0.1, B = 10, Ci = 1.
With the following proposition, test dataset that resembles training dataset will yield small error. Otherwise, there are nut-of-distribution (ood) samples A C D', which can be incorporated into the training dataset. Catastrophic forgetting is not a problem during re-training because when ood samples from the test dataset included as training points, each training point D is still identified with a neuron. By theorem 1, x is still approximated accurately. See appendix A.2.2 for remarks advantage over existing methods.
Proposition 1 Errors monotonous interval. Given finite training. test datasets D, there exists A C D' such that, using TNN constructed with DU A, for all samples in test dataset (x', yy) E D', sample-wise error = yy' (x')) has an upper bound max(/y/ - some k. Proof: repedddi A.4.2.
(y [k+1)|, w/ - - (10 for
There is also mid-point property that be exploited for generalizability arbitrarily high curacy, where data must be sampled such that any instance Frast lies inside either (1) the training dataset (2) is equal some mid-point of two neighbouring training samples; see the proposition
3

---

Under review as a conference paper at ICLR 2022
below. Fig. 1(B) shows how the component of & j N - k 1 is half-activated i.e. the activation value is 0.5. Admittedly, this is an ideal condition for accurate generalizability.
Proposition 2 Mid-point property. The mid-point k (x(() + x((+1)) takes the value of a (ooWaaam I + b) (g) 1))+ Proof: see appendix A.4.3.
TNN pencil-and-paper example. Use TNN to fit the dataset (æ: v) E {(1, 1), (0.5, 2), (0,3))
Then f(x) R (x) = 30 + 5) - ar (20x - 5) - (20r - 15). See appendix A.4.4.
Remarks smoothness property, special case, scalabllityccomplxity and generalizability to higher dimensions can be found in appendix A.4.5. We proceed with SQANN, universal approximation inspired by TNN that allows multi-dimensional input, multi-layer stacking of neurons based on relative strength of neuron activations.
3 SEMI-QUANTIZED ACTIVATION NEURAL NETWORK (SQANN)
SQANN architecture is multi-dimensional universal approximator which (1) retains TNN's idea of using an organized sequence of activations to retrieve a, (2) remotely resembles a Radial Ba- sis Function, but (3) has deep neural network properties, such as the possibility of deep learning (multiple constructed layers) and neuron activations. The difference is, a neuron SQANN corre- sponds exactly data sample SQANN stores its "fingerprints" as neurons' nuclei shown in fig. 3(B). The neurons can be activated with different kinds of responses: (1) distinct peaks and (2) half-activations and (3) weak/zero activations, made possible by double selective activation dasa- They are illustrated in fig. 3(A) and the sketch of proof of proposition 3 in appendix A.5.4, inluition behind daa-
The aforementioned distinct types of activation are key to SQANN's main result and give the model an interpretability at least in the following sense: samples are highly/moderately/nnn recognizable their activation patterns stroggyymodderaelly/eeen training sample x,, where identifications are facilitated by the distinet regions of a The design is "semi" quantized since dasa has approximately "distinct" levels yet remains continuous. Since SQANN incorporates multi-layer structure, it avoids being non-generalizing model that nearest neighbours methods suffer from; the remark in (Pedregosa et al. (201 (a)).
resemble the activation patterns of an existing
Notations. The order of data sample within the dataset matters, thus we define our own indexing to prevent confusion. Let the finite training set be (xx(1), ((()) X x Y k 1,2...., .N. We create the SQANN model that predicts (*) SQANN (zx()) with provably perfect accuracy and generalizes well to similar test distribution. Subscript indicates layer, denotes activation values collected the "synapses", square bracket with subseript denotes vector component so that [V2] is the 4-th vector component of the activation of layer 2. Layer k consists of (Nx, ax), where N& = (m) (m1 1 -² ..... Mr <m22)) stores fingerprints/patterns 0% = 1 .8k <2>
..... Uk <m,))
stores output values. The angle bracket denotes the index after relabelling. Hence, if k-th data sample before relabelling is (z(*), ( k 50 and becomes the first node in layer 2, then we <>> = T/=00 Concatenation. To denote the addition of the new k-th node to the layer 1, use write 1/2
U, where can be for example ((m) the activation of the m-th data layer 2. Alternatively, Ni - concal Nr, n <k>>). We can speak about layer i using N₂ if or is not yet involved. However, once concatenation of N; is decided, always correspondingly concatenate the aj i.c. <k> + (m)). Selective clustering of pı (IX,, 31k) for k- 1,2 is loosely defined for 21, T2 that are close each other such that: if y/, 3/ are similar, then Pl. P2 are clustered together, otherwise two distinct clusters are created; see appendix for formal definition, its effects on interpolation and more remarks.
<k>
M
Double selective activation. Given selective activation <(x, a) = at2² and Super Gaussian 4, 0.5, then the double selective activation is (fig. 3(A)):
a
s,(2,a) = exp( (x(a))2)
Tdsa (I, a1, r) = (1 r)) r) π(z a1) + x (2,))2)
(4)
Nodes activation. Denote the "synapse" or the activation value of node j at layer i by input v as:
[ok]; = Jdsa (Ilv 7/ <3>
(5)
4

---

Under review as a conference paper at ICLR 2022
where 7i <>> N& for j 1..... ,, and n4 is the number of neurons/nodes in the layer. See fig. 3(B). SQANN, activations will be forwarded layer by layer, i.e. [ui]s = dsaa (1|r - 1/i <>>>
<j> where <j> E Ni and [ux+1lz = Jasa (1le me4il) where kkt1 E NN+11.
3.1 SQANN CONSTRUCTION
Outline of SQANN construction with interpretations. SQANN is constructed without optimiza- tion like gradient descent. Each indexed training data sample is converted into "fingerprint" or pattern of neuron activations, which undergoes one of the following:
1. Admission layer k. Sample's new/distinct fingerprint is added into layer if the sample weakly activates existing nodes in the layer (vjj, [u*ly < Tad) and no collision occurs; see fig. 3(D.1).
Collision. sample activates one or more neurons strongly i.c. 3j, kt, [uu]i > Tact- The earliest layer where collision occurs is denoted l.- Such sample is integrated into l., thus very similar samples are selectively clustered. See fig. 3(D.2); the concept is also illustrated in the sketch of proof for proposition appendix A.5.4.
3. Filtering into deeper layer occurs when neither of the above occurs (no strong activation, some moderate activations). Such sample has features loosely similar to previously seen samples, but we need to filter them further to distinguish its finer features.
[A]
Fex OJ xt 10
65-
Inpue, g
"synppees"
"yrropund
Lapol sup
Injer alphas
10]
[.])
[D.2)
adrvictions
Mooro as tuted
calbann
Figure 3: (A) Double selective activation with different parameters. (B) SQANN schematic. Each layer (Ng, 04)) is stylized as a collection of neurons. A neuron stores the main "fingerprint" in nucleus 17 <k> (dark brown) and its corresponding "output" in nucleus a o= (dark red). When strong activation detected, the signal will be redirected to the dark red nucleus aj <k> (C) SQANN used for simple classification. (Left) The large filled dots are training samples, marks are test samples. Bright red indicates y = 1.0, dark red y = 0.5. (Right) Same as left but test samples that interpolated (i.e. strong activation)
annotated with red open circles; colored lines indicate which two training samples are used for the interpolation. Lines are marked with different colors and styles for clarity. (D) Construction of SQANN when (D.1) admission occurs: new neuron is introduced, creating more connection analogous to mammalian brains. (D.2) collision occurs.
Layer 1 construction. To initialize, let N1 = (n> and 01 = = (0) where m/ <1> + (1) and g + n)) Let Tad: Tact be the admission threshold and activation threshold respectively. We typically set Tad = 0.1, Fact = 0.9. We extend the layer to tuples N, = -m ..... 71 ( and on = = ......1 <m1>) through sample-collection function in the pseudo code 1. To do this, take new sample (x), ",, (k)), k: and we check N₁ activation, i.e. let vi
(k) be the activation of current
layer by this new sample, i.e. [o] (K)] aasa (1)=(()) 1/1 <5> II) for all <j> E N₁. Then, either: (1) new sample is admitted to N1 as a new distinct node/neuron. If for all j such that [b(2)] < Tads then N1 - concat(N, (8)) and an + concat (ar.at(a1.y1 ())) (2) collision occurs, when there exists j such
5

---

Under review as a conference paper at ICLR 2022
that Tact < [1] 3 1, thus sample is admitted via collision resolution mechanism. Exclusively for layer 1. sample will simply be admitted into the layer for selective clustering. New sample
(R)
causing [ , is filtered to deeper layers, when neither occurs. Finally, we complete the iteration over all training data, N1 = -1n <n k= 1....... n1) and a1 = (y k l........,
collision is unresolvable; see appendix on ill-defined datasets (3) or new sample
Lemma Layer 1 of SQANN achieves arbitrarily high accuracy on training data subset N1 x on
x x<m)) ); note that x <>>> is not necessarily (:k) except for k for initialization. To prove the lemma, take sample (xr E Ni x a1. Then we must have x= = x xxン>,,, ,V
Proof: Let N1 = = = =1),,
a1 <j> for some j = ,,...,71. RR. Since [vi]y = adsa 7/2 <J'> II) and 7 <i'>> = < for all j' = -..... ni, we get exactly [01], = 1. Furthermore, for other * j. we have [M]] < 1 due to the admission conditions (1) and (2) used during check N1 process. Finally, computing y バーン where
grgmazy | we retrieve the exact value. D
At this point, it may be clearer to readers how SQANN is constructed. In short, for each layer, representative activations become the neurons of the layer. In layer 1, representatives activations are the samples themselves. In deeper layers, they are activations propagated to the layer.
Layer k construction. Layer i construction is similar to layer construction, except collision could occur at any layer Le S & (next paragraph). Assume every layer 1 E A = fl..... k-- have been constructed using XiEa C X. Assume there are still unused data samples i.e. U X UU X. is non-empty, obtained from samples that have been filtered to deeper layers. Let = {u<I> u uu2>,... after re-labelling the indices, with corresponding output values {y<1>,... ,...>,... J. Initialize by first checking 1/ the activation of u ==1> at layer l for collision; if collision occurs, see next paragraph, otherwise, set N₂ (n 0g (yx1>) ie. M/
<>>>>> f uk <1 Similar to layer
I construction, perform check N& activation on (u<t> y)1))) for i > 1 by computing activation oh <> and checking it against the existing nodes. One of the three cases occur (1) admission, when [uk
[ < Tad for all j = 1,.... .i and no collision (2) collision, when there exists index j at collided layer Le S k such that [v] [0 2 > Tact or (3) otherwise. If (1) occurs, the activation uk o. is added as new neuron to the layer, N₂- - concat (Nk, 2f(2). If (3) occurs, filter the data sample for deeper layer. Assuming no collision, the process is repeated for the next unused data sample < until all remaining data samples are checked. Once done, repeat the process for layer k + construction.
Suppose collision happens when we check u um>>> at layer las we use the collision resolution mechanism. We destroy all layers 1 > le and put the collided sample into le, i.e. Ni, concat(Ni, Ni., u <m>> and a,_ + concat(ai, yr wy=²) (push-node in the pscudo code). No layer will be destroyed if Lg = k, the current layer. The data samples used in each destroyed layer are returned to the list of unused samples in the same order they have been used during the construction (return- index() in the pseudo code); we refer to this order integrity. Once the colliding sample is added to le. effectively, the strong activation in this layer is now overshadowed by maximum activation (selective clustering in action), since the exact neuron now included as the representative of itself and its locality. This is possibly practically inefficient process, since we tear down intermediate layers, but we only prioritize the completion of the construction for now.
Computing output via SQANN propagation (prediction). Let an input be x. The output = SQANN(z) is computed by propagating and processing signals through the layers; fig. 3(B).
Then [vil; = assa (IIx- - 1/ (11)) If there exists j such that of <,3> > Tacts then set y = a <3> where
j = (vrla Otherwise, for subsequent layer k; recursively compute [mk]; = dsa (1
<i>l]) II) for all j 1,..., ng.. If there exist j, such that [vx]; Tact, then y ox <j> where j argmazzy [vw]yr. If such layer k is not found, we have to perform interpolation.
Interpolations can be done in many different ways, and this paper implements a simple interpo- lation using values from the two most strongly activated neurons. Suppose V11 [Um]e and
V2 = [enly are the two most activated neurons, then the interpolated value can be, for example,
y Vi Vilam)++V(aal The form of interpolation can be adjusted according to the knowledge
V/+V2

---

Under review as a conference paper at ICLR 2022
have on the dataset, e.g. use TNN with high if we know the dataset is locally constant. See appendix A.5.2 for illustration and remarks.
Each training sample admitted to layer Nr leaves fingerprint, in the sense that it has collection of activations { = I..... }} as it is CQANN--ropageen through the neural network. This collection is unique amongst training samples, especially because of [ux]; within layer k it is admitted into. Furthermore, locality is preserved to the extent that if x' xe I, then activation [v<ll Rs 1 is around the peak and thus, by SQANN propagation, due argmax, it is likely we retrieve y,, the ground-truth value corresponding to æ. For now, we only use argmax, but more subtle adjustment can be done to obtain y' y but y' + y for x' by incorporating information about the manifold at that locality, if such knowledge is available. See appendix A.5.3 on good practice for scalability.
where j is an index
Completing construction. Due to collisions, readers might wonder if the construction will complete at all. During collision, layers are torn down and reconstructed. Suppose during layer & construction, collision occurs at layer for < k. Upon reconstruction back to layer k. layer may be torn down again the next collisions. Is it possible that collision occurs infinitely cyclically? The following proposition addresses the concern through order integrity previously mentioned.
Proposition 3 SQANN construction completes with high probability = 1. See appendix A.5.4 for (1) sketch of proof and a required assumption (2) stronger assumption needed for p = 1.
Arbitrarily high accuracy training dataset D is relatively simple to prove in the following theorem: roughly for each (r.y) E D. there exist 1, k such that x maximally activates the node n thus the correct is guaranteed to be fetched from ag. Catastrophic forgetting resistance is proven similarly: when new samples are used for training, previous samples not forgotten since SQANN stores the particular fingerprint 1i. < for each sample.
Theorem 2 Assume SQANN construction is completed. SQANN achieves arbitrarily high accuracy on a training dataset. Furthermore, it is resistant to catastrophic forgetting. Proof: see appendix A.5.5. Note: Our code provides experimental demonstrations showing zero errors on all training samples; see example results in fig. 3(C).
SQANN pencil-and-pape example. With @1 02 = 0.001, 0.5, Tad: Tact = 0.1,0.9, create
SQANN universal approximator for indexed data [ze 22)), (()) x1)
1, 1.2 - 113]
and Y [yn, /2, V3. y4] = [1, 1,0, 0). See appendix A.5.6 for more questions and solutions.
3.2 EXPERIMENT TO TEST GENERALIZABILITY OF SQANN
Test datasets with increasing spread from training distribution. The accuracy of SQANN on high-dimensional dataset outside the training dataset harder to formalize in theorems. Further- more, real life data is often noisy and possibly not regularly structured. We avoid making any related statements for SQANN for now. We instead provide empirical results on test datasets that similar to the training dataset, the extent that each point in the test dataset is training sam- ple perturbed by uniform random values of increasing magnitude. We refer to the noise magnitude as the test data spread. Fig. 4(A,A.2) show four domains X with different test data spread. Test dataset that has larger test data spread contains data samples that are noisier and further away from the training data points. Fig. 4(B.B.2) show that SQANN naturally performs better with smaller test data spread. As the test data spread increases, larger errors are observed. Likewise, smaller spread means smaller Ninterp. i.c. fewer data samples fail to activate neurons in SQANN strongly. For all, training errors are 0 as expected from theorem 2.
Classification and visualization of SQANN's special interpretabilityy features. We show the use of SQANN for simple classification problem in fig. 3(C). The ring outside is labelled 0.5, while the ring inside 1.0. With activation parameters ai = 22 1, we achieve zero error not only for training dataset (to be expected from theorem 2) but also on test dataset. A special feature in SQANN its ability to tell the user which data samples fail to activate any neurons strongly; such samples' output must be interpolated (see SQANN propagation). In fig. 3(C) right, points marked with red open circles need interpolation. Each such point is interpolated using two training samples whose "fingerprint" neurons most strongly excited. These two samples are shown as the two
7

---

Under review as a conference paper at ICLR 2022
points directly linked by colored straight lines to the open circle. This is possible because SQANN systematically stores indices of training data samples within respective layers. The list of indices organized by layers even be explicitly printed e.g. see SQANN.ipyn,, supp. materials.
SQANN is tested for regression on Boston Housing and Diabetes Datasets to demonstrate its generalizability to unseen/test samples, the following (simplified here): (1) A small subset of samples C 'D (the first 20% of full dataset D) used to train SQANN and 9 other regression methods. (2) Mean Squared Errors (MSE) values are measured on unused data Drest on all 10 models; we expect large errors on some test samples. (3) SQANN's activations are used to collect samples with large absolute errors e (z [SQANN(<
yol T and we treat them as out-of- distribution (OOD) samples. These samples are considered as new distinct samples to be integrated into D) as the new training dataset D'. (4) Train the models, now with D' (5) Then MSE measured again on Diest (yes, there will be partial overfitting) From Boston dataset: for T 5, SQANN MSE improved from 9.90 3.08. Decision tree improves the most with D' (7.36 to 2.07); see table 1. For more details and diabetes dataset, see the appendix A.5.7.
spreet Oot
(4-)
Figure 4: (A) Training/test (circles/x marks) data for demonstration. Smaller/targer test data spread means test samples are closer/further to/from training samples.
(B) Boxplots for data whose dis- tributions are similar to (A). Column 1(3): (fractional) errors on test data samples. Column 2(4), (fractional) errors on test data samples excluding interpolated samples. Column 5: no. of data samples whose predicted values are interpolated. (B.2) Similar top, but for (A.2).
CONCLUSION, LIMITATION AND FUTURE DIRECTION
Limitations and future directions. TNN is clearly limited in regards its application multi- dimensional input data although it can be useful on different types of time series data. SQANN lim- itation and possible future development currently include 1) simple sequential drawing of samples that may result in the imbalance of layer size. In the future, more sophisticated ordering of training samples can be used so that layers are constructed with meaningful and purposeful arrangement e.g- deeper layers can be purposefully reserved for rare cases; more research on this is necessary to op- timize the results 2) layer destruction during the treatment collision cases might be an inefficient mechanism, which can be improved in the future.
Conclusion, we have proposed TNN and SQANN, two interpretable NNs for universal approxi- mation designed to (1) be resistant to catastrophic forgetting (2) have provably high accuracy training datasets and (3) can be used to handle out-of-distribution samples.
Buffer page. Just in
Table 1: Comparing MSE different regression methods for Boston Housing dataset. Row o., or original, shows MSE obtained from models trained on D. Row eT shows MSE from models trained on D' with T. All evaluated on Drest-
Lin
Ridge
Lasso
LSVR
NuSVR
SVR
DTree
kneigh
MLP
SQANN
36.45
7.990
9.834
8.355
8.833
8.712
7.356
7.393
12.77
9.898
e5
5.139
5.135
7.068
5,950
6.295
6.068
5.026
4.798
3,481
7.998
c2
4.993
5.028
7.882
5.832
6.121
5.895
2.072
3.025
3.846
3.076
8

---

Under review as a conference paper at ICLR 2022
Note table 1. The entries in the header denote the models available in scikit-learn (Pedregosa et al. (2011b)): Lin, Ridge and Lasso are the linear models: linear, Ridge (linear least square with L2 regularization). Lasso (linear with L1) respectively; the Support Vector Regression models: LinSVR, NuSVR, SVR are respectively linear SVR, v-SVR and 6-SVR. DTrre: Decision tree; kneigh: k neighbours are selected from the best k 2, 3, .... 16, MLP: multi-layer perceptrons, or the fully-connected neural network, with 2 layers, each layer having 64 neurons each trained for max of 12000 iterations (convergence is attained for both). For SQANN, the initial model trained on D is kept after SQANN' is trained on D'. Thus, we can choose results based on the strength of activations between SQANN and SQANN'.
For Boston Housing Dataset T = 5 (i.e. e5 of table 1), using SQANN we integrated 211 samples from the test dataset into training dataset, so |D' = 311. Overall, 0.615 of the whole 'D is used for new training. For 7 2, i.e. e2 of table 1, using SQANN we integrated 319 samples, so D' 419 iee. 0.828 of the whole D is used for new training. With r 2, regression performance of SQANN improves greatly compared to other methods, except for decision tree regression. We have thus also seen that SQANN can be used to perform sample selections for data that appear to be out of distribution; this has improved decision tree performance greatly. The performance of other models have improved reasonably too, especially MLP. For MLP, however, the randomness used to achieve convergence some local minima might have led it explore other minima; hence get slightly decreased performance for e2 compared to e5.
def sample collection(X.) Y Ilyer)): i=unused indices[O] x=X[i.:]
def fit data(X.Y):
# Main SQANN loop
Lnow=l layer now
x, collision forward crrwarr_oonx(xllyrr---
while True:
ssig, collision check signal(collision)
ssig, collision sample collection(X, Y, Lnow)
nodes, node values new nodes(x, Y(i))
if ssig 'no_more.data':
remove index(i,layer)
break
for in unused indices:
elif ssig is "collision":
x=X[i.]]
Lc =c= collision! [allidee_layer'))
collision = forward forward conss(,laaer--)
for Ljj from Lc+1 to Lnow+1:
ssig, collision bheckkssinallccolisionn)
(cturn.ndeex(l.)
act=activate(x.nodee
kp = colliiiion perpetrator index')
push. ((kpl)) L[kp).l.c 1
update_ (nobesxxnooe(xx, nio. moddeees,noee__llllle values)
remove inyex(i.layer)
now+=1
return ssig, collision
Pseudo code 1: Pseudo code for the construction of SQANN. The function activate corresponds to equation See appendix B.2 for mapping to python code.
ETHICS STATEMENTS
This paper introduces function approximators with novel properties. It is purely mathematical and algorithmic. No specific ethical issues are present. The ethical context depends only on the dataset but in this paper, only common public datasets have been used.
REPRODUCIBILITY STATEMENTS
All codes are available in the supp. materials (to be released to public repository in case of ac- ceptance). Results are easily reproducible even with without random number seeding since data distributions are sufficiently controlled. Jupyter notebook for our particular results are also present. Proofs are all included in the appendix, with sketch proof and additional statement assumptions where applicable.
ACKNOWLEDGMENTS Anonymous for now.
REFERENCES
Alejandro Barredo Arrieta, Natalia Rodriguzd Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins,

---

Under review as a conference paper at ICLR 2022
Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, tax- onomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82 115, 2020. ISSN 1566-2535. doi:
URL http: 93081 03.
/ //www. sci encedi rect. com/ sci http:///cccoeeeeaaaaaaaaaa////aa
C. L. Philip Chen, Zhulin Liu, and Shuang Feng. Universal approximation capability of broad learning system and its structural variations. IEEE Transactions on Neural Networks and Learning Systems, 30(4): 191--1244,,,,, 2019. doi: 10.1109/TNNLSS
C. K. Chui, Xin Li, and H. N. Mhaskar. Neural networks for localized approximation. Mathematics of Computation, 63(2063):07--63, 1994. ISSN 00255718, 10886842. URL http: www..
C. K. Chui, Xin Li, and H. N. Mhaskar. Limitations of the approximation capabilities of neural networks with one hidden layer. Advances in Computational Mathematics, 5(1):233-243, Dec 1996. ISSN 1572-9044. doi: 10.1007/BFF2122445. URL https: /tcl:///ooiiooo /ttp://///o...//......0... 1007/ BFC2124745.
G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, Dec 1989, ISSN 1435-568X. doi: 10.1007/BB02551177. URL https::///// /woi/ooooooooo/ooooo/ddd org/ / 10.1110//F020511111111 0E5551274
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L.. Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80-89, 2018.
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu doi: 10.3390/math7 100992. URL
activations. Mathematics, 7(10), 2019. ISSN 2227-7390.
https:/ /twp//wwwwkk wwww.mdppp wwwpi.pp...mmm....m com/ 2222- /01/999..
Jesse Johnson. Deep, skinny neural networks are not universal approximators. In International Conference Learning Representations, 2019. URL http:t/ http:pss///ppnreeee.eeeeeeeeee wttp:viiwwwweeeoruuttteo id=ryGgSsAcFQ.
Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks, 2020. URL https://cpeereet
net wwwoorrmnddixxxeeeee
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabsa-BBarwinsee Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ting in neural networks. Proceedings of the National Academy of Sciences, 114(13):351-33526 2017. ISSN 0027-8424, doi: 161.3/3/paa..1111351111 URL https: /www. pnw..org////.................... conteet//14413//1/
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Kariiadakis. Learn- ing nonlinear operators via deeponet based on the universal approximation theorem of opera- tors. Nature Machine Intelligence, 3(3):218-229, Mar 2021. ISSN 2522-5839. doi: 10.1038/ s42256-021-0030025. URL https: : /tw.oiioooooooooooooo org.l0rg10000 10. 1038/ 422256 021-00302-23
H. N. Mhaskar. Neural networks for localized approximation of real functions. In Neural Networks for Signal Processing III Proceedings of the 1993 IEEE-SP Workshop, pp. 190-196, 1993a.
H. N. Mhaskar. Approximation properties of a multilayered feedforward artificial neural network. Advances in Computational Mathematics. 1(1):61-80) 1993b. ISSN 1572-9044. doi: 10. BF02070821. URL ht.ttt///////tt
//o.tttpp////....o /oi. wor. /0. 1007 100..00000//000000000000 /BF0000022222111
H. N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neu- ral Computation, 8(1):164-177) 1996.
H.N Mhaskar and Charles A Micchelli. Approximation by superposition of sigmoidal and radial
basis functions. Advances in Applied Mathematics, 13(3):350 - 373, 1992. ISSN 0196-8858. doi: 0ttp:/00000000000000000880000000 con/://cceneee/rrrceeee///////ss
URL https : www. sciencedirectt.
10

---

Under review as a conference paper at ICLR 2022
Michael A Nielsen. Neural networks and deep learning, Jan 2015. Beeraaoottooaaabeeeeenn..a con/ chap4 html.
URL http://
Sejun Park, Chulhee Yun, Jaeho Lee, and Jinwoo Shin. Minimum width for universal approx-
imation. In International Conference on Learning Representations, 2021. URL https:
/ //openreview.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, Weiss, V. Dubourg, Vanderplas, Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Re- search, 12:2825-2830, 201 la. URL pettps://scikiee meeeiiiitteeaan....m org/ www.aan...g//tabe///e/eeee stable/modules: neighbors html.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 12:2825-2830, 2011b.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta Numerica, 1999.
M. A. Sartori and P. Antsaklis. A simple method to derive bounds on the size and to train multilayer neural networks. IEEE Transactions on Neural Networks, 2(4):467-471, 1991.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions Neural Networks and Learning Systems, pp. 1-21, 2020. doi: 10. 109/
TNNLS 222N...02003022314,
Sarah Wiegreffe and Ana Marasović. Teach me to explain: A review of datasets for explainable nlp. ArXiv, abs/2102.1 12060, 2021.
Guang Yang, Feng Pan, and Wen-Biao Gan. Stably maintained dendritic spines are associ- ated with lifelong memories. Nature, 462(7275):920 924, Dec 2009. ISSN 1476-4687. doi: 10. 10.008//aauue008577 URL https: /tt:://puuuue /twp//pueeeeeee ncbi nlm, nih, gov/19946255. pmidd.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
We have reorganized the content based on ICLR 2022 reviews. Also, contents that are less relevant are moved to the Extra section the end of appendix.
A. ALGORITHM CODE AND SUPPLEMENTARY MATERIALS
All the codes are available in the supplementary material, and the link the repository will be made public if the paper is accepted. README. lists the necessary commands to obtain the results found in this paper. Results in Jupyter notebooks format also included.
A.2
CONCEPT DISAMBIGUATION
Link back main text introduction, section 1.
A.2.1 UNIVERSAL APPROXIMATION
Universal approximation by TNN and SQANN is not restricted to specific conditions. Traditional approximation may require strong conditions e.g. Cybenko (1989) shows result on C(In), space of continuous function. Continuous function that approximates two points necessitate continuous interpolation for any point along the path between these two points. However, hypothetically. patho- logical set of data points may exist where the value is not even continuous along the path. Our SQANN, for example, can handle this since we allow user to choose the interpolation function in the case interpolations are opted for output computation (e.g. during weak activations).
A.2.2 CATASTROPIIC FORGETTING
Catastrophic forgetting and online learning. Continual or online learning have posed some known challenges. In Kirkpatrick et al. (2017), catastrophic forgetting is defined as "the tendency for knowledge of previously learnt task(s) (e.g. task A) to abruptly lost as information relevant to the current task (e.g. task B) is incorporated". In our context, we can distinguish task A from task as old and new dataset. Learning from the new dataset which has possibly different distribution from the old dataset might degrade the performance of the model.
Analogy to biological system and resistance to catastrophic forgetting. Unlike artificial NN, mammalian brain retains old information when it learns new information by protecting previously acquired knowledge in neocortical circuits; see Kirkpatrick et al. (2017) and the references thereof. As rats learn new skills, the volumes of dendritic spines in their brains increase (Yang et al. (2009)) while existing dendrites persist, thus they retain old memories. Both TNN and SQANN have similar property. Particularly, SQANN increases the size of a layer as it progressively acquires new samples during construction (learning). This inevitably increases the number of weights that connect the layers; see fig. 3(D.1, D.2).
Furthermore, they exhibit resistance to catastrophic forgetting. In TNN, this is simply because the old sample Vold still yields activation with signature [1, ..., 1,0, 0, ..., 0). in which the last activated neuron (taking the value 1) is still identified with the old sample. For SQANN, each old sample is not forgotten since its exact "fingerprint" is already registered (i.c. input Fold has been converted) to a neuron's nucleus n) <k> for some I, k, as shown fig. 3(B). The activation pattern of a sample Fotd includes the value 1 in a specific node of a "synapse", o7 <k>>> and the particular combinations of values in other synapses.
Advantage over existing methods. Existing ML methods do not typically admit out-of- distributions easily; i.e. even if we re-train them on the new ood samples, they may end up with poor prediction on ood. As illustration, linear regression that includes ood samples may shift the gradient little. but ood sample is still far from the regression line. To make the matter worse, when there are many ood enough to change the distribution of the data sample significantly, the model may forget the previous distribution: we consider this catastrophic forgetting.
A.3 RELATED WORKS
Older works on the construction of universal approximators do not focus on interpretability as well, thus readers have observe for themselves the shape of the resulting networks from the components used in the construction. For example, there are works related to spline functions like Mhaskar & Micchelli (1992); Mhaskar (1993b:a); Chui al. (1994); another example by Sartori & Antsaklis
12

---

Under review as a conference paper at ICLR 2022
(1991) where two-layer NN can be formed by arbitrarily choosing the weights of first layer and computing the weights of second layer explicitly; and section 5 of Pinkus (1999) shows the equations used to obtain weights and proves the existence of such solutions. In some of them and others such as Mhaskar (1996); Chui et al. (1996), the focus lies in error quantification.
A.4 APPENDIX FOR TNN
Link back to main text section The traditional definition of NN with single hidden layer is given by E, x + b:) fol- lowing equation (1) from Cybenko (1989), with 2, y/ E R", by E R where a is any sigmoidal function with the property specified in the paper. A familiar example of sigmoidal func- tion is the sigmoid function 1/(1 + +-*). Compared to the traditional version, TNN has slight generalization on the weights W, which is also form that has been used in modern implementation.
Assumption: linear ordering. Linear order is any binary relation with (1) reflexivity (2) transitivity (3) anti-symmetricity (4) x < or I x. As is customary (Mhaskar (1993a)), the domain of the function is limited to [0, 1)mh where n is the number of dimensions of the input, justifiable for practical dataset with finite domain easily scaled to [0. 1]. Some readers might ask how linear ordering can be performed meaningfully. Also, if linear ordering is done by human, interpretability is not improved, since human user already knows about ordering. To answer that, we again refer to the ECG example we gave carlier: time series a naturally linearly ordered data. Next, linear ordering in TNN can be seen parameter strength tuning, useful for example when TNN is incorporated in larger system.
Ordered activation. Recall that we would like x(1) to activate all neurons, while x(\)) activates only I neuron, so that eventually achieve something like fig. I(A). To approximately fulfil the ordered activation conditions stated in the main text, first need (Wx(1) + b)j 2 for all j. where sub-script j denotes the j-th component in the vector, to distinguish from superscript (k) which denotes the k-th data sample according to the linear ordering. The next iteration will be (2x(2) b)j I -a for N and (Wx(2) + b) other (k) for l: 3, .,..., N. With this, attain eq. (1) and (2).
for I....,N 1, and similarly for
Since we use sigmoid function, these conditions are ideal and not strictly attainable, because sig- moid function asymptotically achieves 0 and at infinities. Nevertheless, we show later that we can achieve arbitrarily small error by adjusting activation threshold a. We define the activation threshold to be the value such that (a)) = 11 8, where 5 is a small number. For this paper, fixing a = 5 is sufficient. The shape of sigmoid function is convenient enough to be symmetrical in the sense that o (-a)= = ô since 1/(1+) ") for the proof later.
6 can be rearranged 1/(1+ (1++6 (-a)) useful
A.4.1 TNN THEOREM PROOF
Theorem 1. TNN achieves arbitrarily high accuracy on the training dataset.
Proof: We need equations (1) and (2). In practical situation, where is finite, we therefore have (k)
"j 2 1 - for < N - (k: 1) and of (k) < 5 for > N - (k: 1) and > 0. Then, define sample error as e(k) = (k) - "/="y)"'a and average error per sample will be e * + (k). Then (k) < 8(N + I)U where U = max (a) is the upper bound for the absolute value of the function over all samples (see proof below). Hence, setting 8 ū(N+TJ guarantees that Since d can be monotonously decreased by increasing a, we have shown that arbitrarily small error can be achieved in this approximation; see fig. 2(B1-3) for plotted examples. Note that (*))
e(t)
iff 5 = 0 iff a = 00.
Show that (k) < 6(N + 1)U where U maxx (1)) We abbreviate (k) as a, fixing k.
13

---

Under review as a conference paper at ICLR 2022
( = -1y(h)
= -1y(t) 0
(y( (g(N-1) - yy(n)os t...
+ (y(h+1) y y( )
0j 2 1-6
(y) -y (((t))) ONN-K+1
+ (2(k-1) )N=
+ (y(k-2) - -y -1lh11)) ON
k+ +
oj S
+ (y y(0))
which we rearrange according to ااا instead to
e(h) = (m
EN ン (oN-t11 ON-j+2
+ ( (0 [ON-K+1 - -+2
- -N + +
Rewriting d: = 0141 - Ji, we now have
c(6)) = I - ( dx d (h) - 51-5 (1-8--1--
8) + ングーbリ - + +
As either o; 2 1 or o; S 6, we have |d:| I 5 for all applicable i, and using triangle inequality,
(k) ( (8) 8 1018 < 6(N 1) max (a)
and we are done.
A.4.2 TNN ERROR BOUND AND RESISTANCE TO CATASTROPHIC FORGETTING
Proposition 1. Errors on monotonous interval. Given finite training, test datasets D, D', there exists A C D' such that, using TNN constructed with D U A, for all samples in test dataset y0) E D', sample-wise error e = y' - TNN(x') has an upper bound mar(ly' - (yltt)) ly' - (()) for some
Proof: Trivial cases occur for example when (1) most samples in test datasets are novel or out of training dataset distribution. The proposition will hold true trivially because then A D' i.e. all test data will be fit into TNN. (2) Most samples are already within the distribution, so sample-wise error is already within bound.
Consider non-trivial case. Since sigmoid function is monotonous, then for any E [Ik #x41 we have (x) E [yk, 3k+1]. The error of test sample is e = Ig' 1 TNN (z)). If x' E [Tg, #x+1] and y/ [Bk 11, the sample-wise eгror upper-bound result already holds. We say that such sample approximately within the training dataset distribution. Note that if turns out to be too large, i.e. the gradient within the interval is too large, we can always include the test data sample in the training dataset to make a more regular model.
(A)
Figure 5: Green circles: training data samples used in TNN. Green and red data samples. (A) The inclusion novel or unseen test sample into the training dataset (B) The inclusion of one test sample (green x) causes another test sample (red x) be out of distribution, and thus it needs to be included also.
Otherwise, if 3/ & [8k Me+1]] then we say the sample is out of training distribution because monotonicity of sigmoid function prevents the model from possibly making the correct prediction. Sample-wise error upper-bound no longer holds. By including this sample to the training dataset and recomputing the weights, recover the upper-bound for this sample. Any such sample is then
14

---

Under review as a conference paper at ICLR 2022
included as the family of subset A C D'. For each test data sample, check whether y' lyk. Vb+1] for some and include them into A if it is out of training distribution. This is shown in fig. 5(A). However, the remaining test dataset U training distribution" " category. This is best illustrated in fig. 5(B). However, by repeatedly running the same procedure to U and expanding A, will eventually terminate point where all (a', v²) E U is within the expanded training distribution D U A, or reaches the first trivial case. D
D' \ A does not automatically fall nicely into "within
As seen above, TNN is resistant to catastrophic forgetting for clear reasons: each data sample in D still corresponds to one exact neuron in TNN, even after adding C D'.
A.4.3 TNN MID POINT
(K) + z((+11) takes the value of Proposition 2. Mid-point property. The mid-point Imid = (xx
a" (W midd + 6) = (1) + 1))
Proof: omiid = oW Imid, + b), a( (W)(W(( + b) + (W (K+1) (W+ + b));- Then NN-+1 a(za + (-)) (0) 0.5. For j S N k, ofuid = while for j 2 N k 2, az 0.
mid
The resulting output of the neural network looks like a² (..... 1, 0.5, 0,... which equal to ja [..., 1, 1, 0,.,.... 1r 3 (.... 1,0,0, ...0 (y(()) ())+ D.
A.4.4 TNN EXAMPLE TNN pencil-and-paper example. Use TNN to fit the dataset (z, y) E {(1, 1), (0.5, 2). ()).
Then f(x = 30 (20x + 5) o20z 5) (20z - 15). Suppose = 5 and we have dataset
{(1, 1), (0.5. (0, })} so that N = 3 and (1)) 1 (2) 0.5 > x3) = 0. which is evenly
spaced thus we can use the simplified formula, for example, Wa =2 5 (3 (3-1) 1)
20 for
k == 1,2, and bj = 5(3 - 2 1) etc. Then W = [20, 20, 20 b [5, 15,5,- -15 and
a= A = (0
1
1
-1
2
-1 0
3
f(x) = at (Wx + 6) = 13,-1, - [20,, 20, 20
+ -5,
where a is applied component-wise, thus f(x) 320r 5) a220r 5) o(20r 15).
A.4.5 MORE REMARKS
Smoothness. From the construction, assuming sigmoid function as the activation function, it is obvious that the function is continuous for finite a. As increases, the function becomes more and more constant around each data sample as shown in fig. 2(A1-3), i.e. becoming more step-wise.
Special case. When the dataset is evenly spaced, (k) = 1 (k- 1k-1),, = N, = 1/(N - 1), the results simplify Wa (2) is fulfilled, we also get (R)( b)y a(1 + 2[N k +1 jl). For the k- th data sample, the activation will then be well-spaced in an interval of 2a, so that ((k) =o([..., -ac....-cce 3a, -a, a, 3a,..... x 1,
2a 1) and by a(3 2k). Not only equation
Scalability and complexity. The bulk of memory space usage comes from W E NX* N is the number of available data points, and this is unusual feature compared to modern DNN architecture. Since N in common datasets can grow very large, the space complexity becomes 8(Nn + mn), still linear w.r.t N. Since it is more likely that simplify to just 2 Nn). To reduce the complexity, we can pick set of representative data points Xyep [Irep] } for NN construction. The selection of representatives depends on our error tolerance. and it can be done through other machine learning methods, such as clustering. Other data points can then used for validation. The resulting complexity S2(Nreph) will thus highly depend on the variability and the structure of the dataset where Nrep = Xrepl Time complexity is almost irrelevant for now, since we are not able to find any meaningful way to compare with the training process of modern DNN, As far we know, there is no decisive rule on how many epochs are necessary for a DNN training through back pack-propagation.
N. it is reasonable to
Generalizability to n-dimensional output. Generalization to scalar input and output is relatively simple. From equation (3), can treat the coefficients for the only
15

---

Under review as a conference paper at ICLR 2022
component of mie-dimensional y. Generalizing E R identify each vector a; with the
component yt- Stacking them up, we can redefine a = |a : a. :... where a semi-colon denotes the next row, and the construction is done. Note that now R R==N
Further generalization to n-dimensional input has been unsatisfactory (see later section of appendix, B. Instead, SQANN has been developed with additional mechanism that TNN does not possess.
A.5
APPENDIX FOR SQANN
Link back to main text section
A.5.1
SELECTIVE CLUSTERING
Selective clustering has been loosely defined in the main text for reasons that will be clear after this. Formally and more generally, selective clustering is defined as the following. Let A = X x Y be a set, and (x, y) be a point. Define d( [4, x) as the minimum distance hetween x and all points in X, ergmin./ex x/|: let (1) A. Given If output values are similar (first case), ly y1l S then add (I, y) as a new member of A. Otherwise (second case), if output values are distinct, /// y1 > e, let (zz y) form its own cluster. In short, a new cluster is created (at least nominally) if we have neighbours with distinct values; they are neighbours, but we may be looking at points at different sides of classification boundaries. This can be easily generalized such that (Ir, y) is added to another set B where ly V2l < c for (12, 1/2 E
tolerances 6, 0. Let d(X, & 6.
As mentioned in the main text, this does have role for interpolation during SQANN propagation. If y, y1 are similar, then approximation is likely good. Suppose 2, is near both and X. In the first case (output values are similar), taking interpolation between strongly activated neurons within X (including 2) will yield approximately the same value taking argmax (the most strongly tivated neuron). This is the nice case. However, in the second case (output values are distinct), can get unstable result. This is because argmax might tilt between the different values in, while interpolation could yield the "averaged" value which may not reside in either cluster. The concept selective clustering thus introduced with the express purpose of talking about such sit- uation. During collision, if two samples strongly activate the neurons but have distinct values, we must be careful about using argmax or deciding to select our interpolation methods. In this paper, only argmax is used; other variations will be left for further studies. For example, when causes activation of 0.99 on one neuron and 0.92 on another. Both are strong activations, but with argmax- based selective clustering, Ig is treated as a member of the first neuron's 's strong activation cluster, thus the attached to the first neuron will be used as the output. There might be no best universal choice of selective clustering, considering that the shape of local manifold may change depending on the dataset distribution. Hence, we leave it for further studies.
(A)
(B)
Y2 [asl2
x
X
y
X1 issst loskt/ual
= =[22ss
(azla
Figure 6: (A) Strong activation at layer node 3 (yellow circle with red boundary). The output is taken as [ca]s (B) No strong activation, only moderate activations (circles with darker shades of oranges). The output shown is a weighted average.
A.5.2 COMPUTING OUTPUT VIA SQANN PROPAGATION (PREDICTION)
Fig. shows two cases occurring during SQANN propagation Case (A) shows case when strong activation occurs at layer 2 node 3, i.e. [U2Ja > Tact, near or even exactly (if x is the exact training sample used during construction). Then y = [a2]a- In case (B), no nodes are strongly activated. The values of the most strongly activated neurons are used for weighted average based on the strength of activations [Uz]s and [03]2, ie. interpolation is performed. Clearly, we can explore different variations, for example, taking three most strongly activated neurons etc.
16

---

Under review as a conference paper at ICLR 2022
Link back main text section 3.
A.5.3 GOOD PRACTICE FOR SCALABILITY
For both TNN and SQANN, if there are I million training data, there will be I million neurons. Do we need all million data? Probably not, since it might be fair to assume that some data points in the training dataset may have similarities. This is where pre-processing root out data that are redundant. SQANN itself can be used to check how redundant they are; for example, subset of the data can be collected and smaller SQANN can be constructed for testing purposes. We can check how the subset of 'similar data' activates each other's s neuron within this mini SQANN.
be done to idcally
Link back computing output via SQANN: go to lemma 1.
1.00
4.25
62683
82000
875
8.50
8:5
01:000 42.3
4201
800
4.25
4283003
0101
100
875
Figure More a1,02 02 variations of double selective activation.
A.5.4 SQANN CONSTRUCTION CAN COMPLETE
Proposition 3. SQANN construction completes with high probability Ro 1.
We start with the sketch of proof: the precise proof will need a strong assumption on data distribution. Note: exclude any dataset which is ill-defined, i.e. when there exists two identical ((k) (k') having different (k)) # y(k') This results in unresolvable collisions, and is not suitable for any function.
Let the activation space A be high dimensional space of pattern activations illustrated in fig. An element of am E AA is a collection of [uy]., and distances between ajk, ares can be, for example, Euclidean. In SQANN construction, however, you might have noticed that the "distance" expressed by eq. (5), i.e., only the between the activations at the relevant layer (during layer check for admission, it will be layer k; during collision, it will be the collision layer. Let red be any activation az in A stored SQANN currently being used to test new sample a check N₂ activation process during construction. Let black open circles be pattern activations of other sample points already incorporated into SQANN layers. Suppose we test a new training sample. From the main SQANN construction, we have seen that possibilities can occur. (1) Admission of sample to layer NR. if the sample weakly activates the existing node without collision, like the green + mark in fig. 8(B) (2) collision it activates an existing node very strongly, like the red mark (3) filtering into deeper layer if neither occurs, i.e. like blue + mark.
17

---

Under review as a conference paper at ICLR 2022
The concern related to the completion of SQANN construction mainly stems from the collision problem. During collision, layers are torn down, and the red + mark is now added beside the red x mark; thus SQANN now can distinguish similar looking points that might be characterized differ- ently (selective clustering in action). The question is, will the black open circles be reconstructed back to the previously destroyed layers in the same manner they were? The answer is, they will NOT be exactly the same as before since the new activations now need to consider a new node contributed by red mark.
Intuition behind dsa- double selective activation. However, there is a high probability that they will come back in the same order. This is the exact reason why double selective activation has a narrow band of strong activation (small red area fig. 8) distinguished from a band of moderate activation by steep gradient, that is in turn distinguished from near zero activation by steep gradients. With such characteristic, A will have a small red region of strong activation, a large orange buffer region of moderate activation and weak activation everywhere else. The main idea revolves around the fact that the buffer region reduces the probability that existing black open circles are too close to red + mark. Hence, the activation characteristic of each black open circle that has been temporarily removed from the torn down layer (because of collision) will be similar to the activation characteristic it has before i.e. it will not suddenly strongly activate node previously did not activate. For example. if sample weakly activates all existing neurons with values 0.01 before, after red mark added into the layer, u still weakly activates all existing neurons with values 0.01, including red + mark which is close to red x mark. More precisely. if gives rise to before collision (let us denote its component by [v:]]. i 1, I...., ,,,, then after resolving collision, the new activation at the same layer is o' such that [u]] x [ug], 1,..., Ml, although now [o]] has the (mi + 1)-th component since the collided sample has been added into the layer.
Iresgentaris
[B]
After
reauivnng
d
Medeirs solvedo
colisiins
Figure& 8: (A) Visualization of a SQANN node in the activation space, marked as red x. Black open circles are the fingerprints of other nodes that are already integrated into SQANN. (B) A data sample whose activation lies within the red/orange region such as red/blue plus mark is strongly/moderately activating the particular node 1j. Otherwise, is weakly activating it, e.g. green plus mark. dataset and SQANN construction settings are desirable if for every training data u, there is reasonably large such that the probability that complication occurs is pt ze O. When a data sample x' strongly activates node 77 (red plus mark), it will cause collision. If this happens, x' will be integrated into the collision layer as nf. i.c. collision is resolved, hence shifting the effective shape of activation shape of y shape (blue arrow). What used to strongly excite 17 might now excite r/ more strongly (say, if it is nearer to r/').
Probability of complication. Moving from the illustration, let any sample u E U where U is the subset of dataset not yet used during SQANN construction. If its activation v < Tact on all existing nodes (i.e. are not activating any existing nodes too strongly), then, if any new sample 81 activates 8 strongly, then u does not activate 81 strongly. Intuitively, this is because 81 is near 8, which is far from u because there is a buffer region, thus there is high probability that u is also far from $ (note: by far, we mean distance in the activation space). It is only with high probability, because complication may occur, as the following. that there are black circles lying at the edges very close to the boundary of red x mark and red mark happens to activates that region. Let the probability that a complication occurs for u E U be pf-
real dataset, there may be a non-zero probability
Assumption for formal proof. Formal result can thus be stated for p z if we have an assumption on data distribution; recall from the statement of the proposition that p is the probability that
18

---

Under review as a conference paper at ICLR 2022
construction is completed. Suppose W, U are subsets of dataset that respectively have already and not already been used for SQANN construction Let A(x) be the set of points strongly activating (in the illustration, this is the area covered by the small red circle). Also let d (illustrated in fig. 8(B)) denote the minimum distance between u E U and any points within A(w) for any w E W. The required assumption is: for given Tact; Tad and the current ordering of dataset, there exists d > O, such that for any u E U, for any w E W, if the minimum distance between A(w) and u is greater than d, then the probability of complication u E A(wi) for any w1 E W is PL 20 0.
Assumption can be weak; defining the strong assumption. With this assumption, each sample at risk of behaving differently after the addition of red + mark contributes to 1 - p probability of reconstructing the same SQANN layer previously torn down. But if there are many collisions and - 5, then å might blow up easily, especially if there are many data points close to each other. In this case, what we really need to consider are (I) our selection of Tad, Tact might be unsuitable, since they cause too many overlaps. As a rule of thumb, decrease Tad and increase Tact to enable constructions with less overlaps between samples' activations (2) there are too many similar data in the samples that could have been represented equally well with fraction of available data. In this case, it might be better to remove some data from the samples and create smaller subsets for SQANN layer construction, which is beneficial, since the remaining data can be used for validations. With these two fixes, the collision distances between two data points are relatively increased, and, there might exist d such that ph The strongest assumption is thus when there exists d such that pm = 0 for all unused training sample u at any time, i.c. assume no complication.
p Tueu (1 p.)
Proof for p = with the strongest assumption: assume there is no complication at each sample checking step. Suppose layer 1,..., k 1 have been constructed, and j 1 nodes have been added to the latest layer k that is being constructed. Recall and note the difference between (*()) and x <<>> Suppose sample وا5 causes collision in layer S. k - 1. Then we tear down all layers after layer e; here is where we will use the order integrity: let Xtemp = {,<
..., x.., <i-1>> be the set of samples that have been returned to the list of unused indices in the same order they have been put into SQANN layer (likewise Ytemp) e.g. (x <r>> is the first sample in layer e c 1. Then, resolve the collision by concatenating (xx(), y00) to (Ne, of layer e. The reconstruction of subsequent layers will occur the following. Check x <r>> for admission, add it into layer c + 1, then checking x <r+1>> the same way it was added through "layer i construction" process in the main text, and so on up to x: <111>> They will be returned to their previous positions the same way they were added to the torn down layers. By the no-complication assumption, samples will not collide with (z-60, , y())). More verbosely, samples admitted via "admission to N₂" will again be admitted to the same Ng at the same node, while sample admitted through collision with some node ny <j'> will collide with the same node, and there is no collision with the node of activation corresponding to (z(6), y(ب) because of no complication assumption. As a consequence, we can always proceed with ,((++1) even if there is need for multiple reconstructions of layers. Thus, the construction can complete. D
A.5.5 SQANN MAIN THEOREM
Theorem 2. Assume SQANN construction is completed. SQANN achieves arbitrarily high accu- racy training dataset. Furthermore, it is resistant to catastrophic forgetting.
Proof: Let the training dataset be D = X x Y. For any x E X, perform SQANN propagation. If N1, then, by lemma 1, we obtain the arbitrarily high accuracy. If I E Nx, k > 1, the heavy-lifting is in fact already done through the SQANN construction algorithm. By SQANN construction, there exist index j and layer k >> I such that either (I) the activation of x undergoes <ラ> or (2) pushed into layer N& during collision. One of the two must occur, otherwise SQANN construction is not completed, contradicting the assumption. In either process, <j> thus [ux]; = (1 - 7/ <j> II) = dsa (0) = 1 > Fact where recursive computation [1)] = a (1l0-1 - 11] for all i = 1, ...., is performed from layer I = 1,...,A k with Up = x. Since such strongly activated neuron exists at layer k, by SQANN propagation, retrieve y = <ーン where j = argmar, [uxly- To ensure that j is unique, we reasonably assume that there no duplicate with different y values that is admitted through collision, otherwise the dataset is ill-defined (as previously mentioned). The uniqueness is made possible because double selective activation is, by definition, has peak with unique value 1 and in the locality of the peak, to each
admission to N₂ as nh
we have Uk-1 n
19

---

Under review as a conference paper at ICLR 2022
side, it is one-to-one and onto, i.c. there is interval of constant value around the peak. Finally, is indeed the index from argmax, because the peak value of double selective activation is by design.
Resistance to catastrophiceffrgettinng For each new training sample added into SQANN, previous samples still exist nodes stored in the SQANN layers. Previously learned samples will therefore not be forgotten. More precisely, suppose SQANN construction is not yet complete. Let Dr C D be the subset of dataset whose samples have either been admitted layers 1,2,..., k through normal admission or by resolving collision. Then for pa (x( ,y()) E De be exactly queried (l) is exactly 1 and j exist exactly at the layer and node where pk: is admitted into SQANN through the main mechanism of layer construction. By SQANN propagation, as before, we retrieve the output ot <ン> 1f stored exactly at i, node through the main SQANN construction algorithm as well, hence, SQANN remembers the previously stored value. Furthermore, if SQANN construction with D has been completed and new training dataset D' is available, construction with all samples in D by drawing samples in the same order will result in the same SQANN. Any new samples from D' can be admitted into SQANN samples used up. In this manner, previously learned samples from D will remain inside SQANN layers, i.e. like before we can find j node corresponding to each sample previously admitted into SQANN. Hence no forgetting will occur.
by obtaining activation some layer 1 I for some so that
aaza (1|vr-1 - Ti <>>11>
the same way specified in layer reconstruction after all D
A.5.6 SQANN EXAMPIE
SQANN pencil-and-paper example. With ai d2 0.001, 0.5, Tad. Tact 0.1, create 112
SQANN universal approximator for indexed data X = [2x x (3) [0] = [1.2 1.2
and Y = [y,. /2. Vs. y4] [1, 0,,00 O). (A) Show layer stores the fingerprints of (x(1), (3)
and layer 2 stores (v(2), v(() i.c. activations of (())) (B) Use SQANN propagation to
verify that we indeed get zero errors on X x Y. (C) Test SQANN on the test dataset Xtest
[x xz(² zt 1.25
(3) 1.25 1125 -1.4 and plot the results, marking the interpolations made by SQANN.
(A)
(B)
-1
Figure 9: (A) Training samples are closed circles, test samples are in x. (B) Open circles show samples that need interpolation, and the corresponding lines point towards the samples from which they are interpolated.
Note that the following can be matched with the demonstration in jupyter notebook SQANN -mall wtt.sllaaaaplllleeeeeeeeeeeealae example.pyynb.
(A) SQANN construction. We start by putting æ1, I1 into the first layer, so put it into the first layer of SQANN, N1 (I1), a1 = (y1). Now check (I(2), (e()),(2)) y()) for admission to N1: if it activates node in N1, then we filter it to a deeper layer, otherwise. we add into N1, a1 as well. We show
that the latter occurs. We only have one node in SQANN now, so the only possible activation is
(o(2)), (2)x - x11il
0.3344 2 Tad- Admission to N₁ only occurs if (2)2)), Fad
hence it is filtered to deeper layer.
20

---

Under review as a conference paper at ICLR 2022
Now check (I((), .y)) for admission to N1 and get [of²), 5.655 x 10-3. It does not activate any node in the layer, thus this is a distinct sample we will admit into Ni.
Now we check (x((4) ( for admission to N1. We have two nodes, so we have to compute both:
[1 = desa (1]x(()) 7/1
<|>><] II) = 4.450 10-6 and [u]
ddsu (122 7/1 n- [l) 0.5676 >
Tod. It does activate node in N1, which is 7n " Hence it is filtered to deeper layer. We have shown that N1 stores (n)1>
,("),)) >><>>> = =1()).. and not the other samples. Note: since the activation < Tact. no node has been activated strongly (otherwise we will have collision). Recall that each time we store (*))) into Nr. we also store y (*) into ag, so now a = (y("), y())
We have gone through the training dataset once. The unused data are ((x=()) ) = 2, as they have been filtered to deeper layer. We now proceed with layer 2 construction. Since it is empty, we put the activation of x2(2) (not the sample itself) into N1 since no collision occurs, i.e. (2)) ]2, [12 12]- To show no collision, i.e. no strong activations in N1, similar to 121>> = =2 (2) = [[02
before, we compute [o]²]] < Tact. which is previously done, and [uf = 6. x 10-5 < Tact- The last sample x44) is admitted into N2 by checking collisions against all N1 (previously done),
and then checking activation against existing N₂ node, 121 i. m/4) dssa (4) 1/2 1 []
0.00732 < Tad- Hence, it is admitted to N2. We have used up all data points, hence we have shown (2/
N2- =(1/²) (2)
112 ).
(B) Using SQANN propagation on x,(*) k = 1, 3, we get deaa (1lu{ - m 13 = dsa (0) = 1
Tact where j = 1, respectively. Since they are strongly activated, we get y = ai <ラン = y y(),y(0)
for j = 1, respectively. Thus the errors are zero.
For 22) we already previously computed 1/ = [[x]") [rg 12]. which becomes 1/2
<>>>>> thus we
will also get the distance of activation value from itself as stored in N2, deaa (1lu)) 1/2 1> 1>1
(2)
dssa (0) 1. Likewise 22(4)
(C) SQANN testing. For 2!"), we expect its fingerprint to be close to x(1) the activation
of (2) since their values are similar. It turns out we get the following activations in layer I,
ve.1 = [0.5053, 4.938 10-5), not strongly activating layer nodes. For layer 2, we have (v.22) as (isa((lv (1 - m2 (1) = 0.5165 and [222112 = 0. 0.0009896. There is no strong activation anywhere, and this is the case when interpolation is needed. Notice that of all the activations we computed, the strongest are [0 and [ 1. which are the first nodes in both layer, due z(!) and (22)
respectively as we have expected. Using SQANN propagation, we fetch its a values, both of which are 1., thus using linear interpolation, y = 0.5053 1.0.1155+ 1.
0.3053+0..166
The next sample in the test dataset =t does activate the SQANN. We expect it to activate either (3(3)
(2) or the activation of (4)), First, get v1, [5.049 1 0.5059), i.e. no strong activation, so (3)
(2) - 1/2 10 0.9880 > Tact, thus we do get strong activation due to the fingerprint of (4)., By SQANN propagation, we get y = 0² <2>> = (4) = 0.
is not strongly activated. But we get desa (1l={1]
The last test sample needs interpolation too, and we leave it for the reader. The plot of results are shown in fig. 9.
A.5.7 EXPERIMENTAL DATA
For all experimental data, we perform no pre-processing beyond simple normalization. We test all the data in nearly their raw forms.
Experimental data in fig. 4(A). [Top row] Training Training samples X are x E R² with components drawn from uniform r.v. (random variable) 21 I22 = t ~ U(-1., 1.) plus Gaussian
noise. Test samples are X plus r.v. from ((-s, s) where s is the test data spread. Also, y = |||] is indicated by the color. [Bottom row) similar to top row, but X with x R(cos(t), sin(t) ()) where t ~ (0, R 2x), R ~ U008, 1.2) and = cos(t). For both experiments in the top and bottom rows, 128 training data samples are used for SQANN construction and tested on 128 test data samples.
SQANN is tested for regression on Boston Housing and Diabetes Datasets to demonstrate its generalizability to unseen (test) samples. The procedure is as the following, Let Boston Dataset be
D (Rx(8) (1))), k 1, .,, 506). (1) small set of samples D (*) y())), k 1, .... 100} (()))
is used to train SQANN and 9 other regression methods; recall that ordered sequence of data
21

---

Under review as a conference paper at ICLR 2022
Table Comparing MSE on different regression methods for Diabetes dataset. Notations similar to table 1.
Lin
Ridge
Lasso
LSVR
NuSVR
SVR
DTree
kneigh
MLP 108.86
SQANN
o.
58.57
57.60
59.21
94.65
84.34
81.04
76.73
66.95
93.80
e40
54.15
54.18
55.65
70.92
74.39
73.57
40.52
53.06
54.17
53.96
used for SQANN construction. (2) Mean Squared Errors (MSE) values are measured on unused data Drest DVD on all 10 models listed in table 1. We expect large errors some test samples, since the training dataset might be too small to be representative, (3) SQANN's activations are used to collect samples with large absolute errors ex ( [SQANN(z) yol
and we treat them as out-of-distributio (OOD) samples. These samples are considered as new distinct samples 10 models, now with D'. (5) Then MSE is measured again on Drest (yes, there will be partial overfitting). Repeat the process for diabetes dataset, where [D] 442; we also start with [D] 100.
be integrated into D the new training dataset D'. (4) Train the
Results for Boston Housing dataset has been described in the main text.
40 yields relatively competitive performance. using SQANN we inte- grated 218 samples, so D = 318 so 0.719 of the whole ID is used for new training. From the results of linear models, the dataset appears to have somewhat strong linear structure; but non- linear models still can perform better.
For Diabetes Dataset,
B MORE COMMENTS
This section consists of contents that we have considered less important, thanks comments by reviewers in ICLR 2022.
B.1 ON DIRECT GENERALIZATION OF TNN
As mentioned in appendix section A.4.5, our attempts at generalizing TNN directly to high- dimensional imput data are unsatisfactory. Readers can skip this without losing any information required for understanding this paper. For the record, we still list these attempts here. Now, we consider multi-diimeniinal input E R" well-known space-filling curve can be constructed for the input data, generalization is immediately done. Otherwise, variable separability might help with modelling system partially using TNN. For example, given an ideal two-variable model F(x, y) = (z)) Suppose it fits the experi- mental data poorly. Assuming that correct, correction can be applied to g by replacing it with (v)) + (2) to account for the errors. Computing NN)) = F(z.g) - g(y) yields the values to be used for weight computations in our triangular construction. Clearly, error correction can be performed on f similarly, though we now have to determine how to allocate values to each factor. One way to order multi-dimensiona. data samples will be ordering by the magnitude of the vector = [x1,,..,, xxx]" - [r, 22, .., |N where = / The sign of can be generally dealt with once we include discrete variables (such as binary variables) in the vector, where weights could be toggled to different values according to the discrete variables. More generally, weights that vary with a continuous variable may be a powerful modification to the current method of construction which uses constant weights. This will lead the loss of linear ordering, in the sense of "larger" vs "smaller" in the inequality of real number, leaving us with more relaxed conditions, possibly partial order. However, they will not be within the scope of this paper.
1 with one dimensional output y for TNN. If the
f())
For now, assume we have already defined a linearly ordering for the dataset, we can now generalize the construction. Now W R where N, n are still respectively the number of data samples and dimensions. Similar to the earlier sub-section computing weights, (k+1)+1) + b) N
and (#z + - a. By subtracting them, now get [W (x+++))] N 2a.
Spelling it out and rewriting component-win wise difference as A (k) - (k+1), we have
F W (b) 2a
(6)
22

---

Under review as a conference paper at ICLR 2022
Since A (k) can be zero, we only need to carefully choose Wx 1 to satisfy equation (6). This is simply done by setting
0
if if Mk) =0
W WN-
(7)
2a N
otherwise
where N1 is the number of non-zero A (k) over all i for the particular k. Finally, by - a- WN 1x and likewise A in the same manner.
The difficulty is in selecting the choice of the linear ordering. We are not able to provide any sensible ordering for high dimensional dataset, though a meaningful ordering we may still exist. Thus we ask does there exist any linear ordering that TNN can be used for high accuracy classification? The proposition indicates that there is.
Given a standard DNN for C classes classification with atr training accuracy, then there exists linear ordering for TNN to achieve atr accuracy. Test accuracy atest of DNN can be achieved by TNN with high probability through squeezed linear ordering.
Proposition 4
Proof: In the stringent case, classification is performed by taking = orgmaral where y = NNN), x the image to be classified, and y E HC the output from the last layer of the neural network DNN N. Suppose C' = 10 and 0, 1, ......9. 9. Partition the unit interval [O, 1] such that the ten classes are evenly distributed, i.e. I = lc. e+1]. For each class c, collect all the training data samples that are elassified as by the DNN, denote this set P.. Then perform the following linear ordering: map Pe to c+00.5 where Pe is the data point with the highest ye component at the last layer Do (ri) DN (r.)) Then, continue constructing the linear ordering Pe T1 <2 .... so that Tk < r whenever (NN 2 N i.c going to the right of the interval, a data sample has the less probability of being classified as c and it is also more likely classified as c + than - 1 if c is excluded. Let L. be the remaining set of data samples classified as c, i.e. Pa U {p=]). Set < l2 < l < Pc so that [&+1 lx whenever DNN,(l)) 2 DNN The linear ordering for training dataset is done and TNN will yield aty-
C of DNN. Let R. {ri P.
Squeezed linear ordering. We can map the ordered samples to the interval Ie arbitrarily, so long as the order is preserved. However, placing each Re, Le nearer to Pe increases TNN chances of achieving arest: iee. we squeeze the linear ordering towards the centers P.. for all c. This is because, ordering test data samples into the linear order made by training samples, where test sample has max Ye (thus likely classified as c), but lies heyond the extreme end of R. or Le- Conversely, such sample is beyond the extreme end of Le+1 or Re-1 respectively. By squeezing, large spaces (the grey area) are reserved between the extreme ends of two classes. Data samples could fall into these intervals, and we reduce the possibility of misclassifying edge cases, e.g. by letting users treat them separately when TNN indicates that these samples lie in the grey area.
may get the situation
Since TNN can achieve arbitrary accuracy on the training dataset, we will recover the same accuracy as the DNN from the newly ordered training data. We do not quantify the exact probability of attaining high atest because distribution of test dataset may not be exactly the same as the training dataset. With DNN as the encoder mapping the raw data into the unit interval is ironic since the hardwork is already spent on training the deep neural network. However, this shows that, in principle, there exist encodings that can achieve linear ordering for TNN attain high accuracy classifications. Notes: when clashes occur, e.g. T and r have the same DN N. components, for the purpose of this simple construction, simply randomly assign order between them. We only aim to prove the existence of linear ordering here.
B.2 MORE REMARKS ON SQANN
Regarding the pseudo code. Mapping names from pseudo code to python code: sample_collection
collection + layer___samppppp s_ssamlle.ooleeeee collection.
push node push_node_to. la_ee.
forward_cons - forward.to.layer..k
ssig + STOP SIGNAL
23

---

Under review as a conference paper at ICLR 2022
collision - COLLISION.
remove remove_inde - remove sideeeinde_tttoee to laaer_node check_signal, new_noder and update. are placeholders for simple check and update sequences in python code.
24


Reviews:
[{"review_id": "jawUNZNk6Jk", "reviewer": "Reviewer_EKmc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed methodology constructs the neural network according to the ordering of the input data and leaves a fingerprint for each data point in each neuron. No optimization is used in constructing the network. This is confusing and limits the applicability of the neural network.\n\nThe authors write that “a neuron in SQANN corresponds exactly to a data sample as SQANN stores its “fingerprints” as neurons’ nuclei”. In this case, how will the neural network be interpretable if it is trained on millions of data units? Will the proposed neural network still have millions of neurons and learned parameters? This questions interpretability. Also, how will the neural network conduct prediction for unseen test data?\n\nThe proposed methodology is very limited in dealing with high dimensional data. It is not clear how high dimensions can be incorporated in this modeling. This is an important issue that should not be overlooked. \n\nUnder the Subsection Generalizability to test dataset, are the authors pointing to using test data in training?\n\nThe experimentation is also limited and not convincing.", "summary_of_the_paper": "This paper propose two new NN architectures, namely TNN, and SQANN. They claim that these networks are resistant to catastrophic forgetting, are interpretable, and are highly accurate.", "summary_of_the_review": "There are several key issues about this paper that are not well-supported and are not clear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c_5aYI55Dh1", "reviewer": "Reviewer_Xw97", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe idea of making neurons reflect training data seems new to me. The authors also tried hard to give readers enough intuition to understand what they are trying to achieve in this paper.\n\nWeaknesses:\n\n1. Although the authors tried to guide readers by intuitive explanation in the paper, they did this at the expense of sacrificing too much rigor and clarity in their statement of the theoretical results. For instance, Theorem 1 is too loose as a theorem. One might write a theorem like this in the introduction but not like this in the main text. Further, what does \"using TNN constructed with $D \\cup A$\" mean in the statement of Proposition 2, given $D$ being the training and $A$ being a subset of the test data? As another example, the word \"synapse\" does not mean much to a machine learning person who knows nothing about neurology.\n\n2. As new as TNN and SQANN look, I highly doubt if constructions alike are really helpful for interpretability issues. The type of interpretability that TNN and SQANN try to garner, at least to me, is the ability to memorize the training data and then tell if a new test data has a similar counterpart in the training data. I understand interpretability is not a well-defined mathematical concept so what it really means depends on the audience. But I do think the authors should write what \"interpretable NN\" means in this paper upfront as there are too many different types of interpretability people are talking about.\n\n3. In the kind of interpretability that TNN and SQANN are pursuing, TNN only works for input in $\\mathbb{R}$. But with input in $\\mathbb{R}$, no matter how complicated the neural networks are, interpretability is not a big issue. \n\n4. One has to constantly refer to the appendix even to understand how SQANN is constructed, the main algorithm of this paper.\n\n5. No running-time analysis or empirical evidence is provided. The test examples are relatively simple (small data and low dimension). The main benefit of neural networks is the high prediction accuracy and scalability to high-dimension and big datasets. With this premise, interpretability then becomes important. Without showing that these new constructions can achieve state-of-the-art prediction accuracy in high-dimensional datasets, I do not see the importance of interpretability.\n\nFurther questions:\n\n1. Is it necessary to write a whole paragraph on \"Generalization to n-dimensions\" for TNN? Maybe it is better to write one sentence and leave all the appendix as the authors did not think their solution is satisfactory in the TNN part.\n\n2. If one uses SQANN in a one-dimensional setting, how does it compare to TNN? Does it also enjoy the properties that TNN has?\n\nSome minor comments:\n\n(1) It would be extremely helpful if the authors could create hyperlinks to all the sections/equations/figures that they refer to.\n\n(2) In the statement of Lemma 1, \"First layer of SQANN\" should be \"The first layer of SQANN\".\n\n(3) The appendix contains many typos and needs further proofreading.", "summary_of_the_paper": "This paper proposes two new neural networks (NN) construction schemes that aim at better interpretability, in the sense that (1) the NN should always memorize the training data; (2) the NN can roughly tell if a new test data has any similarity to any data in the training sample. The authors also provide approximation error bounds on both training and test data under certain assumptions. Some numerical examples are shown to support their theory.", "summary_of_the_review": "I do not doubt the theorems in this paper have any issues. But I do have trouble understanding the motivation of this work, as I have detailed in my weakness section. Without showing a learning algorithm achieves state-of-the-art prediction performance, I do not get why interpretability is of concern. As a result, I do not see how this work is a significant contribution to the ML community and would not recommend the paper to be published by this year's ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WezA71qjFv1", "reviewer": "Reviewer_KGST", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A very similar method for \"constructing\" neural network was presented by Szymanski.etal \"Deep, super-narrow neural network is a universal classifier\".  As far as I can tell, the requirement of enforcing an ordering of training point is equivalent to  Szymanski's projection of the data onto a line...which in effect order the points...and the manner of dividing the points into classes by stacking classifies side by side along the projection seems similar.  That paper shows that the choice of ordering of the inputs has a massive effect on generalisation.  So my question is - how is the choice made in the method proposed in this paper?  Authors claim that ordering of inputs gives rise to interpretability (though it is not clear to me how exactly that works)...but if the ordering is arbitrarily chosen by the user, then this gives nothing more than what the user already knows about the data (when making that ordering choice).\n\nAs far as I can tell, the catastrophic forgetting resistance claim boils down to ability of the model to perform correct approximation on a training dataset supplemented with new data points.  Is this understanding corrct?  Doesn't that imply that some points (presumably at least the two before and after the new point in the sequence) need to be known for the adjustment?  Does this involve adding more architectural elements (presumably at least one more neuron) to the architecture?  Catastrophic forgetting is about ability of a model to learn a new task, often out of distribution of the first task, without having access to training data of the previous task.  Not sure how the arguments presented in this paper relate to catastrophic forgetting.\n\nI am very confused by the claim of generlisation of the proposed model.  My reading of it is that authors show that the error on a test point is bounded, but not by an arbitrarily small number, and the suggestion is to incorporate the test points that have high error into the train data.  Is this high level assessment of the presented proof correct?  If that's so, then wouldn't the model \"generalise\" best if all test points were incorporated into the training set....and wouldn't every model out there \"generalise\" by this standard by simply including the points with test error into the training data?\n\n", "summary_of_the_paper": "The paper presents a method for constructing neural networks where the architecture and the weights are derived analytically from the structure of ordered inputs instead of iterative training to provide arbitrary accuracy on training data.  Authors make a number of unconventional arguments about their architecture being resistant to catastrophic forgetting, and generalisation ability of the proposed model.", "summary_of_the_review": "The proposed model of progressive constructive of neural networks does not seem to be novel and has been shown to provide universal approximation as well as have limits, which the authors of this method do not seem to be aware of.  While the resulting architecture is capable of arbitrary accuracy of approximation on the training data, I don't see that it would generalise or cope with catastrophic forgetting, at least according to the typical notions of what constitutes generalisation and catastrophic forgetting in machine learning.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "QtFxGSjIa2k", "reviewer": "Reviewer_v56a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. This paper provides two new proof of the universal approximation theorem by construction. To my knowledge, the constructions are novel and interesting. \n\nWeakness: \n1. The writing should be improved for proofreading.\n1.1. Some figures are far away from the paragraphs describing them. For example, the double selective activation is defined on page 5, while the illustration of this activation appears on page 3 (Fig 1(A)). \n1.2. Mathematic formula should be more professional. In the 7th line of section 2, x^(k) are n-dimensional vectors and the relation \"<\" is not defined for them. Texts in math formula should not be oblique, such as \"otherwise\", \"argmax\", and \"if\". \n2. The claimed contributions are minor or not supported. \n2.1. Interpretable constructions for the universal approximation is not novel. Chapter 4 of [1] provides a visual (and can be proven rigorously) proof of the universal approximation, which indicates that neural networks can approximate functions using \"almost\" piecewise constant functions. \n2.2. The proof of the universal approximation using TNN only holds for 1-dimensional inputs, as admitted by authors in the paragraph begins with \"generalization to n-dimensions\" at the end of page 4. \n2.3. The claimed “universal approximation” is not exactly the one machine learning cares. This paper only considers fitting training samples, while the universal approximation in machine learning considers approximating a target function on a compact set. \n2.4. The resistance to catastrophic forgetting is not supported by theorems. Authors claim that this contribution is supported by Proposition 1 on page 4, but Proposition 1 has nothing to do with catastrophic forgetting. Catastrophic forgetting is \" the tendency for knowledge of previously learned task(s) (e.g. task A) to be abruptly lost as information relevant to the current task (e.g. task B) is incorporated\"[2], which needs two sequential tasks. Although D and DUA are two training sets, Proposition 1 has nothing to do with catastrophic forgetting in the following sense: i) DUA contains D as a subset, thus the knowledge in the previous task (D) also exists in the current task and forgetting does not exist obviously; ii) the statement of Proposition 1 is trivial since we can always choose A=D’ and the conclusion in Proposition 1 is the same as Theorem 1. \n2.5. The generalization is not the one machine learning cares about. The paragraph begins with “generalizability to test dataset” on page 4 claims that “they (test data) can be incorporated into the training dataset to create a better model with the above-said error upper-bound”. In machine learning, the label of test data should not be used for training. \n\n[1] Nielsen, M. A. (2015). Neural networks and deep learning (Vol. 25). San Francisco, CA: Determination press. \n[2] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.\n", "summary_of_the_paper": "Summary: \n1. This paper proposes two neural networks by construction, i.e., Triangularly-constructed Neural Network (TNN) and Semi-Quantized Activation Neural Network (SQANN). \n2. These two neural networks are universal approximators, which is proven by construction. \n3. These two neural networks are resistant to catastrophic forgetting. \n4. For TNN, strongly activated neurons and half-activated neurons can be identified. \n5. For SQANN, users can identify the samples that are likely out of distribution. \n\nContributions claimed by authors: \n1. TNN and SQANN are proposed. \n2. The universal approximation is proven, using the construction of TNN and SQANN. \n3. Resistance to catastrophic forgetting is proven. \n4. SQANN can identify out-of-distribution samples.\n", "summary_of_the_review": "The contributions are not novel or even not supported. \n1. Interpretable universal approximation is not novel. It is written in a book in 2015. \n2. The resistance to catastrophic forgetting is not supported by theorems. Settings of the claimed theorems are different from that of catastrophic forgetting. \n3. The universal approximation and generalization are not the ones machine learning cares about. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose two new neural network (NN) architectures, namely TNN, and SQANN. The paper claims that these networks are resistant to catastrophic forgetting, are interpretable, and are highly accurate. While the reviewers agree that the idea of making neurons reflect training data is novel, some concerns remain post rebuttal. Most of the reviewers opine that the statements of theorems are unclear, confusing, and hard to interpret (even after the rebuttal and update), thus making it hard to appreciate the contributions of this work. Given this, we are unable to recommend this paper for acceptance at this time. We hope the authors find reviewer feedback useful.
Decision: Reject