Disentangling deep neural networks with rectified linear units using duality
Authors: Chandra Shekar Lakshminarayanan
Amit Vikram Singh
Venue: ICLR 2022 Submitted (2022)

Abstract: Despite their success deep neural networks (DNNs) are still largely considered as black boxes. The main issue is that the linear and non-linear operations are entangled in every layer, making it hard to interpret the hidden layer outputs. In this paper, we look at DNNs with rectified linear units (ReLUs), and focus on the gating property (‘on/off’ states) of the ReLUs. We extend the recently developed dual view in which the computation is broken path-wise to show that learning in the gates is more crucial, and learning the weights given the gates is characterised analytically via the so called neural path kernel (NPK) which depends on inputs and gates. In this paper, we present novel results to show that convolution with global pooling and skip connection provide respectively rotational invariance and ensemble structure to the NPK. To address ‘black box’-ness, we propose a novel interpretable counterpart of DNNs with ReLUs namely deep linearly gated networks (DLGN): the pre- activations to the gates are generated by a deep linear network, and the gates are then applied as external masks to learn the weights in a different network. The DLGN is not an alternative architecture per se, but a disentanglement and an interpretable re-arrangement of the computations in a DNN with ReLUs. The DLGN disentangles the computations into two ‘mathematically’ interpretable linearities (i) the ‘primal’ linearity between the input and the pre-activations in the gating network and (ii) the ‘dual’ linearity in the path space in the weights network characterised by the NPK. We compare the performance of DNN, DGN and DLGN on CIFAR-10 and CIFAR-100 to show that, the DLGN recovers more than 83.5% of the performance of state-of-the-art DNNs, i.e., while entanglement in the DNNs enable their improved performance, the ‘disentangled and interpretable’ computations in the DLGN recovers most part of the performance. This brings us to an interesting question: ‘Is DLGN a universal spectral approximator?’

Under review as a conference paper at ICLR 2022
DISENTANGLING DEEP NEURAL NETWORKS WITH REC- TIFIED LINEAR UNITS USING DUALITY
Anonymous authors Paper under double-blind review
ABSTRACT
Despite their success deep neural networks (DNNs) are still largely considered as black boxes. The main issue is that the linear and non-linen non-linear operations are entangled in every layer, making it hard to interpret the hidden layer outputs. this paper, we look at DNNs with rectified linear units (ReL.Us), and focus on the gating property ('on/off" states) of the ReLUS. We extend the recently developed dual view in which the computation is broken path-wise to show that learning in the gates is more crucial, and learning the weights given the gates is characterised analytically via the so called neural path kernel (NPK) which depends inputs and gates. In this paper, we present novel results to show that convolution with global pooling and skip connection provide respectively rotational invariance and ensemble structure the NPK. To address 'black -ness, we propose a novel interpretable counterpart of DNNs with ReL.Us namely deep linearly gated networks (DLGN): the pre-activations the gates are generated by a deep linear network, and the gates are then applied as external masks to learn the weights in a different network. The DLGN not an alternative architecture per se, but a disentanglement and an interpretable re-arrangement of the computations in DNN with ReLUs. The DLGN disentangles the computations into two 'mathematically interpretable linearities (i) the 'primal' linearity between the input and the pre- activations in the gating network and (ii) the 'dual' linearity the path space the weights network characterised by the NPK. We compare the performance of DNN, DGN and DLGN on CIFAR- 10 and CIFAR-100 to show that, the DLGN recovers more than 83.5% of the performance of state-of-the-arr DNNs, i.e., while entanglement the DNNs enable their improved performance, the 'disentangled and interpretable' computations in the DLGN recovers most part of the performance. We conclude by identifying several interesting future directions based on DLGN.
INTRODUCTION
Despite their success deep neural networks (DNNs) are still largely considered as black boxes. The main issue is that in each layer of DNN, the linear computation, i.e., multiplication by the weight matrix and the non-linear activations are entangled. Such entanglement has its pros and cons. The commonly held view is that such entanglement the key to success of DNNs, in that, it allows DNNs to learn sophisticated structures in a layer-by-layer manner. However, in terms of interpretability, such entanglement has an adverse effect: only the final layer is linear and amenable to a feature/weight interpretation, and the hidden layers are non-interpretable due to the non- non-linearities.
Prior works (Jacot et al. 2018 Arora al. 2019 Cao Gu 2019) showed that training an infinite width DNN with gradient descent is equivalent to kernel method with the so called neural tangent kernel matrix. As pure kernel method, the NTK matrix performed better than other pure kernel methods. However, in relation to 'black box -ness, there are two issues with NTK theory: (i) Issue I: Infinite width NTK matrix does not explain fully the success of DNNs because it was observed that finite width DNNs outperform their infinite width NTK counterparts, and it was an open question to understand this performance gap (Arora et al. 2019), and (ii) Issue II: Since the NTK is based on the gradients, it does not offer further insights about the inner workings of DNNs even for infinite width.
A dual view for DNNs with rectified linear units (ReLUs) was recently developed by Lakshmi narayanan & Singh| 2020) who exploited the gating property (i.e., "on/off" states) of the ReLUs. The

---

Under review as a conference paper at ICLR 2022
Deep Network
Deep Lizcadl Cutzd Network
Irpe
DNN with Ril
Cleing Nicwooks
Trpai
Deep Inear Netwark
Gating Network
Pie sottaittt
Gating Signal
Cueze
Gaing Signal
Inger
DNN winh GaLU/s
Weigtt Network
Urrgot
Irpat =1
INN with Ga
Waigh Natweek
Ourpst
Figure 1: DGN is sctup to understand the role of gating in DNNs with ReLUs. The DLGN setup completely disentangles and re-arranges the computations in an interpretable manner. The surprising fact that constant 1
input is given to weight network of DI.GN justified by theory and experiments in Sections 3.1 and 3.2
dual view is essentially linearity in the path space, i.e., the output is the summation of path contri- butions. While the weights in path are the same for each input, whether or not a path contributes to the output is entirely dictated by the gates in the path, which are 'on/off' based on the input. To understand the role of the gates, deep gated network (DGN) (see Figure I was used to disentangle the learning in the gates from the learning in weights. In a DGN, the gates are generated (and learnt) in "gating network' which is DNN with ReLUs and are applied as external signals and the weights are learnt in a 'weight network' consisting of gated linear units (GaLUs) (Fiat et al.] 2019). Each GaLU multiplies its pre-activation and the external gating signal. Using the DGN, two important insights were provided: (i) learning in the gates is the most crucial for finite width networks to outperform infinite width NTK; this addresses Issue I, and (ii) in the limit of infinite width, learning the weights with fixed gates, the NTK is equal to (but for a scalar) a so called neural path kernel (NPK) which a kernel solely based on inputs and gates. This shifts Issue II on interpretability to that of interpretability of the gates as opposed to interpretability of the gradients.
Our Contribution. We extend the dual view to address 'black box' -ness by completely disentangling the "gating network' and the 'weight network'. Our contributions are listed below.
Disentangling Gating Network. For this, we propose a novel Deep Linearly Gated Network (DLGN) as a mathematically interpretable counterpart of a DNN with ReLUS (see Figure 1] In DLGN, the gating network is a deep linear network, i.e., there is disentanglement because of the absence of non-linear activations. The gating network is mathematically interpretable, because, the transformations from input to the pre-activations are entirely linear; we call this primal linearity.
We present an unnoticed insight in prior work on fully connected networks that the NPK is a product kernel and is invariant to layer permutations. We present new results to show that (i) the NPK is rotationally invariant for convolutional networks with global average pooling, and (ii) the NPK is an ensemble of many kernels in the presence of skip connections.
Dual View (Section 3.1
Disentangling Weight Network. We then argue via theory and experiments that the weight network is disentangled in the path space, i.e., it learns path-by by-path and not layer-by-layer. For this, in Section 3.2 we show via experiments that destroying the layer-by-layer structure by permuting the layers and providing a constant 1 as input (see DLGN in Figure D do not degrade performance. These counter intuitive results are difficult to reconcile using the commonly held "sophisticated structures are learnt in layers' interpretation. However, these experimental results follow from the theory in Section 3.1, In other words, it is useful to think that the learning in the weight network happens math-by-path we call this dual linearity, which (for infinite width) is interpreted via the NPK.
Message. The DLGN is not an alternative architecture per se, but disentanglement and inter- pretable re- re-arrangement of the computations in a DNN with ReLUs. The DLGN disentangles the computations into two 'mathematically" interpretable linearities (i) the 'primal' linearity and (ii) the 'dual' linearity interpreted via the NPK. Using the facts that the NPK is based on input and the gates, and in a DL.G, the pre-activations in the gating network are 'primal' linear, we have complete disentanglement. We compare the performance of DNN, DGN and DLGN on CIFAR- 10 and CIFAR- 100 to show that, the DLGN recovers more than 83.5% of the performance of state-of-the-ard DNNs. i.e., while entanglement in the DNNs enable their improved performance, the 'disentangled and interpretable' computations in the DLGN recovers most part of the performance.
Related Works. We now compare our work with the related works.
Kernels. Several works have examined theoretically as well as empirically two important kernels associated with DNN namely its NTK based the correlation of the gradients and the conjugate
2

---

Under review as a conference paper at ICLR 2022
kernel based on the correlation of the outputs (Fan & Wang 2020; Geifman et al. 2020: Liu et al. 2020 Chen et al. 2020: Xiao et al. 2020; Jacot et al. 2018 Arora et al. 2019 Novak et al. 2018
Lee et al. 2017: 2020b). In contrast, the NPK is based on the correlation of the gates. We do not build pure-kernel method with NPK, but use it as an aid to disentangle finite width DNN with ReI.Us. RRELUU, Gating, Dual Linearity. A spline theory based on max-affine linearity was proposed in [Balestriero et al. 2018; [alestrieroo & Baraniuk, (0188) to show that DNN with ReLUs performs hierarchical, greedy template matching. In contrast, the dual view exploits the gating property to simplify the NTK into the NPK. Gated linearity was studied in (Fiat et al. 2019) for single layered networks, along with a non non-oonniinn gradieadnnn algorithm to tune the gates. In contrast, we look at networks of any depth, and the gates are tuned via standard optimisers. The main novelty in our work in contrast to the above is that DI.GN the feature generation is linear. The gating in this paper refers to the gating property of the ReL.U itself and has no connection to (Srivastava et al., 2015) where gating is a mechanism to regulate information flow. Also, the soft-gating used in our work and in Lakshminaraaaanii & Singh| 2020) enables gradient flow via the gating network and is different from Swish (Ramachandran et al. 20188), which is the multiplication of pre-activation and sigmoid. extensive comparison of finite versus infinite width DNNs. An aspect that is absent in their work, but present in the dual view is the disentanglement of gates and weights, and the fact that the learning in gates is crucial for finite width network to outperform infinite width DNNs. In our paper, we make use of theory developed for infinite width DNNs to provide empirical insights into inner workings of finite width networks.
Finite vs Infinite Width. [ee et al. (2020a) perform
Capacity. Our experiments on destruction of layers, and providing constant 1 input are direct consequences of the insights from dual view theory. These are not explained by mere capacity based studies showing DNNs are powerful to fit even random labelling of datasets (Zhang et al. (016))
PRIOR WORK : NEURAL TANGENT KERNEL AND DUAL VIEW
In this section, we will focus on the dual view [Lakshminarayanan & Singh 2020) and how the dual view helps to address the open question in the NTK theory. We begin with a brief summary of NTK.
NTK. An important kernel associated with DNN is its neural tangent kernel (NTK), which, for pair of input examples x, x' E Rd,,, and network weights θ E Fdm,, given by:
NTK(2, x') = (Veū(z), V V=((z'))
where
(e(+) R is the DNN output. Prior works (Jacot al. 2018 Arora et al. 2019 Cao & Gu 2019) have shown that, as the width of the DNN goes to infinity, the NTK matrix converges to limiting deterministic matrix NTK and training an infinitely wide DNN is equivalent to a kernel method with NTK... While, as pure kernel NTK oo performed better than prior kernels by more than 10%,
Arora et al. (019) observed that on CIFAR-10:
CNTK-GAP: 77.43% I CNN-GAP: 83.30%
where, CNN-GAP is a convolutional neural network with global average pooling and CNTK- GAP its corresponding NTK,. matrix. Due to this performance gap of about 5 6%. they concluded that 20 does not explain fully the success of DNNs, and explaining this gap was an open question.
2.1 DUAL VIEW FOR DNNS WITH RELUS: CHARACTERISING THE ROLE OF GATES
In the dual view, the computations are broken down path-by- path. The input and the gates (in each path) are encoded in a neural path feature vector and the weights (in each path) are encoded in neural path value vector, and the output is the inner product of these two vectors. The learning in the gates and the learning in the weights are separated in deep gated network (DGN) setup, which leads to the two main results of dual view presented in Section 2.2 and Section 2.1.3 wherein, the neural path kernel, the Gram matrix of the neural path features will play a key role.
2.1.1 NEURAL PATH FEATURE, VALUE, KERNEL AND DEEP GATED NETWORK
Consider a fully connected DNN with 'd' layers and 'w" hidden units in each layer. Let the DNN accept input x E Rdd and produce an output de (x) E R.
3

---

Under review as a conference paper at ICLR 2022
DGN (prior woek)
CNN-GAP:800
Giatin/Feaature Network:: Gf
C C+ -G-------------G-
5(x))
max(U,,) c()
CNN-GAP-DGI
Hard: : Gig) (020))
(EL: 79.699 Stundaleee::: 77.1%% 67.0%%)
Soft G(g) Froxpe-ang)
acd sala)
Cacll
of cils
SDON
Weight/Value Network: 6*"
Figure 2: Shows the DGN on the left. Training: In the case of fixed learnt gates, the feature network is pre-trained using V as the oulput, and then the feature network is frozen, and value network is trained with GDGN v as the output. In the case of fixed random gates, the feature network is initialised at random and frozen, and the value network is trained with NOON as the output. In the case of fixed gates, hard gating G(g)) = 1 (2=0) used. Standalone Training: both feature and value network are initialised at random and trained together with ÿDGN as the output. Here, soft gating G(g) network. On the right side is the CNN-GAP and its DGN used in (Lasshminarayanan & Singh 2020). In CNN-GAP, C1, C2, Cs, Ca are convolutional layers, FC1, FC2 are fully connected layers. CNN GAP-DGN, G = 2,3, 1 are the gates layers, the superscripts f, and v stand feature and value network respectively.
-eep(-9-g) is used to allow gradient flow through feature
Definition 2.1. A path starts from an input node, passes through a weight and a hidden unit in each layer and ends at the oulput node. We define the following quantities for a path pt Activity
Ae(r, p) is the product of the 'd- gates in the path.
Value
(p( is the product of the 'd' weights in the path.
Feature
(e(I, p) is the product of the signal at the input node of the path and Ae(I,p).
The neural path feature (NPF) given by (x) (ae p). p = 1,..., pfi) , Th and the neural
path value (NPV) given by ve (va(p), p 1, I....,, Pt) Rda where pfi d is the total number of paths.
Proposition 2.1. The output of the DNN is then the inner product of the NPF and NPV:
(の(x) (x)) ve)
Σ pe(z,))ve(p) PE|P]
(1)
Subnetwork Interpretation of DNNs with ReLUss A path active only if all the gates in the path active. This gives subnetwork interpretation, i.c., for given input E R4,,, only a subset of the gates and consequently only a subset of the paths are active, and the input to output computation can be seen to be produced by this active subnetwork. The following matrix captures the correlation of the active subnetworks for a given pair of inputs x, x, E Rda.
Definition 2.2 (Overlap of active sub- .etwoks....s). The total number of 'active" paths for both and x' that pass through input node is defined to be:
overlap (i, x. x²) e Hpp starts node Ao p) Ao(z' .p) 11H
Lemma 2.1 (Neural Path Kernel (NPK)). Let D E Rdd be a vector of non-negative entries and for U. u' Rd let (u, u')D= = Ene ((i)a(t)ul(()) (i). Then the neural path kernel (NPK) is given by: NPKo(I, x') A (de()))oo(z') = (xx 2 lovorrapp (aar,a))
Deep Gated Network (DGN) is a setup to separate the gates from the weights. Consider DNN with ReLUs with weights @ E Rua The DGN corresponding to this DNN (left diagram in Figure 2 has two networks of identical architecture (to the DNN) namely the 'gating network' and the 'weight network' with distinct weights O' E Rd and @ E Rd The 'gating network' has ReLUs which turn "on/off" based on their pre-activation signals, and the 'weight network' has gated linear units (GaL.ss) (Fiat et al.] 2019 Lakshminarayanan & Singh] (020), which multiply their respective pre-activation inputs by the external gating signals provided by the 'gating network'. Since both the networks have identical architecture, the ReLUs and GaLUs in the respective networks have one-to-one correspondence. Gating network realises der (z) by turning 'on/off" the corresponding GaLUs in the weight network. The weight network realises tee and computes the output (2) (δer (x), ver). The gating network is also called as the feature network since realises the neural path features, and the weight network is also called as the value network since it realises the neural path value.
4

---

Under review as a conference paper at ICLR 2022
2.1.2 LEARNING WEIGHTS WITH FIXED GATES NEURAL PATH KERNEL
During training, DNN learns both (e( as well ve simultaneously, and a finite time char- acterisation of this learning in finite width DNNs is desirable. However, this is a hard problem. An easier problem is to understand in a DGN, how the weights in the value network are learnt when the gates are fixed in the feature network, i.e. how ÜDGN (x) (Øgr (z), ver) is learnt by learning tie with fixed (2). While ÜDGN (a: = (z), wey) is linear the dual variables, is still non-linear in the value network weights @". However, Lakshminarayanan & Singh (2020) showed that the dual linearity is characterised by the NPK in the infinite width regime. We state the assumption followed by Theorem 5.1 in (Lakshminarayanan & Singh 2020), wherein, the (TK x²) ) = (Ver Degav (z), Ver Begga (z')) is due to the gradient of VDON with respect to the value network weights, and the NPK(x, x') = (Øeg (2), Φeg (x')) } is due to the feature network weights.
Assumption 2.1. 0% Lid Bernoulli() ()) over {-a, to} and statistically independent of 6''
Theorem 2.1 (Theorem 5.1 in [Lakshmnnaraynnan & Singh 2020)), Under Assumption 2.1 for fully connected DGN
(T, ( d o NPK ('), as - = d 2(d-1) (1, x') overtap x')
Remark. In the fully connected case, overlap (i, x. x²) is identical for all 1..... din and hence x.) (zr,z')overtall Jovertap(..a.aa) in Lemma 2.1 becomes (r, x²) overlap(2,0r x²) in 2.1 It follows from NTK theory that an infinite width DGN with fixed gates is equivalent kernel method with NPK.
2.1.3 LEARNING IN GATES KEY FOR FINITE WIDTH To BE BETTER THAN INFINITE WIDTH
The fixed gates setting is an idealised setting, in that, it does not theoretically capture the learning of the gates, i.e., the neural path features Φe (x(x). However, the learning in the gates can be empirically characterised by comparing fixed learnt (FL) gates coming from pre-trained DNN and fixed random (FR) gates coming from randomly initialised DNN, and the infinite width NTK. Using a CNN-GAP and its corresponding DGN, [Lakshminarayanan & Singh (2000) showed on CIFAR-10 that (see Figure 2 for details on DGN training and the CNN-GAP architecture):
FR Gates: 87.09% S CNTK-GAP: 77.43% < FL Gates: 79.68% & CNN-GAP: 80.32%
based on which it was concluded that learning in the gates (i.e., neural path features) is crucial for finite width CNN-GAP outperform the infinite width CNTK-GAP. It was also shown that the DGN can be trained standalone (as shown in Figure 2 and is only marginally poor to the DNN
3 DEEP LINEARLY GATED NETWORKS: COMPLETE DISENTANGLEMENT
DLGN: 3 Linnerr=> Pre-activations -> Gates lifting
Primal
(g)(x))
Linear: j(x) (dbr ver)
Dual
The deep linearly gated network (DLGN) has two 'mathematically" interpretable linearities, the 'primal' and the 'dual' linearities. The primal linearity is ensured in via construction and needs no theoretical justification. Once the pre-activations triggers gates, der(() gets realised in the value network by activating the paths. Now, the value network itself is 'dual' linear, i.e., it simply com- putes/learns the inner product ((x) = (dex(2), rer). Gating lifts the 'primal' linear computations in the feature network to 'dual' linear computations the value network. Dual linearity is characterised by the NPK (for infinite width) which in turn depends on the input and gates, and the fact that the pre-activations to the gates are primal linear implies complete disentanglement and interpretability.
Dual linearity is mathematically evident due to the inner product relationship. however, adopting it has the following conceptual issue: it is a commonly held view that 'sophisticated features are learnt in the layers', that is, given that the input E gd is presented to the value network (as in Figure 2. could be argued that the GaLUs and linear operations are entangled which in turn enable learning of sophisticated features in the layers. In what follows, demystify this theory (infinite width case) in Section [.I] and experiments (on finite width networks) in Section and then study the performance of DL.GN in Section 3.2 The layer-by-layer view is demystified by
view via
5

---

Under review as a conference paper at ICLR 2022
showing that (i) constant 1 input can be given to the value network, (ii) layer-by-layer structure can be destroyed. The constant 1 input is meant to show that if the input is not given to the value network then it is not possible to learn sophisticated structures 'from the input' in layer-by-layer manner. In terms of the dual linearity, providing a constant 1 input has only minor impact, in that, the neural path feature becomes ((x; p) = A (lr,p), i.e., it still encodes the path activity which is still input dependent. Since ((x) depends only on gates, the NPK will depend only on the overlap matrix; results in Section 3.1 captures this in theory. Now, could be argued that, despite constant 1 input, the gates are still arranged layer-by-layer, due to which, the value network is still able to learn sophisticated structures in its layers. S3.tion(3.T]aan theory that points out that as long as the correlation of the gates is not lost, the layer-by-laye structure can be destroyed.
3.1 DUAL LINEARITY: NEW INSIGHTTS AND NEW RESULTS
We saw in Section 2.1.2 that dual linearity is characterised by the NPK for infinite width case. In this section, we: (i) cover standard architectural choices namely convolutions with coooallavraggeppollng and skip connections
Theorems 3.2 and 3.3; the prior result Theorem 2. is only for the fully connected case, (ii) present new insights on Theorem 2. by restating it explicitly in terms of the gates in Theorem 3.1] and (iii) discuss how the NPK structure helps in demystifying the layer-by by-leyer view. Note: Results in this section are about the value network and hold for both DGN and DLGN.
3.1.1 FULLY CONNECTED: PRODUCT OF LAYERWISE BASE KERNELS
Theorem 3.1. Les (z)
E
0, 1jw denote the gates in layer I E {1,.... d - for imput x E R.d.. as + 00, we have for fully connected DGN/DLGN:
Under Assumption 2.7 0
Erni Vio
NTKFC (x,2²) 2') + d. 2(d-1) NNKFC(P.. 2') = d 2(d-1) sccaee
(10.2")
II d-1 (G(()))G(("))
w
Product Kernel: Role of Depth and Width. Theorem 3.1 mathematically equivalent to Theorem 2.1 which follows from the observation that overlap(r, x²) 11 (Gı(z), Gi(z")). While this observation is very elementary in itself, it is significant the same time; Theorem 3.] provides the most simplest kernel expression that characterises the information in the gates. From Theorem it evident that the role of width is averaging (due to the division by w). Each layer therefore corresponds to a base kernel CG(()) which measures the correlation of the gates. The role of depth is to provide the product of kernels. To elaborate, the feature network provides the gates (z). and the value network realises the product kernel in Theorem by laying out the GaLUs depth-win wiiee, and connecting them to form deep network. The depth- -wiee layout is important: for instance, if we were to concatenate the gating features as sp(x) (Gr(z), 1,...,,,44 1) E (0, 1 13(d-1)w, would have only resulted in the kernel (v((x), ppxx)) - - a sum (not product) of kernels.
i.e.
Constant Input. This has minor impact, in that, the expression on right hand side of Theorem 3.1
2(d-1) din I G.(z)) i.e. the kernel still has information of the gates.
becomes d- Scale
Destroying structure by permuting the layers. I (G) is permutation invariant, and hence permuting the layers has no effect.
3.1.2 CONVOLUTIOON GLOBAL AVERAGE POOLING: ROTATIONALLY INVARIANT KERNEL.
We consider networks with circular convolution and global average pooling (architecture and notations in the Appendix). In Theorem 3.2 let the eircular rotation of vector E che by "r" co-ordinates be defined as (o, (1) = xri + r). if i + & dm and rot(x,, (i) = ari + r d.n) if i +r > din-
Theorem 3.2. Under Assumption 2. for suitable Ber (see Appendix for expansion of Box):
Ber
dis
NOK (2.
+ dmn
(x, rot('', ,r)) ( as w oc
Ede- (ar rol ovrroopppp(: where the inner "E' is the inner product between x and rol(x', weighted by overlap and the outer
Ede Ehe x(i)rol(x', r) (i)erlla)) x, rot r)),

---

Under review as a conference paper at ICLR 2022
'E' covers all possible rotations, which in addition to the fact that all the variables internal to the network rotate as the input rotates, results in the rotational invariance. It was observed by [Arora et al. are better than vanilla convolutional networks. The
(2019) that networks with seraallavrgge-oooliiie rotational invariance holds for convolutional architectures only in the presence of global-pooling. So, this result explains why global-average-pool
helps. That said, rotational invariance is not new observation; it was shown by Li et al. (2019) that prediction using CNTK-GAP is equivalent to prediction using CNTK without GAP but with full translation data augmentation (same as rotational invariance) with wrap-around at the boundary (same as circular convolution). However, Theorem 3.2 a necessary result, in that, shows rotational invariance is recovered the dual view as well.
die iim overlap(i, æ, rot r)) for a constant 1 input. The key novel insight is that the rotational invariance is not lost and overlap matrix measures the correlation of the paths which in turn depends the correlation of the gates.
The expression in Theorem 3.2 becomes Bey EiT
Destroying structure by permuting the layers does not destroy the rotational invariance in Theorem 3.2 This is because, due to circular convolutions all the internal variables of the network rotate as the input rotates. Permuting the layers only affects the ordering of the layers, and does not affect the fact that the gates rotate if the input rotates, and correlation in the gates not lost.
3.1.3 RESIDUAL NETWORKS WITH SKIP CONNECTIONS (RESNET): ENSEMBLE OF KERNELS
We consider ResNet with '(b + 2)' blocks and 'b' skip connections between the blocks. Each block is a fully connected (FC) network of depth 'dea" and width "w". There are 2 many sub-FCNs within this ResNet (see Definition 3.T). Note that the blocks being fully connected for expository purposes, and the result continue to hold for any kind of block.
Definition 3.1. [Sub FCNs] Let 200 denote the power set of and let J E 2bb denote any subset of [b]. Define the Tth sub-FCN of the ResNet to be the fully connected network obtained by (i) including block,. Vj I and (ii) ignoring block,, Vj $ J.
Theorem 3.3. Let NPK be the NPK of the Tth sub-FCN, and 3%C (see Appendix for expansion of 3% be the associated constant. Under Assumption
2.1 we have:
NTKRES
03 1
NPK J.
00
JE2M
Ensemble. To the best of our knowledge, this is the first theoretical result to show that ResNets have an ensemble structure, where each kernel in the ensemble, i.e., NPK corresponds to one of the 2 sub-architectures (see Definition 3.1 The ensemble behaviour of ResNet and presence of 2 architectures was observed by Veit et al. (2016), however without any concrete theoretical formalism.
Effect of constant 1 input is as before for kernels NPK and translates the ensemble NTK
Destroying structure. The ResNet inherits the invariances of the block level kernel. In addition, the ensemble structure allows to even remove layers. Veit al. (2016) showed empirically that removing single layers from ResNets at test time does not noticeably affect their performance, and yet removing a layer from architecture such as VGG leads dramatic loss in performance. Theorem 3.3 can be seen to provide a theoretical justification for this empirical result. In other words, due to the ensemble structure a ResNet capable of dealing with failure of components. While failure of component itself does not occur unless one makes them fail purposefully as done in (Veit et al. (2066), the insight is that even one or many of the kernels in the ensemble are corrupt and the good ones can compensate.
3.2 NUMERICAL EXPERIMENTS
Section presented theoretical results which demystified the layer- view in value network, in this section we will verify these theoretical results in experiments. We then show that DLGN recovers major part of performance of ofate-of-the-an DNNs on CIFAR-10 and CIFAR-100.
Setup Details. We consider 3 DNN architectures, C4GAP, VGG-16 and Resnet- 110, and their DGN and DLGN counterparts. Here C4GAP is a simple model (achieves about 80% accuracy on CIFAR-10), mainly used to verify the theoretical insights in Section 3.1 VGG-16 and Resnet- 1 10
7

---

Under review as a conference paper at ICLR 2022
chosen for their sfate-of-the-arr performance CIFAR-10 10 and CIFAR- 100. All models are trained using off shellf optimisers (for more details, see appendix[D). The DGN and DLGN are trained from scratch, i... both the feature and value network are initialised at random and trained. In DGN and DIGN, we use soft gating (see Figure 2 so that gradient flows through the feature network and the gates are learnt (we chose 3 10). In what follows, we the notation DGN (arf, =²)) and (LGN(a x²) where x² and x² denote the input to the value and feature networks respectively. For instance, (1, x) means that both the value and feature network of the DLGN is provided the image input, and (1, 1) will mean that the feature network is given the image as input and the value network given a constant as input. DGN .)) and DGN 1) notation works similarly.
Disentangling Value Network. We show that destroying the layer- miyer-by-layer by-layer structure via permutations and providing constant input do not degrade performance. Since our aim here is not state-of-the-art performance, we use C4GAP with 4 convolutional layers which achieves only about 80% test accuracy on CIFAR-10 10, however, enables us to run all the 4! 24 layer permutations. The C4GAP, DGN and DLGN with layer permutations are shown in Figure 3 Once permutation is chosen, it is fixed during both training and testing. The results in Table of Figure show that there is no significant difference in performance between DG æ) vs DGN 1). and (LGN(x, vs DLGN(x, (z. 1), i.e., constant 1 input does not hurt. Also, there is no significant difference between the models without permutations and the models with permutations. These counter intuitive and surprising results difficult to explain using the commonly held 'sophisticated features are learnt layer-by-layer view. However, neither the permutations or the constant 1 input destroys the correlation the gates, and are not expected to degrade performance as per the insights in Section 3.1 This verifies Claim I.
DLGN Performance. For this we choose VGG-16 and Resnet-110. The results in Table II of Figure show that the DLGN recovers more than 83.5% (i.e., 83.78% in the worst case) of the performance of the state-of-ther of-the-art DNN. While entanglement in the DNNs enable their improved performance, the "disentangled and interpretable' computations in the DLGN recovers most part of the performance.
CAGAP
C4GAP-DGN
CAGAP-DIGN
KUU
CC
c
KUUU
OAP
Layer Permutation
((xr)
Layer Permutation
((x)
x
MUUU
((()))
G
O4
G1
04
C1
Table
Dataset
Permute
C4GAP
DGN(2,
DGN(a, 1)
DLGN(a,
DLGN(a, 1)
DLGN(s,1)
No
80.5
77.4+03
77.5+02
75.4-a3
75.4maz
93.66
CI
Yes
77.3tas
77.9
75.9
76.0zus
9-1.40
No
51.8_
47.4
47.3
47.4
48.0
92.66
CIFARIO0
Yes
48.41sa
49.2.n9
47.5210
48.4gm
93.43
Table II
Datasct
Model
DNN
DGN(x,,)
DGN(x, 1)
(LGN(a,
DLGN(2, 1)
DEGN(
VGG16
93.62m2
93.02m1
93.0aa1
87.02m1
87.0aa3
92.94
CIFARIO
1esNetl
94.0guz
93.3
93.2
87.9
87.8
93.40
VGG16
73.4
70.3
70.5
61.52
61.51a1
83.78
CIFARI00
10
72.7
70.8
70.8
62.32na
62.7
86.24
Figure: 3: Here the gates G1, G2, Ga. G4 are generated by the feature network and are permuted as Go Gizy Giz: Gt, before applying to the value network. C1, C2, Ca, C4 have 128 filters cach. Table I and II: All columns (except the last) show the % test accuracy on CIFAR-10 and CIFAR- 100, and % of DNN performance recovered by DLGN is in the last column. Table I: For each dataset, the top row has results for vanilla models without permutations (the results are averaged over runs) and the bottom row has results of 4! - 1= = 23 permutations (except the identity) for each model (the results are averaged over the 23 pernutations).
8

---

Under review as a conference paper at ICLR 2022
CIGAP
C4GAP-DLON-SF
C
((x)
PC ((x))
Table III
Dutaset
CIGAP
CIGAP-IE ENS
CIGAP
CIGAP-16 ENS C4GAP-DLGN-SF
VGG-IGDLGN-SF
(epocha)
(200)
(200)
(2000)
(2900)
150)
(32)
CIFAR-10
22.
63.81
70.0
72.2
84.9
CIFAR-100
38.23
42.2
45.47
47.3
56,3
Figure 4: CIGAP and C4GAP have width = 512 to make them comparable to VGG-16 whose maximum width is 512. The ensemble size is 16 to match the 16 layers of VGG-16. Note that C4GAP in Figure B has width=128. The last two columns show only DLGN-SFL lesser and left it from Table III for sake brevity, All results except for ENS are averaged over runs.
1). We observed the performance of DLGN-SF(st (z, 2)) to be 2%
3.3 INTERESTING FUTURE DIRECTIONS
Motivated by the success of DLGN, we further break it down into DLGN-Shallow Features (DLGN- SF), wherein, the feature network is collection of shallow single matrix multiplications. We compare a shallow DNN with ReLU called CIGAP against DLGN-SF of C4GAP (see Figure 4 and VGG-16 (see Appendix). The results are in Figure 4 based on which we observe the following:
Power of depth in value network and liftiing to dual space. Both CIGAP and C4GAP-DLGN-SF were trained with identical batch size, optimiser and learning rate (chosen to be the best for CIGAP). The performance of CIGAP at 200 epochs is ~ 10% lower than that of C4GAP-DLGN-ST After 2000 epochs of training and ensembling 16 such CIGAPs as CIGAP-1 ENS closes the gap within ~ 3% on C4GAP-DLGNN-
Yet, deeper architecture VGG-16-DI.GN-SF is 10% better than CIGAP-1 ENS.. Note that both VGG-I6-DLGN-S3 and CIGAP-16 6 have gates for 16 layers produced in shallow manner. While in CIGAP-16 ENSN,, '16' CIGAPs ensembled, in VGG- 16-DLGN-SF these gates for 16 layers are used as gating signals to turn 'on/off' the GaLUs laid wise 16 layers of the value network, which helps to lift the computations to the dual space. Thus, using the gates to lift (instead of ensembling) the computations to the dual space in the value network is playing a critical role, investigating which is an important future work.
Power of depth in feature network. By comparing CIFAR- 100 performance of VGG-16-DL.GN- and that of VGG-I6-DI.GN in Figure we 6% improvement if we have deep linear network instead of many shallow linear networks as the feature network. This implies depth helps even if the feature network entirely linear, investigating which is an important future work.
SF in Figure and
DGN vs DLGN In Table I and II of Figure 3 the difference between DGN and DNN is minimal (about 3%), however, the difference between DLGN and DNN significantly large. Thus, it is important to understand the role of the ReLUs in the feature network of DGN. It is interesting to know whether this is simpler than understanding the DNN with ReLUs itself.
Is DLGN Universal Spectral Approximator? The value network realises the NPK which in general is an ensemble (assuming skip connections). The NPK is based on the gates whose pre- activations are generated linearly. It is interesting to ask whether the DLGN via its feature network learns the right linear transformations to extract the relevant spectral features (to tigger the gates) and via its value network learns the ensembling of kernels (based on gates) in a dataset dependent manner.
4 CONCLUSION
Entanglement the non-linear and the linear operation each layer of DNN makes them uninter- pretable. This paper proposed novel DLGN which disentangled the computations in DNN with ReLUs into two mathematically interpretable linearities, the 'primal' linearity from the input to the are-activations that trigger the gates, and the 'dual' linearity in the path space. DLGN recovers more than 83.5% of performance of state-of-the the DNNs on CIFAR- 10 and CIFAR-106 Based on this success of DL.N, the paper concluded by identifying several interesting future directions.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, pp. 8139-8148, 2019.
Randall Balestriero and Richard G Baraniuk. From hard to soft: Understanding deep network nonlinearities via vector quantization and statistical inference. arXiv preprint 11X1v:1810.09274, 2018.
Randall Balestriero al. spline theory of deep learning. In International Conference on Machine Learning, pp. 374-383, 2018.
Anindya Basu and Alexander Kuhnle. PyGLN: Gated Linear Network implementations for NumPy, PyTorch, TensorFlow and JAX, 2020. URL https::://tttttiee http:///w..oo//a////////..acaa// hwtp/a///waaannnnnn/n//m/m/
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, pp. 10835-10845, 2019.
Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang. A generalized neural tangent kernel analysis for two-layer neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint
2019.
Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear- width neural networks. arXiv preprint arXiv:2005. 1879, 2020.
Jonathan Fiat, Eran Malach, and Shai Shalev-Shwarzz. Decoupling gating from linearity. CoRR, 1906. 05032]
Abs/1906.05033 2019, htRl:tttp://aa//..rrgg/////////
Amnon Geifmann Abhay Yadav, Yoni Kasten, Mcirav Galun, David Jacobs, and Ronen Basri. On the similarity between the laplace and neural tangent kernels. arXiv preprint arXiv:2007.01580 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Chandrashekar Lakshminarayanan and Amit Vikram Singh. Neural path features and neural path kernel: Understanding the role of gates in deep learning. Advances Neural Information Processing Systems, 33, 2020.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1rXiv:1711.000 2017.
Jachoon Lee, Samuel S Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. arXiv preprint arXiv:2007.15801 2020a.
Jaehoon Lee, Samuel S Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. arXiv preprint arXiv:2007. 2000:::0007..800,, 2020b.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kernels. arXiv preprint 1rX1v:1911.00809, 2019.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. On the linearity of large non- non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Under review as a conference paper at ICLR 2022
Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang. Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels gaussian processes. arXiv preprint arXiv:1810.05144, 2018.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2018. URL
https::/openrevieee ew net foremt/frum?ii For
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00388/ 2015.
Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. arXiv preprint arXiv: (605.0643,, 2016.
Joel Veness, Tor Lattimore, Avishkar Bhoopehand, David Budden, Christopher Mattern, Agnieszka Grabska-Barwinkk,, Peter Toth, Simon Schmitt, and Marcus Hutter. Gated linear networks. arXiv preprint arXiv:1910.0152: 2019.
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generaliza- tion in deep neural networks. In International Conference on Machine Learning, pp. 10462-10472. PMLR, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 103550161..03530, 2016.

---

Under review as a conference paper at ICLR 2022
On gate
pafit (on)
paln: pa (on)
path Pa pofof)) value: -LL 0.7 activity: feutures
path: pa (off)
vatuec activity:
& 01-03-00
value: aLl 03
valuee -0L-004 104 06 activity: festurec
Off gate
activity: festure:
Teature:
0.2
0.1
0.1
0.3
0.4
0.6
0.6
-0.1
-0.1
0.5
0.5
0.5
8.1-82-00 0.2 0.7
0.1
0.4 107-0-2
0.6
Figure 5: Illustration of Definition 2. and Proposition 2.1 in a toy network with 2 layers, 2 gates layer and 4 paths. Paths pl and p2 are "on' and paths pa and p4 are "off". The value, activity and feature the individual paths are shown. ý is the summation of the individual path contributions.
A FULLY CONNECTED
Here, we present the formal definition for the neural path features and neural path values for the fully connected case in Definition A.I The layer-by-layer way of expressing the computation in DNN of width "w" and depth 'd' is given below.
Input Layer
2x,0(-0)
Pre-Activation
Zz 2 e 1) G e 1)
Ein Obin: lowsl) 2x e 1- 1)
Gating
(e (han)()
Hidden Layer Output
2x e (oour
=
qu 1 1) Groelbour 1) Ei, O ( dom, d) Zr
Final Output
De (x)
Table Information flow in FC-DNN with ReLU. Here, "q's are pre activation inputs, are output of the hidden layers, 'G's are the gating values. E [d - 1] is the index of the layer, lou and in are indices of nodes in the current and previous layer respectively.
Notation A.1. Index maps identify the nodes through which a path passes. The ranges of index maps T/, Iz, E [d - 1] are [dm] and [w] respectively. Ia(p) = =1, 1, E [18]] Definition A.1. Let x E 1dda be the input to the DNN, For this input,
(i) Ao(z, p) A - G2. is the activity of a path.
(ii) (z)
(1
Rh is the neural path feature (NPF).
(iii) 0e A
[p+t] g is the neural path value (NPV).
B CONVOLUTION WITH GLOBAL AVERAGE POOLING
In this section, we define NPFs and NPV in the presence of convolution with pooling. This requires
three key steps (i) treating pooling layers like gates/masks (see Definition B.2 (ii) bundling together
the paths that share the same path value (due to weight sharing in convolutions, see Definition B.3). and (iii) re-defining the NPF and NPV for bundles (see Definition B.4). Weight sharing due to
convolutions and pooling makes the NPK rotationally invariant Lemma B. the architecture.
We begin by describing
We consider (for sake of brevity) a 1-dimensional convolutional neural network with circular convolutions, with dey convolutional layers (1 = 1, I,.... d.v). followed by global-average pooling layer (1 dev + and die (1 dev 2, ... dev + dr. + fully connected layers. The convolutional window size is Wev < dnn, the number of filters per convolutional layer as well as the width of the FC w.
Architecture:
Indexing: Here inn /ou are the indices (taking values in [vo] of the input/output filters. fev denotes the indices of the convolutional window taking values in [Wev]. 4ow denotes the indices (taking values
The results follow direct manner any form of circular convolutions.
12

---

Under review as a conference paper at ICLR 2022
in [d the dimension of imput features) of individual nodes in given output filter. The weights of layers [dev] are denoted by ( lin dout, 1) and for layers [dge + dev are denoted by O (im ooats 1). The pre-activations, gating and hidden unit outputs denoted by qr l (Nivan fous 1). G (iomt,, fout, 1), and Z (fiot Avat 1) for layers = 1, ......dv..
Definition B.1 (Circular Convolution). For E Rd i E [did] and r {o...., dm 1}, define
(i) r. + S din and :r= -Br=itr for + > dia- (ii) rot( 20i r), E [dm]]
(iii) (2.0 (1juut hour 1) = Exerto (Tevs lins lous 1) 22 (ffour @ (fev 1), dine 1).
Definition B.2 (Pooling). Let Gapg (Jfnus Roue, der + 1) denote the pooling mask, then we have Ex.o dev + 1) Eijn. Zr (oors dours dee) Grooo (Ifinet: louts dey + 1), where in the case of global-average-pooling G (ijoues Rours dev + 1) du Viou E [[o], ifour E [dm)]
Input Layer
2x. 2- 1,0)
Convolutional Layers. I [doo]
Pre-Activtion
Gs,te (Kom,, feuts l)
Eneoto tn (ite, liny lout, 1] Es, (Krmn (Vov 1), tin,
Gating Values
Gg.olhour fout: Zq e (foott Rout, l)
1 (q. affrrrshee
Hidden Unit Output
9s,9 ( Hout. Pout 1) Gz (ftmut: fout.
GAP Layer, l = doe +
Hidden Unit Output
Zir es (fout, dev + 1) Emm 2n,es 2n, bout; der) Go (tromss dout, dev + 1)
Fully Connected Layers, I E [die] + (dev +
Pre- Activation
Σ,, (bim fout: 3zz (ins 1)
Gating Values
G Co 1)
1 ( (tta)))))0))
Hidden Unit Output
2x (Rour, (e)))
= e 1): Gz @ 1) E,, (bmm foots d) To. elii
Final Output
Table 2: Shows the information flow in the convolutional architecture described the beginning of Appendix B
B. NEURAL PATH FEATURES, NEURAL PATH VALUE
Proposition B.1. The total number of paths in a CNN is given by pamn = dia (We, (o wold 1),
Notation B.1 (Index Maps). The ranges of index maps L/ Iy", In are [dha), [wer] and [w] respectively.
Definition B.3 (Bundle Paths of Sharing Weights). Let Pomn pm and { [BI...., B pm be
collection of sets such that Vi, j E [Pam]] i + we have B; n B, = 0 and U from B, [ Further, if paths p.p' Bd, then (p) = Iz"
I...., dey and Ii(p) = Ie(p'), VI O,..., dey
Proposition B.2. There are exactly dma paths bundle.
Definition
B.4.
Rdd be the input to the CNN. For this input, ( EpEB, xI (I(
Let x
Ao(x,p)
A (U
(2,0(p)
A
val (Bp)
(
(o i=d+t2
6n A ( [Pc E pmm
NPF
NPV
to (ve (Bp) (a).p e E [pam) E B pm
13

---

Under review as a conference paper at ICLR 2022
B.2 ROTATIONAL INVARIANT KERNEL
Lemma B.I.
N (2,2')
(e, rot(x' r (t (x,,,ot(22")))
due
Σ
(rot(a, r), jorrllp (t.rtttt(((tt))))))))
Proof. For the CNN architecture considered in this paper, each bundle has exactly din number of paths, each one corresponding to a distinct input node. For bundle bp. let bp(t), E [din] denote the path starting from input node i.
Σ
Σ (,(a)( x())=((1A (I. bp)()) Ag (z', bo((()
E
Σ
(i)) (z, b())) Ae (1', (p(,
BE(P)
LiE(d)), M.liaa).--iti -igrr 11
Σ
Σ
(()r (z, bp(()) Ao r),
FEIP
(1...
da-i
Σ
x(i)rot(Z', (i)
Σ
Ae (a, bp(1)) As (rot(e', r), r,,b(())))))))
(E(
PE[P]
rot(x',
vE(d..)
(a,, rot (rrtpp"re))ootttnaaaaatttttt
In what follows we re-state Theorem
3.2
Theorem B.1. Let dev -moml Assumption 2.1 as + oc, with Ber (devoer
for the convolutional layers and oj:
=Srak
Smale for FC layers. Under
vol
2(d.,-1)
OF
2dg
d
2d.r
or
211--1) we have:
Boy din
CORC OOGN
NPR
2
et
Proof. Follows from Theorem 5.1 in
13
C RESIDUAL NETWORKS WITH SKIP CONNECTIONS
As a consequence of the skip connections, within the ResNet architecture there are 2 sub-FC networks (see Definition l). The total number of paths pres the ResNet is equal to the summation of the paths in these 2 sub-FC networks (see Proposition C.I Now, The neural path features and the neural path value are pres dimensional quantities, obtained as the concatenation of the NPFs and NPV of the 2 sub-FC networks.
Proposition C.I. The total number of paths in the ResNet pres dia Eizo (!) (1+2)du-
14

---

Under review as a conference paper at ICLR 2022
Lemma C.1 (Sum of Product Kernel). Let N be the NPK of the ResNet, and NPK9 be the NPK of the sub-FCNs within the ResNet obtained by ignoring those skip connections in the set J. Then,
NPK
Σ
NPK-
JE2M
Proof. Proof is complete by noting that the NPF of the ResNet is a concatenation of the NPFs of the 2 distinet sub-FC-DNNs within the ResNet architecture.
We re-state Theorem
Theorem C.1. Let
Sazis v/0"
Under Assumption 2.1
oc, for Bies (15) + don
( ((1]++)dd--1)
NTKRES e
Σ
BJ mes
NPNK
JE2M
Proof. Follows from Theorem 5.1 in 13
D NUMERICAL, EXPERIMENTS, SETUP DETAILS
We now list the details related to the numerical experiments which have been left out in the main body of the paper.
Computational Resource. The numerical experiments run in Mvidia-RTX 2080 TI GPUs and Tesla V100 GPUs.
All the models in Table I of Figure 3 we used Adam (Kingma & Ba 2014) with learning rate of 3 x 10 and batch size 32.
We tried several values of 3 in the range from to 100, and found the range 4 to 10 to be suitable. We have chosen 3 = 10 throughout the experiments.
In Section 3.2 the codes for experiments based on VGG-16 and Resnet- 110 were refactored from following repository:
For VGG-16-DLGN in Figure and DLGN-SF in Figure the max- max-pooling were replaced by average pooling so as to ensure that the feature network is entirely linear. For the comparison to be fair, replaced the max pooling in VGG-16 reported in Figure By averuge pooling. Batch normalisation layers were retained in VGG-16, VGG-16-DL.GH and VGG-16-DL.GN-SF (all three shown in Figure 8
DLGN of Resnet- 10 was derived from Resnet- 10 a similar manner.
All the VGG-16, Resnet- 10 (and their DGN/DLGN) models in Table II of Figure used SGD optimiser with momentum 0.9 and the following learning rate schedule (as suggested in for iterations [0, 400) learning rate was 0.01, for iterations [400, 32000) the learning rate was 0.1, for iterations [32000, 48000) the learning rate was 0.01, for iterations [48000, 64000) the learning rate was 0.001. The batch size was 128. The models were trained till 32 epochs.
in Table of Figure uses the same optimiser, batch size and learning rate schedule as the models in Table II of Figure as explained in the previous point.
The VGG-16-DLGNN
we used Adam (Kingma & 2014 with learning rate of 10-3, and batch size of 32. This learning rate is best among the set [10 10-2 10 10-03,3 10 10-1) for CIGAP.
For C1GAP and C4GAP Table III of Figure 4
15

---

Under review as a conference paper at ICLR 2022
MODELS USED IN FIGURE 3 AND FIGURE 4 E
64
(()))
((x))
61
((r)
((())
((x)
((
((x)
0:
((5)
((x))
((r)
(()))
Figure 6: Shows the permutations 1 - 12 C4GAP-DLGN in Table I of Figure The top left is the identity permutation and is the vanilla model.
16

---

Under review as a conference paper at ICLR 2022
61
G1L
((30)
05
((00
((x)
42
((x)
((x))
((x))
11
61
61
((
((x)
12
((x)
((x)
Figure 7: Shows the permutations 13 - 24 C4GAP-DLGN in Table of Figure 3
17

---

Under review as a conference paper at ICLR 2022
Ste)
((a)
Global Aug
Globall Aeg. Pool
Gloebal Ang Pool
ReI.U+Arg (1/2)
G12 - GallU+Aog.. Pool
Gia GaLUUAAv Pool
BN
BN
8N
Rel LT BN
G12
G12
Gailll
BNN
BN
6N
BN
ReLU
G1
GULU
GALU
BN
BN
BM
BN
BN
(3..5,
-Cu
RcLU's Avg. Poel (1/2)
Avg. Poal, Gia GalUU+ Ang.
Arg.
Gun GalLU- Avg.
BN
BN
BN
DN
BN
1
KeLU
GULUU
GGilUU
BN
RNN
HN
RN
RN
1
(1.03. 5120
C4
R
BN
BN
BN
BN
BN
C3
C2
ReL.U+Aug (1/2)
Avg. Poal GallUU+Agg Pool
Pool
Gal,U-Ar Pool
BN
BN
BN
1N
IN
(.200
C.
Cr
G:
Cr
RELU
GaLU
GGiLU
BN
RN
1
(1x 200) C
C4
6%
04
C4
ReLU
GaLU
GaLUU
BN
BN
(1033, 201)
Cr
C3
Ral (1/2)
Arg Pool G2 - Goll.I++Arg Pool
Aug. Pool
G1 Gal Alg Pool
BN
(38) C2
C
C4
ReLU
G5 GaLUU
GaLU
RN
(3.33 128)
ReLL:+Avg (1/2)
Arg. Poul Go - Aoig Pool
Aug. Pool
G5 Coil. Aug. Pool
BN
3.69)
C2
RELU
64
Gial.11
GGallI
BNN
BN
BN
BN
[x 3.60
C
Figure 8: Shows VGG-16 (left), VGG-16-DLGN (middle), VGG.I66.L..N--FFiigel
18

---

Under review as a conference paper at ICLR 2022
F CONTINUAL LEARNING
In this section we compare DLGN and Gated Linear Network (GLN) of Veness al. (2019). We highlight two closely related aspects of the GLN (as a result of the context based gating) namely (i) context induced increased capacity(CIIC) and (ii) context induced sparse activity (CISA), These two aspects were present in the work by Veness et al. (2019). yet they were not highlighted because these two become apparent only from the equivalent flattened computational graph (see Appendix F.3) of the GLN (which was not present in Veness et al. (2019)). While Veness et al. (2019) attributed cossie similarity' for the success of GLN in continual learning tasks, we show that cosine similarity itself is not sufficient and context induced increased capacity and context induced sparse activity are also necessary. We show via experiments that
(i) GLN with CIIC and CISA performs well in continual learning.
(ii) GLN without CIIC and CISA performs poorly in continual learning.
(iii) DLGN-SF with increased capacity and sparse activity performs well in continual learning.
(iv) DLGN-SF without increased capacity and sparse activity performs poorly in continual learning.
Continual Learning Task. We compare the continual learning capabilities of the GLN and the DLGN-SF on the permuted MNIST dataset. We first generate 8 different permuted MNIST datasets and then use the same set of 8 tasks to study the continual learning capabilities of all the models.
F.1 NON-COMPARABLE ASPECTS OF DLGN AND GLN
We first list and discuss the non-comparable aspects between the two works.
The DI.GN in our paper a close cousin of the DNN with ReL.Us and is trained by backpropagation. The GLN the other hand is conceived and motivated from the experts setting and every node of the network is trained in a backpropagation free manner.
Training Algorithm:
Learning in the Gates: In our paper, the parameters of the feature network (whose layers trigger the gates) are learnt. In the GLN, the gating is fixed and not learnt. This a very important difference because was shown by Lakshminarayannn & Singh (2020) that the learning in the gates is very important aspect of the DNNs with ReLUs.
Gating Mechanism: In the case of DLGN, gating mechanism exists for fully connected, convolu- tion, residual networks. In the GLN, there are no (equivalents of) convolutional/residual architectures, and such
say the GLN is equivalent to a fully connected architecture. Now, even in the case of fully connected architecture, the gating between DLGN and GLN differ quite a bit. In the case of DLGN, each unit is controlled by gate which corresponds of GLN, for each unit, the number of hyperplanes is equal to the so called context-dimense the binary encoding of these hyperplanes is mapped into one of 2 dimennion contexts which is activated/selected.
hyperplane. In the case
and
F.2 COMPARABLE ASPECTS OF DLGN AND GLN
Despite the aforementioned differences, DLGN and the GLN share the following commonalities (1) separate gating, (2) both models are data dependent linear networks. For the purpose of comparison to be close enough, let consider fully connected DLGN-SF (shallow features) whose feature network initialised at random and not trained. This way the DLGN-SF is similar to the GLN that (a) the gates are fixed and (b) gates generated via random hyperplanes. We now look at the data dependent lincarity part by comparing the computational graphs of the DLGN-SF and GLN.
F.3 FLATTING THE COMPUTATIONAL GRAPH OF GLN AND COMPARING IT WITH DLGN-SF
We now compare the computational graphs of GLN and the value network of DLGN-SF in Figure 9 For the sake of lucidity, in the Figure 9 well in the discussion below, we have chosen input dimension to be 2, layers to be 2 and units per layer (i.e., width) to be In Figure we have omitted the gating part which will be discussed in the text below, along with other relevant details.
19

---

Under review as a conference paper at ICLR 2022
M
HE
11
122
HEZ
GLN
GLN: Plattened Computational Graph
DLGN-SF (Vilue Network)
Figure 9: Shows comparison of the gated linear network (GLN) and the value network of DLGN. For sake of clarity we have ommitted the details about gating which we have described in the text. Here x = (a(1), x22) E IR² is the input. Each of the 'w's 's' is weight vector with 2 components. The bold lines connecting the units with their weight vectors denote their one-to-one correspondence. The 2 thin lines emanating from each w are its components (bold lines denote the entire weight vector).
Left Most Diagram (Figure depicts (without gating) the GLN manner similar to Figure of Veness et al. (2019). Here, i and jı are the 2 units in layer and iz and je are the two units in layer The are weight vectors in R². The figure does not depict the 4 context maps C,, C, Ci.s Cj. where each C.
(ci, 02, C3, c.). For example, if Cj (r) its output will be 5
then unit ji uses weight w and
Middle Diagram (Figure 9) is a flattened version of the GLN in the left most diagram. The aim of flattening Flattening is achieved by expanding the units i1, Ji, i2, context wise into corresponding sub-units i , 22 .5th & = 1,2,3, The function of the context maps can achieved by one-hot encoded gating signals (this is not shown in Figure 9), G. (1)), G Gj (I:), G (z), G Gezu (z), k = 1,2,3, 4.
depict the diagram in the left in the form of standard computation graph.
For instance, if C,2 (x) C3. then we will have G (x) and sga (x) 0, Vk + 3. The output of unit o is G (x) w. x. The output of unit 1 is the summation of the outputs of i f 1 24
Right Most Diagram (Figure 9 shows the value network of a DLGN. The gating signals G G Gigy G3z have been left out in the diagram. The output of unit i is G., (x) W,y
F,4 CONTEXT INDUCED INCREASE IN CAPACITY AND SPARSE ACTIVITY
Two notable aspects about the GL. are the following:
Context Induced Increase In Capacity. Note that due to the presence of contexts, after flatten-
ing, we can see that the width of the GLN gets multiplied by a factor equal to no.of.contexts = goontex--iimnsion In Figure 9 the width of the GLN is 2 (in the left more diagram) and after flattening becomes 2a (in the middle diagram). The number of parameters in a GLN is equal to the number of parameters of the DLGN-SF times the same factor con.eetstttttss.ooooooooooooo... pecoteex-dimmension i..., say in the first layer, instead of w,, and W,, in the right most diagram, in the middle diagram we have w St k 1,2,3,4.
20

---

Under review as a conference paper at ICLR 2022
Context Induced Sparse Activity: by the factor equal to co.ot.ccnssxts = 2comext- in each of the flattened layers only units active for any given input.
Eventhough the flattened width of the GLN gets multiplied
no.of.context
F.5 CONTINUAL LEARNING IN GLN WITH AND WITHOUT CIIC AND CISA
The main reason for the success of GLN in continual learning (in permuted MNIST task) that is highlighted by Veness et al. (2019) is "inputs close in terms of cosine similarity will give rise to similar data dependent weight matrices. Since each task-specific cluster of examples is far from each other in signature space, the amount of interference between tasks is significantly reduced, with the gating essentially acting as an implicit weight hashing mechanism". However, we show in the experiments below that cosine similarity is not sufficient, context induced increased capacity and context induced sparse activity are also necessary.
Experimental Setup. We trained two GLN models namely GLNM-CD4 and GLNM-CDI each of which have 10 one-vs-all GLNs each with 4 layers with 128 units in the first 3 layers, and one unit in the final layer which serves as the output unit. The context dimension of GLNM-CD4 is equal to 4 and the context dimension of GLNM-CD1 is equal to 1. The code for the GLN (GLN training) from Basu Kuhnle (020). While is mentioned in Veness al. (2019) that 98% accuracy is achieved (on single MNIST taksk), we obtain only 94.32%. There are two reasons for this (i) in order to keep the comparison between GLN and DLGN-SS fair, we did not de-skew the dataset (which is the default option while using the GLN in Basu & Kuhnle (2020)) and if skewing is added the accuracy improves to about 96.5% and (ii) we also confirmed with one of the authors of Veness et al. (2099) that 128 units might not be sufficient to achieve 98%. However, since aim here is to understand the trend in continual learning we accept 94.32% as a reasonable ballpark performance. We now compare GLNM-CD4 and GLNM-CD1 in terms of CIIC and CISA.
CTC (GLNM-CD4 vs GINM-CD1). In GLNM-CD4, the flattened width is 128 x 16. In GLNM- CDI, the flattened width is 128 2.
CISA (GLNM-CD4 vs GLNM-CDI). GLNM-CD4 the only 128 out of the 128 x 16 units are active for any given input, i.e., only 1 of the units are active. In GLNM-CD1, 128 out of the 256 units active, i.e., 1 of the units are active.
T6
2
Thus we can see GLNM-CD4 is GLN with CIIC and CISA, and GLNM-CD1 is a GLN without CIIC and CISA. The results are Tables A, and B where the entry in 8th row and jth column stands for the test accuracy in Task-j while training Task-i. All the results are based on 3 runs.
Table A: GLNM-CD4
Test
Test 2
Test 3
Test
Test 5
Test
Test 7
Test 8
Train
94.322a1
Train 2
92.98 s021
94.27=007
Train 3
92.372um
93.632006
94.52
Train 4
91.41 1ums
92.55
13.15
94,48gm1
Train 5
89.53
91.70=0027
92.33+aa2
93.41
94.31
Train 6
86.99
90.94
91.07
91.84
93. 14.1
94.20
Train
84,91
89.94=0.4
90.14:20.
91.21=os 90.41
93,06=028
94.122000
Train
82.86
85.53
88.22
89.00zas
90.49
92.11
92.62am0
93.982031
Table B: GLNMD-CDI
Test 1
Test
Test 3
Test
Test 5
Test 6
Test 7
Test 8
Trainl
90.28
Train2
81.89=145
90.29
Train3
75.192445
84.18
89.42034
Train4
72.69=cm1
80.95
84.69+157
91.3+as
Train5
61.02_2..
13.1622.5
76.062229
83.14
91.42=a7
Train6
53,22-247
70.77+255
73.63+1m
68.56+0s8
84.92++aa
91.66=0.3
Train7
49.21
61. 2.
64.35
72.85
79.13
83.25
91.29200
Train8
42.49-731
49.30+a.2
61.62+297
59.35
73.58=s2
78.97=656
-1.87
90.19+2a1
Observation. From Tables A and B, we can observe that the GLNM-CD4 with CIIC and CISA
performs well and GLNM-CD1 without CIIC and CISA performs poorly, from which can conclude
21

---

Under review as a conference paper at ICLR 2022
that cosine similarity is not sufficient, context induced increased capacity and context induced sparse activity are also necessary.
F.6 CONTINUAL LEARNING IN DLGN-SF WITH AND WITHOUT CAPACITY AND SPARSE ACTIVITY
Cosine Similarity holds in DLGN-SE. Even in DLGN-SF, the gates nothing but hyperplanes and similar inputs should trigger similar gates and activate similar data dependent linear networks. Therefore cosine similarity holds true for DLGN-SF as well.
Increasing Capacity DLGN-SE. In order to make capacity of DLGN-SE comparable to that of let the width of the DLGN-SF to be WDLGN-SF WGLN x 2eonlex--dimensien, Thus we consider two models namely DLGN-SF-1120 6 which is comparable GLNM-CD4 and DLGNNSF- 12x2 which is comparable GLNM--D1, DIGGN-SF- 128x1 has 10 one-vs-all DI DIGN-SFs, each with layers with 128 x 16 units in the first three layers, and one unit in the last layer which serves as the output unit. DLGN-SF-I28% has 10 one-vs-all DLGN-SFs, each with 4 layers with 128 x 2 units in the first three layers, and one unit in the last layer which serves as the output unit.
GLN with WGLN and given context- dimeestn,
Inducing Sparse Activity in DLGN-SF. The gates in DLGN SF-SFFFRRR are triggered by the feature network which contains WDLGN-SF hyperplanes. Since we are considering DLGN-SF with randomly initialised (from symmetric distribution) and fixed feature network, it is reasonable to expect that WangNes of the gates are active/on. Thus in order to induce sparse activity we sort the pre-activation signals of the gates and activate the gates corresponding to the top 2ommerdneeroo pre-activation signals. Thus, 16
for the case of DLGN- RE--SF-IT 128x16 to ensure sparse activity we activate only the top 128x 128 gates. For the case of DLGN-ST-128x2. since we are comparing it with GLNM-CDI whose context
dimension is 1, we need trigger the gates.
to active, and hence we just let the pre-activations to directly
Experimental Setup. We trained with SGD, batch size of 32 and learning rate of 0.32 (this is equivalent learning rate of 10- and batch size of 1). All the results are based on 3 runs. Table C shows DLGN-SF- 12-x128x6 16 with sparse activiy, and Table D shows DLGN-SF- 122x11216 without sparse activity. Table E shows DLGN-SFF-12x> which does not sparse activity.
Table DLGN-SFF-128116
Sparse Activity: Top 128 Gates Active
Test
Test 2
Test 3
Test 4
Test 5
Test
Test 7
Test 8
Train
96,08 aum
Train 2
95.77 292s
96.522007
Train
94.61 #0.7
96.45
96.55
Train
93.64
12
96.45
96.7 73
Train
91.43
95.93
96.32.nns
26.41.om
96. 28
Train
90.13
95.66
95.66=0004
96.08 20.2
95. 05.94=00.8
96.50=a.
96.95+a1
Train 7
89.34 1.18
95.03
95.72
95.73.019
96.07
96.70..0s
96.92
Train 8
88.87
94.27=0.30
95.46=aam
95.77=0015
95.98
96.72=004
96.57
96.79*014
Table D: DLGN-SFF-12XL
No Sparse Activity: 128x16 Gates Active
Test 1
Test
Test
Test 4
Test 5
6
Test 7
Test
Train
96.87amm
Train 2
96.07=am
97.04+am
Train 3
90.60дии
96,35
97.061mm
Train 4
81.85
11.89
96.21 ama:
96,97
Train 5
62.41
84,61 4375
93.194a41
95.72
96.92anm:
Train 6
59.0224m
66.48
80.7441m
91.732128
96.45
97.1220ok
Train 7
46.57+1011
45.194236
66.82+157
87.26+1.4
94,36+029
96.63+a.10
97.2+000
Train 8
29.15
$4.17+426
62.69+3mz
76.67+423
82.23+306
90.83+4m1
92.94+433
93.96=454
22

---

Under review as a conference paper at ICLR 2022
Table E: DLGN-SF-128x2
No Sparse Activity: 128 Gates Active
Test 1
Test
3est
Test
Test
Test
Test
Test
Trainl
96.29=o.1
Train2
94.85-c.7
96.62+008
Train3
87.44_o.
94.96a0e
96.42an2
Train4
76.54_147
87.6220.2
93.74iuuz
96.59
Train5
60.69=1.13
80.16
87.55+433
94.58+0
96.442044
Train6
40.89 =7.0
71.62:1x.s
83.75
94.46.110
96,51
Train7
37.86 2.78
46.3
61.7522884
76.75
2..4
94.5ma.
96.5520.17
Train8
29.7
42.08
55.73
63.63
70.29
82.15
90.7
96.36+006
Observation. From Tables C with Tables D and E. we can observe that the DLGN-SF-128x SF-128x with CIIC and CISA performs well and BLGN-SF-128xXI without CIC and CISA as well as DIGN-SF-128x2 without CIIC and CISA perform poorly.
F.7 IMPROVING CONTINUAL LEARNING IN DLGN-SF BY INTRODUCING SPARSE ACTIVITY
We now show that DLGN- SF- 128x2 can be made to perform well if we introduce sparse activity. For this we sort the pre-activations to the gates and then trigger only the top 16 gates. The results are shown in Table F.
Table F: HLGN-SF-128x2
Sparse Activity: Top 16 Gates Active
Test 1
Test
Test 3
Test
Test 5
Test 6
Test 7
Test 8
Train1
94.48_6.1
Train2
92.86=041
98.18
Train3
91.61 2ons
94.262044
95.562013
Train4
91.35
93.8
95.14
95.47
TrainS
89.65=o.0
93.41
94.6+0.1
94.87+031
95.44=204
Train6
88.71 =0:00
92.5
93.58
94. 04.
94.92.1015
95.54
Train7
24.59
90.9
92.822222
93.662as4
94.41 gans
95.332000 94.83
95.01 2000
Train8
87.54=0x3
88,84 20.59
92,41
93.57
94.03
94.65 +
95.68
Observation The difference between Table and Table is that the model in Table F has sparse activity, and we observe that it sparse activity ensures good performance.
F.8 CONCLUSTON
We conclude this section on continual learning by saying DLGN- SF with increased capacity and sparse activity performs well in continual learning. A comparison of the continual learning capabilities of GIN and DLGN to find out which is the optimal model out of the two will require a more thorough study which defer future work. Further, perform well in continual learning, the models should be capable of learning many tasks, which naturally requires the models to have more capacity, In the models that perform well above, we that capacity has been built in two ways (i) via the 10 models due to 'one-vs-all' and (ii) the context in the GLN increases the capacity by factor equal to no.of.contexts, and in the DLGN-SF we ensure capacity increase by increasing the width. Sparsity helps to ensure that the different parts of networks are responsible for different contexts and they do not interfere with one another. High capacity and interference minimisation have also been mentioned in Cheung et al. 2019) in the context of continual learning.
23


Reviews:
[{"review_id": "e8Zxxe7_StB", "reviewer": "Reviewer_nByk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n1. Disentangling neural networks in the path space is a really interesting perspective, which may give new insight to understanding neural networks.\n2. By theoretical analysis, the paper proves some novel propositions (e.g., ResNets have an ensemble structure, NPK is rotationally invariant) which might be useful to understand deep learning models.\n\n## Weakness:\n1. Since the paper is largely based on the previous work DGN (Lakshmi-narayanan & Singh (2020)), it would be better to provide more details and formal statements about the prerequisites. Otherwise, it would be hard to follow for readers who are unfamiliar with DGN. (e.g., Definition 2.1 should be a more formal definition with mathematical symbols and formula.)\n2. The paper claims that \"the learning in the weighted network happens path-by-path\" with experiments and theoretical analysis. However, the \"input 1\" experiment and the \"layer permutation\" experiment on DLGN only prove that **DLGN can** be learned path-by-path, and this conclusion doesn't necessarily hold for the general neural network.\n3. The paper claims that \"Destroying structure by permuting the layers\", but actually, according to Figure 3 (C4GAP-DLGN), I think it's somewhat overclaimed. Let q_1, ..., q_4 denote the pre-activation which generate G_1, ..., G_4. And we have q_i = (C_i^f C_{i-1}^f ... C_1^f)(x^f). Since C_i^f C_{i-1}^f ... C_1^f is linear, we can merge (C_i^f C_{i-1}^f ... C_1^f) to a single weight matrix/convolution kernel, which is denoted as P_i. Then P_i is just a common linear operator, and different q_i has its own unique P_i. It's obvious that we can permute them and get similar performance. I don't think it can be called a \"destroying structure\".\n\n## Questions:\n1. In Table I, why is the performance of models whose layers are permutated slightly higher than the original models?\n2. Can the linear network functions replace the non-linear network?  If not, why try to design the network in a linear manner?", "summary_of_the_paper": "The paper extends the framework of the deep gated network to the deep linearly gated network. Based on this extension, the paper investigates the separate part of the network. By theoretical analysis and empirical experiments, the paper argues that the neural network is learned path-by-path instead of layer-by-layer. Also, it present that the neural path kernel has several interesting properties.", "summary_of_the_review": "The paper gives an in-depth theoretical analysis of neural networks. But some of its claims are not well-supported or well explained. The writing of the paper is ok, but the prerequisites of the paper can be more comprehensive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "d7JHe4uthF6", "reviewer": "Reviewer_pMWb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. It is a good idea to adopt deep linearly gated networks (DLGN) for interpreting DNNs with ReLU\n2. This work also offers some theoretical analysis to better understand the proposed method.\n3. Compared to the prior work on dual view, this paper presents more insights and new results for convolutional networks with global average and ResNet\n\nCons:\n1. Did not conduct experiments on complex tasks, such as ImageNet and CelebA. I am curious about what experimental results they will be.\n2. This work did not discuss the impact of $\\beta$ in the soft gate on the experimental results\n3. It is hard for the audience without background to understand some theorems.", "summary_of_the_paper": "This paper proposed deep linearly gated networks (DLGN) for interpreting DNNs with ReLU activations based on a dual view. The proposed framework is able to completely disentangle the ‘gating network’ and the ‘weight network’. Finally, the experiment results demonstrate that DLGN can achieve good performance for classification on two benchmark datasets compared to the existing DNNs.", "summary_of_the_review": "The idea of using deep linearly gated networks (DLGN) with dual view looks novel and interesting. I think this paper should be above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JPBQAKtau1s", "reviewer": "Reviewer_hqwS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The view of this paper is interesting since the success of the DNN is its non-linear ability. And the paper proposed to reduce its non-linearity.\n2. And both the theory and experiments are sufficient to support their claims.\n\nWeakness:\n\nWay too theory. It is a little bit hard to read for me.", "summary_of_the_paper": "This paper deals with the entanglement in the DNN through two steps. First, replacing the rectified linear units (ReLU) in the traditional DNN with Deep Linearly Gated Network (DLGN). Second, demonstrate the weighted network is disentangled in the path space.", "summary_of_the_review": "The problem of this paper deal with is important, and the results show the effectiveness of their proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DuNu3gUgBn", "reviewer": "Reviewer_4UFG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* This work provides an alternative perspective on understanding and interpreting how deep networks work. In particular, the current study overcomes the limitations of NTK framework, by proposing NPK (Neural Path Kernel) for DNNs with gating and dual pathways. \n\n* Beyond interpretbility, what is the functional role of gating? Is there any computational benefit of using a gating network? Recent work in gated linear networks [1,2] have provided functional examples where the usage of gating pathways gives rise to computational benefits such as superior performance in continual learning tasks [1] and implementation of local learning rules [2]. Perhaps the connection to these lines of work can be discussed more clearly.  \n\n* Does the separation of gating network and value network introduce twice as many parameters? Is there any way to reduce the number of parameters in gating network, and can this be used to explain why the performance of the DLGN is a bit worse than SOTA? \n\n* Description in Figure 1 is a bit confusing and not self-contained. Can Fig 1 be more detailed? For example, illustrate \"Gating Signal\", with a small figure of the functional form of G, to illustrate GaLU. (I understand that it becomes clear in Figure 2, but perhaps it can be drawn in Figure 1 too)  \n\n* I’m not sure if the entanglement of linear and and non-linear operations of DNNs being uninterpretable is a strong motivation for developing a new (interpretable) class of networks, especially given many ongoing studies on interpretability methods that can be used directly on deep nonlinear networks. Perhaps, the paper could benefit from an additional clarification on the role of interpretable models (as opposed to interpretability methods on uninterpretable models). \n\n* Concluding a paper with a meta-note that this paper concludes with the question if DLGN a universal spectral approximator makes the paper sound a bit like a long introduction to the next paper. Perhaps it should be stated as an interesting direction for future study. Admittedly, exploring the expressivity and capacity of DLGN is an interesting question in its own right and it's unclear why this is  emphasized only at the end of the paper. \n\nReferences \n\n[1] Veness, J., Lattimore, T., Bhoopchand, A., Budden, D., Mattern, C., Grabska-Barwinska, A., Toth, P., Schmitt, S. and Hutter, M., 2019. Gated linear networks. arXiv preprint arXiv:1910.01526.\n\n[2] Clark, D.G., Abbott, L.F. and Chung, S., 2021. Credit Assignment Through Broadcasting a Global Error Vector. arXiv preprint arXiv:2106.04089.", "summary_of_the_paper": "The authors of this paper proposes a DLGN (Deep Linear Gated Networks), a novel class of deep networks, inspired by a recent dual view where the computation in DNNs is broken into two parts: learning in the gates and learning in the weights. The DLGN disentangles the computations into 2 parts: (1) \"primal\" part between input and the pre-activations in the gating network, and (2) \"dual\" part in the weights network, conditioned on inputs and gates. DLGN’s performance recovers 83.5% of SOTA DNNs. This development may lead to more interpretable deep network models that are also highly performant.  ", "summary_of_the_review": "I think this paper continues an interesting line of research. The goal and the motivation of the paper, and the connection to the existing prior work could be made more clear, but once those are addressed I am willing to change my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines a sum-over-paths representation of ReLU networks, for which learning can be broken into two parts: learning the gates, and learning the weights given the gates, the latter of which being described by the Neural Path Kernel. The paper introduces a dual architecture, Deep Linear Gated Networks (DLGN) that parameterizes these two processes separately. The DLGN is argued to aid in interpretability of ReLU networks, with a main conclusion being that the neural network is learned path-by-path instead of layer-by-layer.

The reviewers generally found strength in the motivation and perspective and thought that the DLGN could serve as a useful architecture for aiding interpretability. Some reviewers found the presentation hard to follow, and others were not entirely convinced by the ultimate conclusions. Overall, the reviewers opinions were mixed.

I believe the ICLR community would generally find interest in the DLGN and the interpretations it might afford to deep ReLU networks. However, the number and strength of the conclusions obtained in the current analysis are rather weak. The conclusion that networks learn path-by-path instead of layer-by-layer was emphasized but the implications were not highlighted, and it remains unclear to me and at least some reviewers what the concrete significance of this observation actually is. Another major claim is that the DLGN recovers more than 83.5% of the performance of state-of-the-art DNNs, but a priori it is not obvious what this number means, or if it is even good or bad performance. A more detailed analysis with additional common baselines, ablations, etc., would really help readers understand the significance of the performance gap.

Overall, this is an interesting direction with significant potential, but for the above reasons I cannot recommend the current version for acceptance.
Decision: Reject