Deep Networks Learn Features From Local Discontinuities in the Label Function
Authors: Prithaj Banerjee
Harish Guruprasad Ramaswamy
Mahesh Lorik Yadav
Chandra Shekar Lakshminarayanan
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
Feature learning
Interpretable
Local Discontinuities
Deep learning theory
Deep neural architectures
Supervised learning
Primary area: interpretability and explainable AI
TL;DR: Deep neural networks outperform kernel machines by learning features through discontinuities in label functions during gradient descent training, showing better performance and offering greater interpretability compared to ReLU networks.

Abstract: Deep neural networks outperform kernel machines on several datasets due to feature learning that happens during gradient descent training. In this paper, we analyze the mechanism through which feature learning happens and use a notion of features that corresponds to discontinuities in the true label function. We hypothesize that the core feature learning mechanism is label function discontinuities attracting model function discontinuities during training. To test this hypothesis, we perform experiments on classification data where the true label function is given by an oblique decision tree. This setup allows easy enumeration of label function discontinuities, while still remaining intractable for static kernel/linear methods. We then design/construct a novel deep architecture called a Deep Linearly Gated Network (DLGN), whose discontinuities in the input space can be easily enumerated.  In this setup, we provide supporting evidence demonstrating the movement of model function discontinuities towards the label function discontinuities during training. The easy enumerability of discontinuities in the DLGN also enables greater mechanistic interpretability. We demonstrate this by extracting the parameters of a high-accuracy decision tree from the parameters of a DLGN. We also show that the DLGN is competitive with ReLU networks and other tree-learning algorithms on several real-world tabular datasets.

Published as a conference paper at ICLR 2025
DEEP NETWORKS LEARN FEATURES FROM LOCAL DISCONTINUITIES IN THE LABEL FUNCTION
Prithaj Banerjee, Harish G. Ramaswamy, Yadav Mahesh Lorik, Chandrashekar Lakshminarayannn Indian Institute of Technology Madras, India
{prithaj, maheshyadav/@cse, {hariguru, chandaaaccarrreeaaaaddaaaaaaaaa iitm. ac. in
ABSTRACT
Deep neural networks outperform kernel machines on several datasets due fea- ture learning that happens during gradient descent training. In this paper, we an- alyze the mechanism through which feature learning happens and use notion of features that corresponds to discontinuities in the true label function. We hypoth- esize that the core feature learning mechanism label function discontinuities at- tracting model function discontinuities during training. To test this hypothesis, we perform experiments on classification data where the true label function is given oblique decision tree. This setup allows easy enumeration of label function discontinuities, while still remaining intractable for static kerne//linear methods. We then design/construct novel deep architecture called Deep Linearly Gated Network (DLGN), whose discontinuities in the input space can be easily enumer- ated. In this setup, we provide supporting evidence demonstrating the movement of model function discontinuities towards the label function discontinuities dur- ing training. The easy enumerability of discontinuities in the DLGN also enables greater mechanistic interpretability. We demonstrate this by extracting the param- eters of high-aceuracy decision tree from the parameters of a DL.G.. We also show that the DLGN is competitive with ReLU networks and other tree-learning algorithms on several real-world tabular datasets.
INTRODUCTION
Neural networks and deep learning have demonstrated exceptional performance across diverse do- mains, but their learning mechanism and exact reason for outperforming linear/kernel models still remain largely mystery. Another issue with neural networks is the black box nature the models, by which any parameters (other than the first layer parameters) are difficult to interpret directly.
There have been several tools built to study the learning dynamics of gradient descent and perform post-hoc interpretation of learned models. The most common characteristics of these follows:
Approximate the learning dynamics of neural networks with much simpler setups like kernel methods (Arora et al. 2019) deep linear models (Saxe et al. 2014).
Study dynamics of single (or two) hidden layer models under special data and settings to go beyond the settings of kernel methods or deep linear models (Damian al. 2022 Ba
et al. 2022 Nichani et al., 2023).
Approximate the learned non-linear neural model with linear approximations around given query input (Ribeiro et al. 2016 Selvaraju et 2017).
These approaches valuable purpose, but have their share of cons, namely
Deep linear models are incapable of learning non-linear models, and fixed kernel methods are incapable of learning "data-depeenden--feaaues""
Analyses of single hidden layer models for specific data settings like the parity function fail to capture the need/behaviour of multiple layers.

---

Published as a conference paper at ICLR 2025
All post-hoe interpretability approaches based on local approximations suffer from lack of faithfulness guarantees- i.c. how useful/valuable is the 'interpreted' feature to the learned model?
To this end, we introduce and study novel architecture which is more powerful than kernel ma- chines and deep linear networks (i.e. it is capable of learning non-linear data-dependent features) while being 'mechanistically interpretable" (i.e. the model output breaks down exactly into sum of components, each of which has a simple structure). Using this architecture, we gather and provide supporting evidence for claims below that would not be possible without it.
The advantage of feature learning methods (like ReLU networks) over kernel machines lies in their ability align the model discontinuities to the label function discontinuities.
The existence of multiple layers enables the 'local' nature of the discontinuity finding each layer operates in the context set by other layers, and hence, discontinuities that might be minor (or not even exist) in a global context can become significant and drive the learning process.
1.1 OVERVIEW/CONTRIIUUNE
In Section 2
we review recent literature on feature learning in deep networks. In Section we define the concept of label function discontinuities and give an interesting setting where these form natural features and can be easily enumerated. In Section 4 we propose a novel architecture called the deep linearly gated network (DLGN). In Section [ we provide supporting evidence for our claim of label function discontinuities attracting model features during training. In Section 6 we demonstrate that the DI DL.GN architecture enables better control and is more amenable to being mechanistically interpreted by constructing a decision tree directly from the parameters of learned DL.GNs. In Section
/ we give empirical results on some binary classification datasets and show that feature learning methods (like the DLGN) outperform fixed kernel methods and are comparable to ensemble methods like random forests.
1.2 NOTATION/SETUP
We consider a binary classification task, with training set S = (1x1881). v1)...................... (x,, vn)} where
x; E IRd is drawn from some distribution D, and yi = (x,) E {+1, The ultimate goal is to generalize well by finding a classifier f [+1,++, such that f(x) (x) with high probability over x drawn from D. For any positive integer a we denote the set 21.2..... a) as [a].
We denote by 1 (conditionn) as a {0, 1} valued variable that takes 1 if the condition is true and 0 otherwise.
RELATED WORK
A brief overview of some related work in the topic of neural network feature learning is given below.
Neural Nets as Kernel Machines: An infinite-width neural net at initialisation equivalent a Gaussian processes with particular kernel, and learning can be viewed as computing the posterior of the label function, based on the training data likelihood (Lee et al.] 2017). Jacot
al. (018) introduced the Neural Tangent Kernel (NTK), which approximates an infinite-wid neural network trained with gradient descent to kernel machine. Arora et al. (2019) analysed the width at which neural networks act approximately similar to kernel machine, this number while being finite is much larger than typical widths seen in practical networks. [Daielly & Malach (2020) showed that neural nets are not just equivalent to kernel-methods, and are capable of succeeding in situations where kernel methods provably fail. The constant NTK setting simply asserts no feature learning takes place, in contrast to empirical results. Tools to study the interesting and active area of research. (Atanasov
change of the NTK features during training
et al. 2022 Damian et al. 2022 Hu et al. 2020 Ba et al. 2022 Chizat et al. 2019 Liu et al. 2020).
Feature Learning in Neural Nets: ReLU networks (Fukushima 1969) have been a workhorse of
deep learning and are the current focus of several theoretical results that aim explain the success of
2

---

Published as a conference paper at ICLR 2025
deep learning over kemel methods (Ghorbani et al. 2020 ReLU networks can learn important data characteristics known as "features", though the term can be ambiguous. Neural networks undergo feature learning, developing data representations during the process (Damian et al., 2022) Some research proposes that neural nets learn low-frequency functions (Rahaman et al. 2019 Cao et al.)
2019) and simple features (Shah al., 2020) first while training, and the iteration progresses, gradually learns more complex features (Kalimeris et al. 2019 Hu et al. 2020). Other schools of
thought on this being the last layer neurons (Daniely) 2017 Lee al.) 2019) are a natural feature
choice as the prediction function can simply be viewed as a linerr function of the last layer.
Modifying neural nets to understand feature learning: A ReLU network can be viewed as a deep linear network with an depuddeeeedent sub-network selection. Lakshminarayanan & Vikram Singh (2020) make this explicit by constructing an alternate architecture which builds two separate mod- els one for input-dependent sub-network selection and another for the deep linear network that the input goes through and showed that feature learning happens via learning of active sub- networks. Disjunctive normal networks (Sajjadi et al., 2016) that utilize the disjunctive normal form, Gated Linear Networks (GLN) (Veness et al.) 202 Budden et al. 2020) Saxe et al., (2022) backpropagation-frre free architecture where each unit (neuron) learns locally using data-dependent gating (context) are some other recent papers that enable a better study of feature learning in deep networks by constructing different and (arguably) simpler architecture than standard ReL.U nets.
LOCAL DISCONTINUITIES IN THE LABEL FUNCTION
From the perspective of learning algorithm trying to fit a classification model on the training data, the only visible aspect is d-dimensional scatter plot with labels coloured by their label. From this viewpoint, the most interesting aspects are regions of the input space that see a colour change. This has been well studied for linearly separable data using the language of geometry and maximum-margins. Extending this idea to linearly-non-sepan this behaviour using notion of 'discontinuities' in the label funetion. Intuitively, the number of such discontinuities has to be small for the problem to be learnable with a reasonable number of data points (Klivans et al. 2008
data is tricky. We try to capture
We define the local discontinuity coefficient (LDC) of a contiguous region R C Rd and a manifold M fx E Rd f(x) 0} cutting through the region R for function .R >R follows:
rR, f) - 3 B
> -E <
where the expectation is w.r.t (X.Y) with X D and Y ( We will call (R f) as the global discontinuity coefficient (GDC) of f. Note that the LDC is a property of the data distribution D and labelling function f". We will say that the scope of discontinuity at f is R if (R, f) is large.
For example, if the data can be separated by a hyperplane fu w w x + = 0). then (IR', w x + b) and the scope of discontinuity of b is the entire space TRd. The region R corresponds to the scope of the discontinuity, and (R. f) captures the accuracy of the classifier sign(f(z)) within this scope. We argue that such local discontinuities in the label function are what drive feature learning in deep networks.
As an illustrative running example, we consider a labelling function I given by an Oblique Decision Tree (ODT) (Bertsimas & Dunn 2017 Murthy et al., 1994; Wickramarachchi et al. 2016 Carreira- Perpinán & Tavallali 2018). ODTs are decision trees with linear internal node functions u x + b, and a data point branches left/right depending on the sign of this value. These generalise standard decision trees. ODT labelling functions are particularly hard to learn for fixed-feature learning methods such as kernel machines, but allows for easy enumeration of all its discontinuities
In particular, we consider complete ODTs with node hyperplanes orthogonal to each other and each leaf node having equal number of data points (we call these COB-ODT: short for complete or- thogonal balanced ODTs).
are particularly hard for classic greedy tree learning methods such CART (Breiman et al. 1984] to learn. This is because the hyperplanes corresponding to in- ternal nodes (and the root node in particular) are all balanced (they have an equal number of positive and negative data points both sides of the hyperplane). Hence they do not get picked by greedy
3

---

Published as a conference paper at ICLR 2025
7
3
4
1
10
13
12:
6
2
14
11
(a) ODT h-planes of root, children (b) ODT h-planes of grandchildren
(c) Shaded leaf node regions.
Figure I: Hyperplanes and labelling function for a complete ODT of depth over a 2d input space. The children of internal node are 2i + and 2; 2. Green and red shades indicate positive and negative labels respectively. Note that the hyperplanes are not orthogonal here the input dimension must be greater than the number of internal nodes for COB ODT to exist.
methods under metrics such as information gain or Gini-Index. Other ODT learning methods that purportedly global (Zantedeschi et 2020 Bertsimas Dunn 2017 Lee & Jaakkola| (020) also fail to identify internal node hyperplanes for unknown reasons. Finite decision trees that do not contain the internal node hyperplanes a COB-OD ODD label function are guaranteed have poor test error. On several such COB-ODT settings, where all fixed-feature learning methods and tree algo- rithms fail, deep ReLUU nets and another deep architecture introduced the next section succeed. Experimental results supporting these observations are given in Table 2
important to note that the entire hyperplane corresponding to an internal node not discon- tinuity. For example, in the ODT in Figure [[c), only parts of the line corresponding root node 0 (or internal nodes to 6) is discontinuity. Also note that, despite the importance the hyperplane corresponding to node 0, it would not be picked first by any greedy tree learner the ratio between a number of positive and negatively labelled points is approximately the same on both sides of the hyperplane. This is exactly
for any data labelled COB-ODT.
Clearly, all the internal node hyperplanes correspond to label function discontinuities - i.e. a randomly chosen point close the hyperplane boundary has chance of flipping labels if it moves to the other side of the hyperplane. No other hyperplane has this property. All internal node linear functions have an appropriate region R such that (R, f) but the total mass of the region R, given by P(X E R) is of the order of dedepdh(atternll bode), An illustration of the computation of 7 is given in the Appendix.
To test our hypothesis of 'model features align with data features' we need deep network archi- tecture whose features (which also simply discontinuities in the model function) can be easily enumerated. To this end we propose the deep linearly gated network architecture.
4 DEEP LINEARLY GATED NETWORKS(DLGN)
The DLGN architecture serves as more transparent stand-in for ReLU networks and can be viewed "a midway point between deep linear networks and ReLU networks". DLGNs (like ReLU nets) learn useful non-linear features and outperform kernel methods while having structural simplicity (like deep linear networks).
4.1 ARCHITECTURE DETAILS
The deep linearly gated network has architecture similar to ReLU network and is defined by neu- rons residing in multiple layers. For simplicity, we assume the architecture consists of L hidden lay- with m neurons in each layer. The architecture is parameterized by matrices W W/2 ..... W. and U L and vectors u u 1++1 The matrices W W?...., W and U2 (??.... U are all of shape m. Wi has shape d. u² and u are vectors of size

---

Published as a conference paper at ICLR 2025
The architecture is most naturally described using the notion of paths, which denote by (f1....,
.L) ) E [m]", giving the sequence of hidden nodes that the path consists of. Let II [m]] 7.
denote the set of all paths in the network. The output of the model is given follows:
7(x)
Σ 9r. fx) TEI
(1)
where f= is called the path gating function for path π and In E IR is the value of path и. The path gating function f= is defined by (what we call) the gating network - deep linear network with weights W ..... WL The path gating function Ja is decomposed as product of individual neuron gating functions that make up the path æ. The path gating function for path π = (f1,..., ... is
fx( II F=1
(ni, (x) 2 0)
34 E [4],
nx(x - 1 (x) =V V'x
(3)
where (x) = x and VŁ [L] the matrices V E R form the "effective" weights of the neurons
in layer E and are given V w .............. W We call the hyperplane {x m (x) 0} as the gating hyperplane corresponding to neuron it in layer L.
The value 9x of path (ii..... iz) is also defined by network called the value network deep linear network with weights U² ......' U-, u no biases, and input given by u². It is simply
the product of weights along the path i.e.
ii
II T inie-1
f+1 u
(4)
The model defined in Equation 11 seems computationally hard implement in forward pass, but due to standard matrix multiplication properties, can be easily implemented at cost that is less than twice the cost of ReLU net with the same mL hidden nodes. i.e.
(5) 1x7((x) 2 0) (U*)(-1(x) for <>> 1. The gates 1) are as defined in Equation 3] The symbol o represents elementwise multiplication. proof for this equality is given in the Appendix along with a short note how the DLGN is related the ReLU network.
(x) = (u--1, (x))))
where (x) (x) 2 0) u² and hf(x)
In order learn the gating function parameters and back-propagate the gradient W, we replace the indicator function by a sigmoid. i.e. (a 2 0) with ((Ba) where a is the standard sigmoid function and B > hyperparameter. A figure illustrating the DLGN architecture with an example is given in the Appendix.
4.2 VARIANTS
We also consider two natural variants of the DLGN architecture. The first is a simple re-paramtterisation. where the gating network fo is parameterised directly by the matrices VI, V V4, each of shape m d, instead of using the parameters W "y.,.., WL. Clearly, this reparameterisation does not lose any representation power. We call this variant as DLGN-SF (for shallow features).
In another variant, we use explicit parameterisation of the coefficients Is instead of using a value network. This parameterisation is clearly more powerful than the standard DLGN parameterisation in the previous section. However, this requires parameter tensor of size m and is not practical for large L and m. We call this variant as DLGN-VT (for value tensor).
4.3 DISCONTINCITY ENUMERATION
The model ÿ in Equation equation is a linear combination of path gating functions. The discontinu- ities in ÿ are simply the union of discontinuities of fa over all paths т. While there are exponentially many gating functions fn, they decompose further into product of neuron gating functions, which are indicator functions over half-spaces. Thus, the set of all discontinuities can very simply be enumerated by considering the m. hyperplanes corresponding the neuron gating functions.
5

---

Published as a conference paper at ICLR 2025
(a) Layer! trained hyperplanes
(b) Layer2 trained hyperplanes
(c) Layer3 trained hyperplanes
Figure 2: An illustration of DLGN hyperplanes after training data in Fig Ic.
TRAINING ATTRACTS MODEL FEATURES TOWARDS LABEL FEATURES
We now have labelling function S" based on COB-ODT and model function ÿ based on DLGN. The discontinuities both f" and can be easily enumerated, and the discontinuities of both functions in the form of hyperplanes. We can evaluate the main claim of the paper.
A DLGN trained on data labelled by a COB-ODT shows several interesting properties. The most important of these properties is tendency of the effective hyperplane of the gating neurons given by V to cluster around the ODT node hyperplanes.
Figure 2 illustrates example scenario when 3-hidden layer DLGN is trained on data given in Fig- ure c). The initial hyperplanes (lines in this case) rx() = O as shown in Appendix Figure aa--) essentially random. However, after training, the hyperplanes in the later layers show remark- able tendency to move towards the hyperplanes corresponding to the decision tree particularly that of nodes close the root (See Figures (()-))) The relative idleness of the first layer hyperplanes seems to be an artefact of parameterising the gating network by deep linear network. This does not happen for DLGN-SF where the gating network effective weights V parameterised directly.
Table 1: Number of DLGN hyperplanes (after training) within given distance of the label function ODT hyperplanes. The 15 ODT internal nodes are numbered to 14, with 0 as the root. The distance of the closest (initial and trained) DI.GN hyperplane all the ODT hyperplanes is given in the last 2 rows. The DLGN has 4 hidden layers with 20 neurons in each layer, i.e. = 4, m 20.
Distance 0.1
4
6
8
12
13
14
2
1
1
1
1
1
0.2
0.3
2
Closest (ist(init)
1.26
1.22
1.21
1.23
1.24
1.23
1.17
1.27
1.23
1.27
1.22
1.26
Closest (isaa))na))
0.05
0.07
0.06
0.07
0.1
0.06
0.05
0.06
0.09
0.08
0.07
0.07
0.06
0.09
0.06
Note that Figure is schematic illustration, and the decision tree hyperplane-seekingg behaviour of the DLGN hyperplanes is exaggerated illustrate the idea properly. Table Table gives the results of typical experiment on synthetic dataset in which the data is a 100-dimensional vector uniformly distributed on the surface of the unit sphere and labelled by a depth-4 COB-ODT with 15 internal nodes. A 4-hidden layer DLGN with 20 neurons in each layer was trained on this dataset containing 30000 data points. Table@giver the same for DL.GN architecture with 100 neurons in each hidden layer. For each node in the ODT, we count the number of DLGN hyperplanes within a distance
of 0.1, 0.2, and 0.3 from it. The distance between two hyperplanes (V) = {x x = 0} and
H (z) {x x 0} is min (I
For large d, most pairs of vectors orthogonal to each other, and hence, typical value of dist( z) is about v2. At initialisation, there are almost no DLGN hyperplanes close (say distance less than 0.3) to any of the ODT hyperplanes. But it can be clearly seen from Table Ihht the training process attracts the DI.G hyperplanes towards the ODT hyperplanes. This happens most notably for the

---

Published as a conference paper at ICLR 2025
(a) Level 0
(b) Level
3
(c) Level 2
Figure 3: DLGN based ODT Learning. Illustration of the recursive procedure where for each level starting at O, we have Leeve DLGNs trained on different splits of the data given by previous levels. The main result of each training run is the largest cluster of the learned DLGN hyperplanes, which shown in the figures by red dashed lines.
root node, (13 out of the 80 DLGN gating hyperplanes end up close to the root node hyperplane), despite the root node having almost zero information gain or accuracy increase in the greedy decision tree construction setting. We conjecture that discontinuities in the label function serve attractors for the DLGN gating hyperplanes under SGD. The strength of this attraction towards hyperplane fx) = 0 is proportional to the scope of the label function discontinuity around this hyperplane (or the largest region R. for which (R, /) 1).
We believe that there is fundamental principle of feature learning in deep networks play here, and studying the reason for this behaviour and proving theoretically is promising future direction that is beyond the scope of this paper.
MECHANISTIC INTERPRETABILITY
DLGNS
The exact decomposition of the DLGN sum of the product of half-space indicator functions allows for a degree control and reusability of the learned parameters that are not possible with a black box architecture like the ReLU network. We demonstrate this aspect of DLGNs by construct- ing completely different type of model, namely an ODT, directly by copying parameters from learned DI.NNs.
The decision tree extraction procedure is based on the principle conjectured in the previous section. [egure3]giver schematic illustration of the procedure. the first stage, DLGN is trained on the entire data. Our conjectured principle would then imply that the hyperplane with the largest scope of discontinuity (which would be the hyperplane the root node of the ODT corresponding to f*) would attract more DLGN hyperplanes than others, resulting in a detectable cluster around it. A clustering is performed over the learned DLGN hyperplanes, and the largest cluster is chosen (See Figure B()). The cluster centre corresponding to it is chosen as the root node in the final decision tree. Based on the root node hyperplane, the training data can be split into two halves, and the procedure can be repeated on both halves recursively (See Figure ((b,)) until the data for training becomes too small or contains only one class. An oblique decision tree can thus be constructed by incorporating the largest cluster centres of the trained DLGN hyperplanes in the appropriate nodes of a tree.
The details of the above procedure are given in Algorithm I It returns a decision tree consisting of internal nodes and leaf nodes. Internal nodes are represented by a hyperplane and pointers to two child nodes. Leaf nodes represented by value that is either +1 1. The key subroutine in
7

---

Published as a conference paper at ICLR 2025
Algorithm Building a decision tree from trained DLGN
Arguments: Binary classification training set containing pairs (xi, y4)
Outputs: Root of an ODT
1: function BCILDTRFF(date)
2:
if IMPURE(data) and LARGE(data) then
left.right.v" DiSCONTHYYEERPPANNCCatS 4: left + (UILLDRRBE(EEeEt))
5: rightST BUILDTREH(EIght)
6: return NODE(V", LeftST, rightsT)
7: end
8: Iv MAIORITYLABELLCGtt)
9: return NODE(value=Iv)
10: end function
Algorithm Finding Discontinuous Hyperplane
Arguments: Binary classification training set containing pairs (Xi, 3k)
Outputs: Data split into halves along a hyperplane, and the hyperplane parameters w
1: function
2: model TRAINDII.GN(dtta)
3: V- GATEEAYYERRTLLANSSSSOOC
4: + LARGISTC.USTTERCNTTRR()
5: left, right + SPLITDATA(dat..")))
6: return left, right,v'
7: end function
Algorithm| is the DiSCONTHYPERPLANE function detailed in Algorithm 2 classification dataset, clusters the DLGN hyperplanes, and splits the data based on this hyperplane. In our experiments, we used the DBScan (Ester et al. 1996) algorithm for clustering the DLGN hyperplanes as it is robust to outliers.
It trains DLGN on
We call this end-to-end procedure of constructing an ODT from learned DLGNS as DLGN-DT. The total complexity the DLGN-DT procedure about depth times the cost of learning a single DLGN even though 2depth DLGNs are learned, most of them are learned significantly smaller subsets of the data, and the complexity of clustering simply depends on the architecture size which is typically small constant.
We emphasise that we are not proposing DLGN-DT as an alternate decision tree procedure. The main purpose of the DLGN-DT procedure is to show that have better control and understanding of parameters in the DLGN. This is analogous to mechanic demonstrating an understanding of car, by dismantling it and building two motorcycles and diesel electricity generator from it. This is fundamentally different from reusing the first few layers of a pre-trained convolutional net for feature extraction for the following reasons. Firstly, the atomic element that copied is the level of parameters and not sub-networks. Secondly, the new model is decision tree which is completely different type of model from neural networks trained via gradient descent. Thirdly, the new architecture (ODT) is not fine-tuned or retrained based on the data.
7 EXPERIMENTS
DLGN and its variants are more interpretable than ReLU architecture, while still having powerful feature learning capabilities. Thus, they form an important tool in understanding the learning process deep neural network. DLGNs the special setting of a COB-ODT label function can also be used to illustrate the core feature learning hypothesis in the paper. In this section, experimentally show that DLGNs, apart from being more interpretable, outperform static kernel methods, tree and non-tree algorithms and ReLU methods COB-ODY synthetic datasets and are comparable performance to ensemble tree methods on real tabular datasets.
8

---

Published as a conference paper at ICLR 2025
Table 2: Test accuracy on synthetic datasets
Dataset char.
DI.GN
Non free algo
Tree algo
Data
n
dlgn
digns
dlgnv
relu
disnn
gln
svl
svm
sdt
cart
zan
tao
dlgnd
SDI
20000
20
96.1
97.4
98.4
90.6
89.8
66.8
63.4
76.5
68.9
57.3
65.9
84.9
60.3
93.1
SDII
30000
100
94.3
90.3
95.8
82.3
74.8
61.3
61.9
66.8
62.2
54.5
58.8
64.7
56.1
88.2
SDIII
50000
500
65.1
63.6
62.2
62.1
61.6
59.6
60.7
62.3
61.9
51.5
54.6
61.3
52.7
62.5
Table 3: Test accuracy on tabular datasets
Datasct char.
DLGN
Non tree algo
Tree algo
Data
dlgn
dlgnv
relu
disnn
gln
Svm
sdt
cart
rf
tao
Adult
29000
14
85.7
85.0
83.2
84.6
83.3
81.1
84.2
84.8
83.0
86.1
84.5
85.1
Bank
27000
16
91.6
91.3
89.6
91.2
90.4
89.9
90.7
91.3
90.3
91.3
91.1
90.9
Card
18000
23
81.6
81.2
78.4
80.9
80.4
80.6
81.1
81.3
77.6
81.2
81.3
80.6
Telese
11000
10
88.2
87.2
87.7
87.7
85.6
79.4
87.1
86.4
82.9
88.1
87.0
84.9
Rice
2000
92.6
92.6
92.3
92.5
92.3
91.2
91.4
92.6
91.9
91.8
92.2
90.1
Stat
100
20
77.6
72.6
72.2
76.5
74.0
71.1
71.7
75.2
65.7
77.4
75.3
68.0
Spam
3000
57
94.4
94.0
94.1
94.0
92.1
92.3
93.1
93.4
89.4
94.8
93.1
91.2
Gyro
19000
98.8
98.3
98.4
98.4
98.3
98.3
98.1
98.4
98.7
99.1
98.6
98.6
Swar
14000
2400
100
100
99.6
100
100
99.9
100
100
99.9
Credit
12000
10
76.2
75.4
75.5
75.9
70.2
70.4
74.5
74.6
76.1
78.2
75.4
76.2
Elec
27000
7
82.8
80.6
82.7
79.6
77.5
73.6
78.5
75.8
86.3
87.6
76.5
76.4
Cover
396000
10
94.7
93.3
92.9
81.1
78.0
78.3
80.4
77.4
91.6
94.2
78.6
84.9
Pol
7000
26
98.9
98.0
98.2
97.9
93.8
87.8
97.7
98.2
96.6
97.4
97.6
95.4
House
9000
16
87.8
86.7
87.4
86.5
85.6
83.0
87.2
86.3
84.0
88.0
85.3
85.1
Mini
51000
50
93.6
93.4
93.2
91.0
88.6
89.3
89.3
88.9
89.9
93.1
91.7
88.8
Diab
50000
7
60.6
60.6
59.4
60.5
60.2
58.0
60.3
60.6
60.3
60.6
60.4
60.3
Jannis
40000
54
78.6
78.0
75.0
77.2
73.8
73.9
77.4
78.4
74.6
78.6
75.1
74.1
Bior
2000
419
76.2
74.3
76.9
73.6
74.3
73.5
77.5
76.9
70.5
72.5
72.9
Calif 14000
88.8
88.1
87.2
86.6
82.1
84.7
86.6
85.4
84.5
89.0
85.3
85.5
Heloc
7000
22
71.9
71.6
66.2
71.3
70.2
70.7
71.4
71.3
68.6
71.1
71.2
68.4
We evaluate the DLGN(dlgn)) DLGN-SF(dlgns) DLGN-VT(dlgmy) and DLGN-DT(dIgnd) algo- rithms, with width and depth hyperparameters chosen on a validation set. They are compared against the following standard algorithms.
Non-tree algorithms: ReLU networks (Fukushima, 1969): Classic multilayer ReLU activation
neural network with depth and width as hyperparameters. Disjunctive normal networks (disnn) (Saijadi et al. 2016): A neural network architecture explicitly designed to learn a union of the intersection of halfspaces, which exactly corresponds to the ODT labelling function with number of polytopes and number of half-spaces per polytope hyperparameters. Gated linear networks (gin) (Veness et al. 2021): A buckpropagation-Fir free architecture where each unit (neuron) leams locally and models nonlinear functions using data-dependent gating (context) with layer sizes, input size and context map size as hyperparameters. Support vector machines (SVM) (Cortes & Vapnik] 1995): Constructs linear separators a high dimensional space given by non-linear kernel. separate
The kernel choice and its parameters and the regularization constant are the key hyperparameters. SVM with a linear kernel. Saft decision tree SDT(sdt) (Frosst & Hinton 2017): A soft decision tree made out of trained neural network. Although it is termed decision tree, gives predictive distribution of classes at the leaf with the maximum path probability rather than explicitly dividing the data into left and right nodes recursively. It uses input dimension, output dimension, depth and the regularisation coefficient as hyperparameters.
Linear SVM (svl) corresponds
Tree algorithms: Classification and regression tree CART(cart) (Breiman et al. 1984): It splits the data greedily in a parallel-axis fashion and uses maximum depth, minimum samples split and mini- mum samples leaf as main hyperparameters. Standard random forest (rf) (Ho) 1995): An ensemble method operating by constructing multiple axis-parallel decision trees at training Number of trees estimators, maximum depth, minimum samples split and minimum samples leaf are main hyper- parameters. Zan-DD((aa)) (Zantedeschi al. 2020): A binary ODT learning algorithm that coneur-

---

Published as a conference paper at ICLR 2025
rently optimizes discrete and continuous parameters through sparse relaxation of program, facilitating gradient flow for joint optimization with tree depth, regularization, number of layers and dropouts hyperparameters. Tree alternating optimization TtO(tao) (Carreira- Carreira-Perpimnn & Tavallllli] 2018): An ODT learning algorithm where the initial tree parameters are optimized by decreasing the misclassification error, using alternating optimization over node subsets that sep- arable with number of iterations, minimum sample nodes, minimum leaf samples and regularization parameters hyperparameters.
Appendix A. details the experimental setup and all the hyperparameter sets used for these algo- rithms. The models exhibiting the highest performances are highlighted in bold in Tables and giving results on synthetic and real tabular datasets.
7.1 PERFORMANCE ON SYNTHETIC DATASETS
We construct three different synthetic datasets, where the data labels comes from COB-ODTs. The three datasets (SDI, SDII, and SDIII) have different input dimensions (20, 100, and 500, respec- tively) and use decision trees with depth 4. The dataset characteristics (number of training points and input dimension d) and results these synthetic datasets are described in Table 2 Almost all the standard algorithms, including ODT learning algorithms, failed to perform satisfactorily on these datasets. Standard ReLU nets give good accuracy for some of these datasets, but they are sensitive to hyperparameter settings, particularly for higher dimensional data. Another qualitative property of DL.GN-TT that is not fully reflected in Table is that the hyperplanes the learned tree closely match those in the true ODT. The main goal of designing DLGN and its variants is the study of feature learning, but is also important to observe that they are indeed learning models with high accuracy and outperforming all other algorithms in some settings. The models exhibiting the highest and second highest performance are highlighted in bold and blue respectively in Tables
7.2 PERFORMANCE ON TABULAR DATASETS
We assessed the performance of our methods, (LGN(dlggn) and DLGN-VT(dlgnv), compared to standard algorithms on real tabular datasets with the details of dataset characteristics (number of training points n and input dimension d) outlined in Table 3 We used a total of 20 tabular binary classification datasets for the comparative study of our models, Most datasets are available in the UCI repository[ Some are taken from OpenMI. benchmark: (Grinsztajn al. 2022 P performed the experiments with 5 different train test splits and the mean is considered here. The models exhibiting the highest performance are highlighted bold, and models whose accuracy lie in 95 per cent confidence interval of the best performing model is marked by blue in Tables We observe that the linear and kernel methods are almost always beaten by other algorithms which potentially learn data-dependent features. DLGN and random forests the best among the other algorithms. DLGN-VT also yielded competitive results while being more interpretable. This illustrates that feature learning is useful property for these datasets, and DLGNS are capable of learning powerful features.
We have
8 CONCLUSION
Feature learning in deep networks is a fundamental problem and this paper makes several advances on this problem it gives a novel hypothesis, a novel data setup a novel architecture and way to verify if feature learning indeed happens during training. Exploiting this progress by getting better learning algorithms and theoretically proving that the proposed feature learning mechanism is present in training are exciting directions of future research. A further discussion of the feature learning narrative in this paper is given the Appendix.
https::/ww.. org
tasks_incl Eakksinccluded41-225
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
HGR and CL thank the Robert Bosch Center for Data Science and Artificial Intelligence (RBCD- SAI) and the Wadhwani School of Artificial Intelligence (WSAI) at IIT Madras for its support.
REFERENCES
Sanjeev Arora, Simon Shaolei Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Neural Information Processing Sys- tems (NeurIPS), 2019.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners. the silent alignment effect. In International Conference on Learning Representations, 2022.
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932-37946, 2022.
Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106: 1009-10088, 2017.
Leo Breiman, Jerome Friedman, Richard A. Olshen, and Charles Stone. Classification and Re- gression Trees. Chapman and Hall/CRC, 1st edition, 1984. doi: 10. 10.1201/9781315119477
David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, and Joel Veness. Gaussian gated linear networks. Advances in Neural Information Processing Systems, 33: 13:16508- 16519, 2020.
Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint 1912.01112..11122.. 09980199......111111111111111181 2019.
Miguel A Carreira-Perpinán and Pooya Tavallali. Alternating optimization of decision trees, with application to learning sparse oblique trees. Advances in neural information processing systems, 31, 2018.
Lenaic Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differentiable program- ming. In Neural Information Processing Systems (NeurIPS), 2019.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297, 1995.
Alexandru Jaamian,Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. Conference on Learning Theory, pp. 5413-5452. PMLR, 2022.
Amit Daniely. Sgd learns the conjugate kernel class of the network. Advances in Neural Information Processing Systems, 30, 2017.
Amit Daniely and Eran Malach. Learning parities with neural networks. Advances in Neural Infor- mation Processing Systems, 33:20356-20365, 2020.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdl, volume 96, pp. 226-231, 1996.
Nicholas Frosst and Geoffrey E Hinton. Distilling neural network into soft decision tree. 1.. 2017.
Kunihiko Fukushima. Visual feature extraction by multilayered network of analog threshold ele- ments. IEEE Transactions on Systems Science and Cybernetics, 5(4):322-333, 1969.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 14820-14830, 2020.
11

---

Published as a conference paper at ICLR 2025
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 507-520, 2022.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume pp. 278-282. IEEE, 1995.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early- time learning dynamics of neural networks. Advances in Neural Information Processing Systems, 33:17116-17128, 2020.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Dimitris Kalimeris, Gal Kaplun, Prectum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
Adam R Klivans, Ryan O' Donnell, and Rocco A Servedio. Learning geometric concepts via gaus- sian surface area. In 2008 49th Annual IEEE Symposium on Foundations of Computer Science, pp. 541-550. IEFE, 2008
Chandrashekar Lakshminarayanan and Amit Vikram Singh. Neural path features and neural path kernel: Understanding the role of gates in deep learning. Advances in Neural Information Pro- cessing Systems, 33:5227-5237, 2020.
Guang-He Lee and Tommi S. Jaakkola. Oblique decision trees from derivatives of relu networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL BKe8UR4FPF
net forum?
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks gaussian processes. arXiv preprint 2017.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel constant. Advances Neural Information Processing Systems, 33: 15954-15964, 2020.
Sreerama K Murthy, Simon Kasif, Steven Salzberg, and Richard Beigel. Ocl: A randomized algo- rithm for building oblique decision trees. Citeseer.
Sreerama K Murthy, Simon Kasif, and Steven Salzberg. A system for induction of oblique decision trees. Journal of artificial intelligence research, 2:1-32. 1994.
Eshaan Nichani, Alex Damian, and Jason D. Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks. Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL /ttpitttss///oeeeeeiieee net www.:/eee/foummmicccmmm i== Shhbommykrr
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301-5310. PMLR, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1135-1144, 2016.
12

---

Published as a conference paper at ICLR 2025
Mehdi Sajjadi, Mojtaba Seyedhosseini, and Tolga Tasdizen. Disjunctive normal networks. Neuro- computing, 218:276-2888 2016.
Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference on Machine Learning, pp. 19287- 19309. PMLR. 2022.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning deep linear neural network. In In International Conference on Learning Representations, 2014.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp- 618-626, 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9588 2020.
Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Grabska- Barrinkka,,,, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks. In Proceedings the AAAI conference artificial intelligence, volume 35, pp. 10015- 10023, 2021.
Darshana Chitraka Wickramarachchi, Blair Lennon Robertson, Marco Reale, Christopher John Price, and Brown. Hheart: an oblique decision tree. Computational Statistics Data Anal- ysis, 96:12-23, 2016.
Valentina Zantedeschi, M.tt Kusner, and Vlad Niculae. Learning binary decision trees by argmin differentiation. In International Conference on Machine Learning, 2020. URL https: //api. semanticsen Semanticscholar.orrg htt.://cc.....oooooooooooooaaaaa
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
DLGN ILLUSTRATIONS
b
b
W
W
2x) 48g 5
2
X1
1xi 2x 0
1x 2x,
Xgn
Path value =
2
45
Path valum 10
4x
2xg
42
-10
2
Input
Layer
Layer
-15
-10
(a) Gating Network
-20
n
U
2X,
4Xg
5
-xx
Xy
Xg
Path value
Path value =
Output
-2
1x
1xg
Layer
Layer
18g
(b) Value Network
Figure 4: An illustration of DLGN network and node hyperplanes
Figure aa) and bb) give example DLGN operating 2-dimensional input with 2 layers and
2 neurons per layer, iec. d = m L = 2. Figure 4[c) illustrates the region of activation for all the m² 4 paths given by the gating network 4 a). The value network in Figure ((bb gives the path value for all these paths by multiplying the appropriate weights in the path.
A.2 DEGN GATING HYPERPLANE CLUSTERING WITH LARGER WIDTH
Table Number DLGN hyperplanes (after training) within a given distance of the label function ODT hyperplanes. The 15 ODT internal nodes are numbered 0 to 14, with 0 as the root. The distance of the closest (initial and trained) DL.GN hyperplane all the ODT hyperplanes is given in the last 2 rows. The DLGN has 4 hidden layers with 100 neurons in each layer, i.e. L 4, m 100.
Distance 0.T
14
0
0
0
0
0
0
0
0
0
0.2
0
0
0.3
Closest (init(
1.19
1.21
1.20
Closest (Gsa((nal)
0.06
0.08
0.09
0.08
0.38
0.22
0.15
0.45
0.12
0.12
0.57
0.14
0.08
0.68
0.11
A.3 DETAILS THE DLGN ARCHITECTURE VARIANTS
We also consider two natural variants of the DLGN architecture. The first is simple re- re-parameterisation. where the gating network fo is parameterised directly by the matrices V V.. 5 each of shape this reparameterisation does not lose any representation power. We call this parameterisation as DLGN-SF (for shallow features).
d, instead of using the parameters W'...,,''s Clearly,
14

---

Published as a conference paper at ICLR 2025
(a) Layerl initial b-planes
(b) Layer2 initial h-planes
(c) Layer3 initial h-planes
(d) Layer! trained h-planes (e) Layer2 trained h-planes (f) Layer3 trained h-planes
Figure 5: An illustration of DLGN hyperplanes before and after training on data Fig Ic.
Table Distance table for DLGN-SF Layers: 4 Nodes per layer: 10
Distance
7
9
10
11
0.1
[
0
0
0
0
0
0
0
0
0
1
1
0
0
0.3
1
Closest (init)
1.20
1.2
1.26
1.27
1.24
1.24
1.25
1.22
1.29
1.27
1.26
1.15
Closest (inta(nna))
0.05
0.07
0.08
0.08
0.12
0.11
0.13
0.09
0.18
0.26
0.08
0.16
0.10
0.14
0.21
Another variant that we use an explicit parameterisation of the coefficients 9r instead of using value network. This parameterisation is also clearly more powerful than the standard DLGN param- eterisation in the previous section. However, this requires parameter tensor of size m² and not really practical for large and m. We call this reparameterisation as DIGN-VT (for value tensor).
All of these models can extended to allow a bias parameter for the gating network so that the functions v7 can also be affine functions instead of strict linear functions, but we do not discuss this version here for the purpose of simplicity.
Number of DLGN hyperplanes (after training) within given distance of the label function ODT hyperplanes of VT and DLGN-SF are shown in Table 6 and Table
initialization, all numbered 0 to 14, with 0 as the
these numbers are equal to zero. The 15 ODT internal nodes root. Results of similar experiments as DLGN (Table T shown in Table (lbleg)andd and [ for DLGN SF and DLGN. VT respectively depicts identical hyperplane-seeking properties of these two variants of the DLGN architecture as well. For each node in the ODT, we count the number of DLGN hyperplanes within distance of 0.1, 0.2, and 0.3 from it.
A.4 LOCAL DISCONTINUITY COEFFICIENT ILLUSTRATION
Figure gives an example of regions R1, R2 and R3 and manifold functions fi, Based on the definition of the local discontinuity coefficient y, we have the following. The manifold given by the root node fi (x) = 0 has high y value in the smaller scope R, but has much smaller 7 value the larger scope of R2. i.e. >(R,, f1) z and (R2, fi) Rs 0. The manifold given by any ODT internal
15

---

Published as a conference paper at ICLR 2025
Table 6: Distance table for DLGN-VT Layers: Nodes per layer: 10
Distance
0.1
2
2
0
2
2
2
0
2
0.3
2
2
0
2
2
2
Closest (init)
1.16
1.27
1.23
1.22
1.28
1.22
1.18
1.17
1.27
1.19
1.21
1.21
Closest (inal)
0.03
0.05
0.06
0.35
0.08
0.09
0.06
0.09
0.24
0.06
0.04
0.07
0.05
0.07
0.05
f1(x) m1
x
+
01
0
R1
R2
R3
f2(x) U2 x =0
Figure 6: An illustration of Local Discontinuity Coefficient.
node hyperplane has scope within which it discontinuous, e.g. the scope of discontinuity of the manifold fz xx) = O would correspond to R3. i.c. 7(R3, f2) ss 1.
A.5
PROOF OF DLGN COMPUTATION EQUIVALENCE
In this section we prove that Equation D is implemented by Equation 5
For any é E [Z]. let D be x diagonal matrix, whose ii entry is 1 if the 7th hidden node in hidden layer is active and otherwise, i.e.
[x(()),,, = 1( (x) 2 0)
where 77 is the gating function defined Equation 3 We suppress the dependence D on x for convenience below. Consider the following function x
(xx) = (u++1)T D ...
D'U D4
(n=+
II D'U
D'u
(6)
We will show that the above expression is equal to the RHS of equation T when the path values Ir represented product of pairwise terms in Equation 4
Writing the equation above as product L L+ matrices, with the first and last being vectors of size and the intermediate matrices (D*U*) matrices of size m, and expanding the terms of the matrix product as sum of m terms
7(x)
Σ
u I+
II Em
Wiutehieee D Dl
Σ
D (pine(())) v=
II
(nt.
L II
Ulese-))
u
L+1
E fx(x) we(m]]
E=2
16

---

Published as a conference paper at ICLR 2025
To see that the RHS of Equation 6 also corresponds to the RHS of equation 5
just observe that h² (x) T2= (x))77) DJ (x) Thus the computational procedure given by Equation
exactly implements the conceptual path decomposition given in Equation []
A.6 RELATION BETWEEN THE RELU NETWORK AND THE DLGN
While the DLGN might seem like a completely different architecture from the ReLU network, is quite intimately related to and motivated from the ReLU network.
The ReLU network can also be expressed in a matrix product format like Equation However, the elements of the diagonal matrices D' (except the first layer with & 1) not simple halfspace indicator functions. They still correspond to whether the corresponding node is 'active" or 'not active'.
In this case, we can still view elements of the matrices De as features of the network, which cor- responds to the standard usage of features the activation maps of neurons. The ReLU network features would correspond to 'bent hyperplanes'. However, the simple view of connected but sepa- rate features that the DLGN affords is longer viable. We currently have no insight into why bent hyperplane features could be better than straight hyperplanes.
A.7 VALUE OF THE FEATURE LEARNING NARRATIVE FROM DLGN
We believe that the feature learning insights drawn from DLGN are applicable in general. More importantly, believe that it can shed light several phenomena that remain mysteries with deep networks. Some of the mysteries and candidate answers motivated from this paper are given below.
Q: Is neural network training using gradient descent parameter efficient?
A: No. The greedy nature of gradient descent, pushes multiple interconnected parts towards the same feature when it is potentially not required. e.g. In TableI]the distances of learned DL.GN hyperplanes (with width m 100) to the ODT hyperplanes are given. While the root node hyperplane is discovered separately by multiple gating hyperplanes, some of the other internal nodes are left in the lurch. This causes poor generalisation. Potentially, this narrative can be exploited to make gradient descent less greedy.
Q: How does gradient descent discover the label function discontinuities in high dimensional input space?
A: The full labeling function may be far from being linear separator, but when data is re- stricted within a small enough scope, one of the manifolds making up the decision bound- ary could very well be a good classifier (this argument uses the insight that even in high- dimensional data, if it is linearly separable, picking up the separator can be done in compute and data efficient manner). This causes that manifold to be picked up by one of the components of the deep network, and thereby making the rest of the learning problem easier and kickstarting a virtuous cycle.
Q: Why does neural network pruming enable better learning of smaller architectures than train- ing the smaller architecture from scratch?
A: We assume effective learning happens only in problems where the number of label function discontinuitics is small. Significant parts the neural network are simply not necessary to represent the discontinuities, which can be done by small fraction of the trained deep net- work model. A large width network is still better for training, because (say) doubling the number of neurons per layer increases the number of paths by factor of 2 This increases the chances of some path picking right scope during training enabling the learning of the appropriate label function discontinuity. Once the learning is complete, significant frac- tion of the network which were unlucky to not have gotten a good scope can be removed.
Q: What is the role of layers in deep networks? Is increasing the number of layers always beneficial?
A: The main role of layers is to give context/scope other layers. For example, with depth 4 ODT labelling function, a DLGN with 3 or lesser number of hidden layers would not be able give good scope any neuron. A DL.GN with more layers is just unnecessary.
17

---

Published as a conference paper at ICLR 2025
Table Synthetic Datasets Generation
dataset
total samples
train samples(n)
n. features(()
depth
seed
thres
generate_ data
SDI
40000
20000
20
4
365
0
through code
SDII
60000
30000
100
4
365
through code
SDIII
100000
50000
500
4
365
0
through code
This is reflected our experiments as well, where depth 4 DLGN performed best for depth 4 ODT labelling function.
A.8 DETAILS OF SUBROUTINES USED IN DIGN--TT
This function takes into account the DLGN model after training. It iteratively calculates the effective weights the layer using the formula V = W' W as lines 3 of the algorithm. finally returns each layer's effective weight (hyperplanes) vector.
Algorithm 3 Return gates of trained DLGN model
Arguments: A DLGN model with parameters W W'...., Wh, u _ ..... U²,
U?....,uh, U',t 1+1
Outputs: m hyperplanes in the input dimension
1: function GATEHYYERPLANESSSN
2: for [ +- to L do
3: V + W we-t-1 W
4: end for
5: return V/ + V1 VL
6: end function
A.9 DATASETS USED
We have used 3 synthetic datasets and 20 tabular datasets evaluate the performance of model DLGN and its variants against some standard algorithms. In this section, we will elaborately de- scribe the datasets used.
A.9.1 SYNTHETIC DATASETS:
This dataset is synthetically generated with specified dimensions from a labelling function f" given by an Oblique Decision Tree (ODT) with depth and defined number of data points given in Table 7 The datapoints x are drawn uniformly from the surface d'-dimensional sphere radius 1, centred the origin. We used
mentioned in Section 3 with biases kept at zero. The leaf node labels are chosen so that sibling labels get opposite signs. The final output includes the pruned data, labels, and information about the tree's structure. Three synthetic datasets (SD) used, named SDI, SDII, and SDIII
Table presents the parameters used for constructing the datasets.
A.9.2 TABULAR DATASETS:
We used total of 20 tabular binary classification datasets for the comparative study of our mod- els. Most datasets are available in the UCI repository https: arch ve. ics. ucd edu datasset Some are taken from the OpenMI. benchmark, as given the paper (Grinsztajn al.] 2022). httphhttphcccc........ /ttp://wwwwwm www. openml
httpuudyyyyyyeeeeeeeee tspk
sort tasks incl uded& 298 The dataset download URL is in Table g After download- ing the datasets, they are preprocessed by dropping with missing values, converting categorical features using Label Encoding, Standardizing numerical features, and Encoding the target variables to and 1.
Figure Jillustrates an example scenario when 3-hidden layer DLGN is trained on data given in Figure c). The initial hyperplanes(h-planee given by V and 22 and 3 shown in F
18

---

Published as a conference paper at ICLR 2025
Table Tabular datasets
data
download link
Adult Bank
http:///eeadiuieieaa.a.aass///a2 marketing.2ip
Card
Telesc
t telescope.22pp_____
http:///eecciaaaaaaaaaaaaaaslsas and andfoosmancc...22pp
Rice
Stat
Spam
Swar
be
Credit
arff
Elec
http:///g.....eecoooo/o/aaa////a e electricity.arff
Cover
covertype.arif
Pol
House
6H.arff
Mini
https:/
MiniBooNE.arff
Diabetes13OUS.arf.
Jannis
https:/app.ccemm
arff
Bior
Bicresponse.arff
Calif
Cal california.
Heloc
arff
are essentially random. However, after training, the hyperplanes in the later layers show remark- able tendency to move towards the hyperplanes corresponding to the decision particularly that of nodes close to the root (See FiguresB[d-f))
A.10
EXPERIMENTAL SETUP DETAILS
A.10.1 SAM.IVVAIDAATON.TSSS SPLIT:
For the synthetic datasets SDI, SDII, and SDIII, the dataset is split into 50% train, 25% test, and 25% validation set. Models are trained the training data and validated the validation set, and then the test score is reported against the test data with the best hyperparameters. Similarly, for the tabular datasets, the dataset is split into 60% train, 20% test, and 20% validation
19

---

Published as a conference paper at ICLR 2025
A.10.2 NUMBER FOLDS:
Based on the algorithms used, the number of folds used is also varied. For standard ML algorithms like CART, Random Forest, SVM, and SVM Linear, we use 3-fold cross-validation; for most other algorithms, including DLGNs, we use one-fold size.
1.1..3 HARDWARE:
All the experiments are performed on Kaggle, and all the neural network-based experiments use GPU, whereas traditional ML algorithms are performed on CPU. Kaggle provided GPUs, such as GPU T4 x2 and GPU PI00.
A. 10.4 HYPERPARAMETER TUNING:
Each algorithm used in this paper has different set of hyperparameters, and hyperparameter tuning one of the most important aspects for getting the best accuracy. Here, for each algorithm, we extensively searched from a collection of hyperparameters and validated their result on the valida- tion set to obtain the best-performing hyperparameters. Test results are reported based on the best hyperparameters. The below tables give list of all hyperparameters for each algorithm used.
Table 9: DLGN and DLGN-SF hyperparameters space
Parameters
Set of values searched on
num layers nodes(each layer)
3, 4, 5
10, 20, 50, 100, 200, 500, 1000
beta
3, 10, 20, 30
learning rate(Ir)
0.1, 0.01, 0.02, 0.05, 0.001, 0.005, 0.0001
epochs
200, 300, 500
batches
1, 10, 100
optimizers
Adam, SGD 0, 0.1
weight decay
Table 9 lists the hyperparameters for training DLGN and DLGN-SF across all datasets. The best combination, selected via validation, determines test accuracy. Key parameters include num layers, nodes (per layer), and beta.
Table 10: DLGN- hyperarmeees space
Parameters
Set of values searched on
num layers
3, 4
num nodes(each layer)
10, 20
beta
3, 10, 20, 30
learning rate(Ir)
1., 0.1, 0.01, 0.02, 0.05, 0.001, 0.005, 0.0001 200, 300, 0,0,,1000,5000,0000,,,01
epochs
batches
1, 10, 100
optimizers
Adam, SGD
weight decay
0, 0.1
C
0.1, 0.03, 1.0
max iter
100, 500, 1000
penalty
11, 12
solver
liblinear
Table 10 lists the hyperparameters for training DLGN-VT across all datasets. The best combina- tion, selected via validation, determines test accuracy. Along with the DLGN parameters, has C, max iter, penalty and solver additional parameters.
20

---

Published as a conference paper at ICLR 2025
Table 11: DLGN-DT hyperpaattesss space
Parameters
Set of values searched on
num layers
3, 4,
num nodes(each layer)
I0, 20, 50, 100, 200, 500, 1000
beta
3, 10, 20, 30
learning rate(Ir)
0.1, 0.01, 0.02, 0.05, 0.001, 0.005, 0.0001
epochs
200, 300, 500
batches
1, 10, 100
optimizers
Adam, SGD
weight decay
0, 0.1 0.1, 0.3
cps
min.samples
1, 3,7, 10, 15, 22, 40 1 to 10
max depth
Table 11 lists the hyperparameters for training DLGN-DT across all datasets. The best combination, selected via validation, determines test accuracy. Key parameters include eps, min samples, and max depth for clustering and tree depth.
Table 12: ReLU hyperparameters space
Parameters
Set values searched on
num layers
3, 4, 5
num nodes(each layer)
10, 20, 50, 100, 200, 500, 1000
learning rate(Ir)
0.1, 0.01, 0.02, 0.05, 0.001, 0.005, 0.0001
epochs
200, 500, 1000
optimizers
Adam, SGD 0, 0.1
weight decay
Table 12 lists the hyperparameters for training ReLU models across all datasets. The best combina- tion, selected via validation, determines test accuracy.
Table 13: SVM Linear hyperparameters space
Parameters
Set of values searched on
0.1, 0.5, 1, 2,
kernel
Linear
Table 13 lists the hyperparameters for training Linear SVM across datasets. The best combina- tion, selected via validation, determines test accuracy, with the kernel set to Linear.
Table 14 lists the hyperparameters for training Non-linear SVM across all datasets. The best com- bination, selected via validation, determines test accuracy, with the kernel set to rbf or sigmoid.
Table 15 lists the hyperparameters for training the CART model across all datasets. The best com- bination, selected via validation, determines test accuracy, with max_depth the key parameter.
Table T6 contains the hyperparameter set used in training the random forest model on the datasets. The trained model is tested a validation set, and the best hyperparameter combination found, which is then used to get the test accuracy. n_estimators define the number of estimators in the random forest
of the most vital hyperparameters.
Table contains the hyperparameter set used in training the SDT model on the datasets. The trained model is tested validation set, and the best hyperparameter combination is found, which then used to get the test accuracy. depth is the tree depth of SDT which is the most vital hyperpa- rameter tune.
21

---

Published as a conference paper at ICLR 2025
Table 14: SVM hyperparameters space
Parameters
Set values searched on
C
0.1, 0.5, 1, 2, rbf, sigmoid scale, auto, 0.001, 0.01, 0.1, 1, 10 2, 3, 4,5
kernel
gamma
degree
Table 15: CART hyperparameters space
Parameters
Set of values searched on
criterion splitter max max_depth
gini, entropy
best, random
1 to 10
min min.samples_spe spllt min samples leaf
I to 10
1, 2, 4, sqrt, log2
max features
Table 16: Random Forest hyperparameters space
Parameters
Set of values searched on
criterion n_estimators
gini, entropy
10, 20, 50, 100
max max_depth min samples split min samples leaf max_ mea_features
1 10
I to 10
1, 2, 4, sqrt, log2
Table 17: SDT hyperparameters space
Parameters
Set of values searched on
depth lamda learning rate(Ir)
0.1, 0.01, 0.02, 0.05, 0.001
0.1, 0.01, 0.02, 0.05, 0.001, 0.005, 0.0001 200, 300, 500
epochs
batches
32, 64, 128
weight decay
0, 0.1, 0.0005
Table 18: TAO hyperparameters space
Parameters
Set of values searched on
n_iters
10, 20,
mex_leaffnnole
5, 10, 15
randomize.tre tree update_scoe scoring
True, False accuracy
min wone..ode... mode samples tao min leaf samples tao reg. param
1, 3, 4,
1, 4,
0.1, 0.01, 0.02, 0.05, 0.001
22

---

Published as a conference paper at ICLR 2025
contains the hyperparameter set used in training the TAO model all the datasets. The trained model is tested on a validation set, and the best hyperparameter combination found, which is then used to get the test accuracy. iters, max leaf nodes and min node samples are important hyperparameters to train.
Table 18
Table 19: Zan-DT hyperparameters space
Parameters
Set of values searched
depth
I to 10
reg
0.1, 0.01, 1.48, 1.5,2
mlp layer
3, 5,
dropout Ir epochs batches
0.0, 0.01, 0.05, 0.07, 0.1 0.1, 0.01, 0.02, 0.05, 0.001
200, 300, 500
32, 64, 128,512 512
Table 19 contains the hyperparameter set used in training the Zan-DT models on all the datasets. The trained model is tested validation set, and the best hyperparameter combination is found, which is then used to get the test accuracy. depth, reg and layer are vital parameters.
Table 20: Disnn hyperparameters space
Parameters
Set values searched on 15,20,30, 100 15,20,30, 100
n.pooytopes list
m list
contains the hyperparameter set used in training the Disnn model on all the datasets. The trained model is tested on a validation set, and the best hyperparameter combination found, which then used to get the test accuracy. A value of 10-15 for n-polytopes and m_list works best.
Table 20
Table 21: GLN hyperparameters space
Parameters
Set of values searched on
layer_sizes num_noder
3,4,5
5,10,20,50 2 to 10
context.map.sinn Ir
0.1, 0.01, 0.02, 0.05, 0.001,0.00025
Table 21 contains the hyperparameter set used training the GLN model on all the datasets. The trained model is tested on validation set, and the best hyperparameter combination is found, which then used to get the test accuracy. layer sizes, num_nodes and context.map map.ssie are important parameters. context map.mmmmpppaaa size 4 gives the best result.
23


Reviews:
[{"review_id": "tPOPvWHOex", "reviewer": "Reviewer_tLyw", "summary": "This paper proposes a novel model architecture (deep linearly gated network, DLGN) and studies the way in which this model seeks out “label discontinuities” in the data during training. This analysis is enabled the fact that we can enumerate such label discontinuities in a DLGN. The synthetic datasets with known discontinuities are generated by another model — an oblique decision tree (ODT). The authors also show how an ODT can be constructed from a trained DLGN, for the purpose of interpretability. Finally, the paper presents results of fitting a DLGN to several UCI regression tasks, comparing performance to several tree-based, kernel-based, and NN-based baselines.", "questions": "- Authors suggest that they use a sigmoid instead of an indicator function to make the DLGN differentiable for training. Have authors considered using a temperature parameter for the sigmoid (potentially annealed during training), as common in other continuous relaxation methods?\n- On lines 406-407 authors suggest that they use DBScan for clustering hyperplanes due to this algorithm being “robust to outliers” — why do authors expect outliers to be a significant issue here?\n- How do authors explain the observation that certain layers of the DLGN are more prone to matching the true discontinuities than others  (Figure 2)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- Originality. The DLGN is an interesting architecture that combines deep linear networks with the gating mechanism to construct a novel class of non-linear models. One could also treat DLGN as a novel decision tree parametrization which, when relaxed using a sigmoid, can be learned by back-propagation. To the best of my knowledge, DLGN is a novel, original model architecture, though its connection to soft decision trees should be studied more carefully.", "weaknesses": "- Lack of focus. At the moment, the paper’s focus is split between a study of feature learning using DLGN (chapter 5), and a study of the DLGN itself as an expressive, yet interpretable model architecture (chapter 6). To me these are two orthogonal contributions, and the paper would be stronger if authors focused on one of these.\n- Significance. The significance of the proposed model and the feature learning study presented in the paper is not clear to me. \n  1. Capturing “label discontinuities” (are these not simply decision boundaries?) is at the core of solving a classification problem, hence it is not surprising that a model which works well on the task has to discover such hyperplanes — I don’t see an alternative way that a model can solve a task. The real question is how an over-parametrized model can correctly identify high-dimensional decision boundaries given limited data — this is the main mystery in the theory of deep learning at the moment, and one that this paper doesn’t shed much light on.\n  2. To understand the significance of the findings for deep learning, we would need to understand the relationship between a DLGN and a DNN. While it is nice (albeit, not surprising) to see how a DLGN uncovers the true “label discontinuities”, how can we know that a DNN will demonstrate the same behavior?\n  3. While the proposed architecture is appealing due to potentially being both expressive and interpretable, results on the real (UCI) datasets suggests that DLGN is comparable in performance to standard tree algorithms. There is little evidence that we should prefer the proposed architecture to e.g. the well-studied random forests, which we could also argue to be “interpretable”. (I do not consider results on synthetic data to be good evidence, given the connection between ODTs used to generate the data and DLGN.)\n  4. While the authors claim that the proposed architecture is interpretable, no interpretations of the models fit to real data are given. If interpretability if the main selling point of the architecture, I would expect a deeper analysis focused on interpretability.\n- Presentation. Please consider moving the DLGN model diagram to the main text: it’s difficult to understand the model architecture from the formulas alone. Also, please try to stick to academic language, and avoid informal phrases like “well nigh impossible”, “handily outperform”, “succeed comfortably”, “about the same”, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wElabVg6A3", "reviewer": "Reviewer_D8rp", "summary": "The authors introduce a model called the Deep Linearly Gated Network (DLGN) to study feature learning, specifically in binary classification tasks defined by an oblique decision tree labeling function. They use DLGN to test the hypothesis that during training, the model’s discontinuities move towards label function's discontinuities. The paper includes evaluations on dozens of open tabular datasets to compare DLGN with ReLU networks and tree-learning algorithms.", "questions": "1. What's the purpose of defining the manifold $\\mathcal{M}$ in line 133?\n2. In line 239, Equation (4), is there a missing transpose on $\\mathbf{u}_{i_1}^1$?\n3. I find it difficult to understand how the computational cost of a forward pass for Equation (1) is less than twice that of a ReLU network with $mL$ nodes. Could the authors provide further clarification on this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Clear writing. The authors study a specific class of problems with great clarity. \n2. The authors' persistence in tackling a challenging yet manageable problem setting is commendable.\n3. Like a black-box learner, the DLGN is able to learn non-linear features. Yet, it still provides mechanistic interpretability.\n4. DLGN outperforms both tree-based and non-tree algorithms, as well as ReLU networks, in the oblique decision tree setting, while maintaining strong competitiveness on real-world tabular datasets.\n5. The authors provide a framework that paves the way for future research and development.", "weaknesses": "I'm not seeing effective weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0SWVfxyHq", "reviewer": "Reviewer_n82d", "summary": "In this paper, the authors propose a mechanism to intuitively explain why neural networks can surpass kernel methods through feature learning. They hypothesize that the feature learning process involves the alignment of model function discontinuities with label function discontinuities during training. To explore this, they introduce a new network architecture called the Deep Linearly Gated Network (DLGN), designed as a surrogate for ReLU networks. They argue that this architecture retains similarities to ReLU networks while offering easier interpretability. Under this framework, they provide empirical evidence showing how model function discontinuities move toward label function discontinuities during training, facilitated by feature learning.", "questions": "See the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper is a bold attempt to deepen our understanding of feature learning in deep learning. The hypothesis, data setup, network architecture, and approach to interpretability are unconventional, and they bring a fresh perspective to the literature. This work has the potential to inspire new ideas and serve as a valuable starting point for further exploration.", "weaknesses": "From my perspective, while this work is intriguing, it is not yet ready for formal publication. My concerns are as follows:\n\n1. The authors reference previous studies examining the dynamics of single hidden layer models under specialized data and settings to push beyond kernel methods or deep linear models (Damian et al., 2022; Ba et al., 2022). They appropriately note that these analyses, often focused on specific data settings like the parity function, fall short of addressing the needs or behaviors of deeper networks. However, numerous works also investigate complex feature learning with deep neural networks, such as https://arxiv.org/abs/2305.06986 and https://arxiv.org/pdf/2311.13774. These studies should be acknowledged and compared with the current work.\n\n2. I am skeptical about the extent to which the new architecture resembles a ReLU network. Additionally, it is unclear how this intuition or design could extend to CNNs or transformers. Besides, the current version only applies to binary classification, whereas a universal feature learning mechanism should ideally apply across setups, such as binary classification, multi-class classification, and regression. I am unsure how the proposed intuition extends to these broader contexts. For example, existing papers (such as the two mentioned above) demonstrate that neural networks can efficiently learn $h = g \\circ p$ with $p$ quadratic and $g$ nonlinear via feature learning. How would this intuition explain such cases?\n\n3. In Section 3, the introduction to ODT progresses too quickly. The paper would benefit from a more mathematically detailed introduction to the new concepts presented in this section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oYncB19C14", "reviewer": "Reviewer_xQ4v", "summary": "This paper proposes a theory to explain how feature learning happens in neural networks. The authors posit that neural networks align (or rather, get attracted to) the discontinuites in the way the target label changes over the input domain. As a pilot test for this theory, the authors consider a setting where the target function to learn is an Oblique Decision Tree (ODT). ODTs correspond to decision trees where each internal node is a linear threshold. Thus, the hyperplanes associated to the nodes in the decision tree split up the input space into different labeled regions (see Figure 1). The hyperplane corresponding to the root is \"most discontinuous\" in terms of how the label changes on both sides of it (owing to further and further splits by its many descendants). The authors posit that the training of non-linear neural networks procedurally aligns the model with these discontinuities.\n\nAs a tractable model to further empirically study this theory, the authors propose a novel neural network architecture which they term Deep Linearly Gated Networks (DLGNs). DLGNs are somewhere in between standard nonlinear (e.g., ReLU activated) deep networks and deep linear networks. The function computed by a DLGN can be writen as a large summation of terms of the form $f_\\pi \\cdot g_\\pi$. Here, $f_\\pi$ is a product of the signs of activations of neurons in a deep linear network (a single neuron from each layer). $g_\\pi$ is a product of the activations themselves (a slight detail is that the weight matrices producing the activations in $f_\\pi$ and $g_\\pi$ are different). Thus, we can think of $f_\\pi$ as an indicator of the intersection of halfspaces, while $g_\\pi$ is the weight we add up if the indicator turns out to be true.\n\nThe authors fix the target function to be an ODT, and train the DLGN architecture on synthetic data labelled by the ODT. The suprising observation is given in Figure 2. At the end of training, if one plots the hyperplanes corresponding to the linear threshold at every neuron, one observes that the hyperplanes (at least in the later layers of the DLGN) align very well with the hyperplane splits in the ODT! Furthermore, Table 1 shows that most of these linear thresholds align with some hyperplane in the ODT. That is, the neurons in the DLGN architecture are \"getting attracted\" to aligning with some internal node in the ODT.\n\nThe authors next use this empirical observation to extract a decision tree out of a trained DLGN. Namely, they plot the linear thresholds corresopnding to all the activations after training, and then cluster these thresholds. The center of the largest cluster is chosen to be a root node hyperplane, and then we recurse this procedure on data on either side of this hyperplane. This procedure is pictorally well illustrated in Figure 3. Again, one can see that on performing such clustering-based decision tree generation, the first cluster center aligns very well with the root node hyperplane, and the phenomenon continues as we recurse. The upshot is that one can extract an interpretable model from the DLGN..\n\nFinally, the authors perform experiments on real-world classification data to illustrate that the classification accuracy of DLGNs is somewhere in between standard ReLU networks and other simpler non-neural network algorithms. Thus, DLGNs have the added benefit of being interpretable, but more powerful than stanard decision trees.", "questions": "1) Why do you even introduce a manifold $M$ in your notation in line 133? It is never used \n2) On line 192, I don't think I agree that \"no other hyperplane other than the internal nodes has this property\". e.g. consider any other hyperplane that is not one of the internal nodes in the plot in Figure 1; doesn't it also have points of both labels on either side of it?\n3) You introduce this formal notion of $\\gamma(R, f)$ as the local discontinuity coefficient in Section 3, but then never mention it anywhere later in the paper. How would you explain the results in say Figure 2 and 3 in the context of this quantity? Could you elaborate on this a bit?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed theory is intriguing and indeed very interesting. The plots in Figures 2 and 3 are indeed striking---they illustrate how faithfully the DLGN neuron activations are increasingly aligning with the ODT hyperplanes over the process of training. These observations support the theory put forth by the authors about neural network architectures possibly picking up on the discontinuities in the labelling function. The clustering procedure to extract a decision tree is also very interesting, and empirically seems to work well (as suggested by Figure 3). Overall, I find the theory proposed by the authors, along with the striking empirical illustrations, very interesting. These could motivate further theoretical investigations for such phenomena.", "weaknesses": "The pilot experiments done by the authors are admittedly specialized. Namely, they don't actually consider standard ReLU networks, but only the DLGNs they propose. Furthermore, they only consider cases where the target function is conveniently an ODT. While this is totally okay as an initial starting point, it does raise the question about whether such empirical phenomena of the neurons aligning with the discontinuities also arise in cases where the target function has discontinuities of a different nature (like curvy discontinuities, etc). But this is not a significant weakness, as it seems beyond the scope of a pilot study. But it would be really interesting to visualize the activations of the neurons (say with quadratic thresholds) for when the target function is also composed of curvy discontinuities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)