Not All Features Are Equal: Feature Leveling Deep Neural Networks for Better Interpretation
Authors: Yingjing Lu
Runde Yang
Venue:  (2020)

Abstract: Self-explaining models are models that reveal decision making parameters in an interpretable manner so that the model reasoning process can be directly understood by human beings. General Linear Models (GLMs) are self-explaining because the model weights directly show how each feature contributes to the output value. However, deep neural networks (DNNs) are in general not self-explaining due to the non-linearity of the activation functions, complex architectures, obscure feature extraction and transformation process. In this work, we illustrate the fact that existing deep architectures are hard to interpret because each hidden layer carries a mix of low level features and high level features. As a solution, we propose a novel feature leveling architecture that isolates low level features from high level features on a per-layer basis to better utilize the GLM layer in the proposed architecture for interpretation. Experimental results show that our modified models are able to achieve competitive results comparing to main-stream architectures on standard datasets while being more self-explainable. Our implementations and configurations are publicly available for reproductions.

Under review as a conference paper at ICLR 2020
NOT ALL FEATURES ARE EQUAL: FEATURE LEVEL- ING DEEP NEURAL NETWORKS FOR BETTER INTER- PRETATION
Anonymous authors Paper under double-bling review
ABSTRACT
Self-explaining models are models that reveal decision making parameters in an interpretable manner so that the model reasoning process can be directly under- stood by human beings. General Linear Models (GLMs) are self-explaining be- cause the model weights directly show how each feature contributes to the output value. However, deep neural networks (DNNs) are in general not self-explaining due to the non-linearity of the activation functions, complex architectures, obscure feature extraction and transformation process. In this work, we illustrate the fact that existing deep architectures are hard interpret because each hidden layer carries a mix of low level features and high level features. As a solution, we pro- pose novel feature leveling architecture that isolates low level features from high level features on a per-layer basis to better utilize the GLM layer in the proposed architecture for interpretation. Experimental results show that our modified mod- els are able to achieve competitive results comparing to main-stream architectures on standard datasets while being more self-explainable. Our implementations and configurations are publicly available for reproductionst.
INTRODUCTION
Deep Neural Networks (DNNs) are viewed as back-box models because their obscure decision making process. One reason that makes deep neural networks hard to interpret is that they able to magically extract abstract concepts through multi-layer non-linear activations and end-to- end training. From human perspective, it is hard understand how features extracted from different hidden layers and what features are used for final decision making.
In response the challenge of interpretability, two paths are taken unbox neural networks' deci- sion learning process. One method is to design verifying algorithms that can be applied to existing models to back-trace their decision learning process. Another method design models that "ex- plain" the decision making process automatically. The second direction is promising in that the interpretability is built-in architecturally. Thus, the verification feedback can be directly used to improve the model. [
One class of the self-explaining models borrows the interpretability of General Linear Models (GLMs) such as linear regression. GLMs are naturally interpretable in that complicated interactions of non-linear activations are not involved. The contribution each feature to the final decision out- put can simply be analyzed by examining the corresponding weight parameters. Therefore, we take a step forward to investigate ways to make DNNs as similar to GLMs as possible for interpretability purpose while maintaining competitive performance.
Fortunately, GLM model naturally exists in the last layer of most discriminative architectures of DNNs (See appendix A.3 for the reason that the last layer is GLM layer). However, the GLM could only account for the output generated by the last layer and this output not easy to interpret because potentially contains mixed levels of features. In the following section, results to demonstrate this mixture effect. Based on this observation, one way to naturally improve interpretation is prevent features extracted by different layers from mixing together. Thus, we
empirical
Public Repo URL annonymized for review purpose-See code folder for detailed implementation

---

Under review as a conference paper at ICLR 2020
directly pass features extracted by each layer to the final GLM layer. This can further improve interpretability by leveraging the weights of the GLM layer to explain the decision making process. Motivated by this observation, we design feature leveling network structure that can automatically separate low level features from high level features to avoid mixture effect. In other words, the low level features extracted by the ktth hidden layer can be readily used by the GLM layer, we should directly pass these features to the GLM rather than feeding them to the l + 1 hidden layer. We also propose feature leveling scale to measure the complexity of different sets of features' in an unambiguous manner rather than simply using vague terms such as "low" and "high" to describe these features.
In the following sections, we will first lay out the proposed definition of feature leveling. We then will illustrate how different levels of features reside in the same feature space. Based on the above observations, we propose feature leveling network, an architectural modification on existing models that can isolate low level features from high level features within different layers of the neural network in unsupervised manner. In the experiment section, we will use empirical results to show that this modification can also be applied to reduce the number of layers in an architecture and thus reduce the complexity of the network. In this paper, focus primarily on fully connected neural (eNWorks(FCNN) with ReLU activation function in the hidden layers. Our main contributions follows:
We take a step forward to quantify feature complexity for DNNs.
We investigate the mixture effect between features of different complexities in the hidden layers of DNNs.
We propose feature leveling architecture that is able to isolate low level features from high level fcatures in each layer to improve interpretation.
We further show that the proposed architecture is able to prune redundant hidden layers to reduce DNNs' complexity with little compromise on performance.
The remaining content is organized follows: In section 2, first introduce our definitions of feature leveling and use a toy example to show the mixture effect of features in hidden layers. In section 3. give a detailed account of our proposed feature leveling network that could effectively isolate different levels of features. In section 4, we provide a high level introduction to some related works that motivated our architectural design. In Section 5, we test and analyze our proposed archi- tecture on various real world datasets and show that our architecture is able to achieve competitive performance while improving interpretability. In section 6, we show that our model is also able to automatically prune redundant hidden layers, thus reducing the complexity of DNNs.
2 FEATURE LEVELING FOR NEURAL NETWORKS
The concepts of low level and high level features are often brought up within the machine learning literature. However, their definitions are vague and not precise enough for applications. Intuitively, low level features are usually "simple" concepts patterns whereas high level features are "abstract" or "implicit" features.
Within the scope of this paper, we take a step forward to give formal definition of feature leveling that quantizes feature complexity in an absolute scale. This concept of a features' scale is better than simply having "low" and "high" as descriptions because it reveals unambiguous ordering between different sets of features. We will use a toy example to demonstrate how features can have different levels and explain why separating different levels of features could improve interpretability.
2.1 A TOY EXAMPLE
We create toy dataset called Independent XOR(XORR. IXOR consists of set of uniformally distributed features X : {(x1. x², x Ja' E [-2.2], 2)2 x² E [-2, 2]. x E [0, 1]} and a set of labels
)) {0, 13. We use top indices for attributes of this toy example. The labels are assigned as:
{
y 1 x² 0 0.5
y 0 0 otherwise
2

---

Under review as a conference paper at ICLR 2020
10
-39
1.0
Figure 1: Toy IXOR dataset, with x²,, x² being two coordinates and xt being the size the point
In this dataset, (x², "0,,"",²)) clearly have different levels of feature. x can be directly used by the GLM layer it has a linear decision boundary. (x²,x²) is more complex as they form an XOR pattern and cannot be linearly separated, thus requiring further decomposition to be made sufficient for the GLM layer. To make correct decisions, the DNN should
layer decompose the XOR into lower level features, and directly transport x value to into the GLM layer.
2.2 CHARACTERIZE LOW AND HIGH LEVEL FEATURES WITH FEATURE LEVELING
From IXOR we can see that not all features have the same level of "complexity". Some could be directly fed into the GLM layer, others may need to go through one or more hidden layers to be transformed to features that can directly contribute to decision making.
Thus, instead of using "low" and "high" level characterize features, we propose frame the complexity of different features with the definition of feature leveling.
For dataset D consisting of N i..d samples with features and their corresponding labels {(a1, y1), .... (aN, }. We assume that samples a; E DD contains features that requires at most K hidden layers be transformed to perform optimal inference.
For DNN trained with K hidden layers and GLM layer, we define the set of 1 level feature as the set of features that requires be sufficiently utilized by the GLM layer. In the following paragraphs, we denote 4 E The as the 1thh level features extracted from one sample and Lg denotes the of 4tth level feature to be learned in the target distribution. The rest of high level features are denoted by HlG that should be passed to the tt layer to extract further level features. In this case, IL and HHg should that is LL n HH 0. In the case of the toy example, x is 4, level one feature, the first hidden layer to directly transport its value to the GLM layer. (xx, x²) is h1 The XOR can be decomposed by hidden layer with sufficient number of parameters to be directly used by the GLM layer to make accurate decisions. Assuming the first hidden layer /i has sufficient parameters, it should take in h and output 22-
hidden layers to extract under the current network setup to
disjoint, it is learned by
2.3 How THE PROPOSED MODEL SOLVES THE MIXTURE EFFECT AND BOOSTS INTERPRETATION
However, common FCNN does not separate each level of feature explicitly. Figure 2 shows the heatmaps of the weight vectors for both FCNN baseline and proposed feature leveling network trained on the IXOR dataset. We observe from FCNN that I value is able to be preserved by the last column of the weight vector from the first layer but mixed with all other features in the second layer, before passing into the GLM layer. Our proposed model, the other hand, is able to cleanly separate 3 and preserve its identity as an input to the GLM layer. In addition, our model is able to identify that the interaction between (a+ x²) can be captured by single layer. Thus, the model
3

---

Under review as a conference paper at ICLR 2020
eliminates the second layer and pass (x" 2²? features extracted by the first hidden layer directly to the GLM layer.
Looking the results obtained from the toy example, able to solve the mixture effect of features and gives out correct levels for features with different complexities in the context of the original problem. Therefore, the model is more interpretable in that creates a clear path of reasoning and the contirbution of each level features can be understood from the weight parameters in the GLM.
clearly see that the proposed model
tt Identity preserved by the last column
x sent the GLM directly by gate
10
eliminated by the gate
gate GLM
2
fistt layer
sccond layer
-10
first layer
second layer
Baseline DNN
Proposed
Figure 2: Weight heatmap of Baseline and proposed model with the initial architecture of 3-16-8-2. Arrows denotes information flow. 2 in the proposed model is gated from mixing with other features input to the hidden layer.
OUR PROPOSED ARCHITECTURE.
Inspired by our definition of feature leveling and to resolve the mixture of features problem, design an architecture that is able to recursively filter the hth level features from the 1tth layer inputs and allow them to be directly passed to the final GLM layer.
We start with definition of a FCNN and extend that to our model: we aim to learn a function JF parametrized by neural network with K hidden layers. The function J can be written as:
(
(1)
Su is the kth hidden layer function with parameters Os. d(-) is the GLM model used for either classification, or regression. Thus, the goal to learn the function F such that:
1 R(0) N
Σ C(F(a;))..))
0" arg min(((()))
(2)
In formulation, each hidden layer can be viewed separator for the 4 level features and extractor for higher level features. Thus, the output of fü has two parts: 4 the set 1th level feature extracted from inputs and be readily transported the GLM layer for decision making. And h2 is the abstract features that require further transformations by ft- In formal language, we can describe our network with the following equation (*) "denotes set subtraction):
F= (4,22,2,....4 fx (fk- (fx- -1x)
In order for fü to learn mutually exclusive separation, we propose a gating system for layer k paramatrized by Øk. that is responsible for determining whether certain dimension of the input feature should in l or hg. For a layer with input dimension J, the gate (.),,..2;)) forms the corresponding gate where z {0, 1}. OR is the parameter that learns the probability for the gate > to have value for the input feature 7th dimension to be allocated h4 and L otherwise.
In order to maintain mutual exclusiveness between 4 and hg, we aim to learn Oa such that the allows feature to pass to 4 if and only if the gate is exactly zero. Otherwise, the gate is and the

---

Under review as a conference paper at ICLR 2020
feature goes to hg. Thus, rewrite the neural network JF with the gating mechanism for the ht sample a, from the dataset:
F F=d F=
B(zz) hatp:///aa.aaiiaaaooaaa////////a
Here, @ acts element-wise multiplication. The function B acts as binary activation function that returns if and only if the value of and otherwise. The function B allows level feature 4 = (zx)) @ fx-1 to be filter out if and only if it does not flow into the next layer at all.
Then the optimization objective becomes:
R(0, Φ)
2 = g(ok)
(5)
With an additional Lo regularization term encourage less ha to pass into the next layer but more l to flow directly to the GLM layer. g( act as transformation function that maps the parameter the corresponding gate value.
Feature Leveling Block
Feature Leveling Block 2 Feature Leveling Block 3
Gate
Input
GLM
Label
Figure 3: Illustration of the model with three hidden layers. Yellow denotes hidden layer that typi- cally has ReLU activations and green denotes the h level feature separated out by the gates. Thick arrows denote vector form of input and output. The dimension between the input of the hidden layers and the output can be different.
To achieve this discrete gate construction, we propose to learn the gating parameters under the context of Lo regularization. To be able to update parameter values through backpropogation, we propose use the approximation technique developed by (Louizos et al. 2017) differentiable Lo regularization. We direct interested readers to the original work for full establishment of approxi- mating Lg and will summarize the key concept in terms of gating mechanism below.
Although the gate value E {0, 1} is discrete and the probability for certain gate to be 0 or is typically treated
Bernoulli distribution, the probability space be relaxed by the following: Consider s to be a continuous random variable with distribution g(s(Ø)) paramaterized by Φ. The gate could be obtained by transformation function m(-) as:
~ g(s()),
m(s) = min max (0, s))
(6)
Then the underlying probability space is continuous because & continuous and can achieve exactly 0 gate value. The probability for the gate to be non-zero is calculated by the cumulative distribution function Q:
q(= + 0
Q(s 01ø)
The authors furthers use the reparameterization trick to create a sampling free noise p(e) to obtain s: ni(e, Φ) with a differentiable transformation function ra(-), and thus g(-) is equivalent m where denotes function composition.
Then the objective function under our feature leveling network is:
K
R(0,0)
Σ E(aFFaa, B, +
A
Σ
Q(5x
5 {(6))
N
K
(8)
k=1
g(0x, E), g( mon(er:e)) e~ p(e)
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Interpreting existing models: The ability to explain the reasoning process within a neural network is essential to validate the robustness of the model and to ensure that the network is secure against adversarial attacks (Moosavi-Dezfooll et al., 2016 Brown al., 2017: Gehr et al. 2018). In recent years, Many works have been done to explain the reasoning process of an existing neural network either through extracting the decision boundary (Bastani et al. 2018 Verma et al. 20102 Wang et al.)
2018
Zakrzewski 2001), or through variety of visualization methods (Mahendran & Vedaldi, Zeiler & Fergus 2014 [i et al., 2013). Most of those methods are designed for validation purpose. However, their results cannot be easily used to improve the original models.
2015
Sell' explaining models are proposed by (Alvarez Melis & Jaakkola 2018) and it refers to models whose reasoning process is easy to interpret. This class of models does not require separate validation process. Many works have focused on designing self-explaining architectures that can be trained Aud-to-end(Zhang et al., [2018] Worrall et al.) 2017 Li et al.] 2018: Kim & Mnih 2018 Higgins et al. 2017). However, most self-explaining models sacrifice certain amount of performance for interpretabilityy Two noticeable models among these models are able to achieve competitive performance on standard tasks while maintaining interpretability. The NIT framework (Tsang et al., 2018) is able interpret neural decision process by detecting feature interactions in Generalized Additive Model style. The framework is able to achieve competitive performance but is only able to disentangle up to K groups of interactions and the value K needs to be searched manually during the training process. The SENN framework proposed by (Alvarez Melis & Jaakkola 2018) focuses on abstract concept prototyping. It aggregates abstract concepts with a linear and interpretable model. Compared to our model, SENN requires an additional step to train an autoencoding network to prototype concepts and is not able to disentangle simple concepts from more abstract ones in a per-layer basis.
Sparse neural network training refers to various methods developed to reduce the number of parameters of neural model. Many investigations have been done in using L2 or L1 (Han et al., 2013 Ng] 2004| Wen et af.] 2016) Girosi et al.) 1995) regularization to prune neural network while maintaining differentiability for back propagation. Another choice for regularization and creating sparsity is the Le regularization. However, due to its discrete nature, it does not support parameter learning through backpropagation. A continuous approximation of Le is proposed in regard to resolve this problem and has shown effectiveness in pruning both FCNN and Convolutional Neural Networks (CNNs) an end to end manner (Louizos al.) 2017). This regularization technique is further applied not only to neural architecture pruning but to feature selections (Yamada et al.) 2018). Our work applies the Lo regularization's 's feature selection ability in a novel context to select maximum amount of features direct inputs for the GLM layer.
Compared to Residual Structures, our model is able to explain features at different levels and their contributions separately due to the linear nature of GLM. ResNet (He et al., 2016). Highway Networka (Srivastava et al. 2015) cannot isolate each level as their skip features are further entan- gled by polling, non- linear activation and if the following blocks. Different from ResNet with full connection to all features, we propose to learn which feature to pass to GLM from a probabilistic perspective.
5 EXPERIMENTS
We validate our proposed architecture through three commonly used datasets MNIST, and Califor- nia Housing. For each task, we use the same initial architecture to compare our proposed model and FCNN baseline. However, due to the gating effect of our model, some of the neurons the middle layers are effectively pruned. The architecture we report in this section for our proposed model is the pruned version after training with the gates. The second to last layer of our proposed models is labeled with a star to denote concatenation with all previous 4 and the output of the last hidden layer. For example, in the California Housing architecture, both proposed and FCNN baseline start with 13- 64- 32- 1 as the initial architecture, but due to gating effect on deeper layers, the layer with 32* neurons should have in effect 32- + (13 - 10) + (64 28) = 71 neurons accounting for previously gated features. (13 10 = 3 for 4, 64 28 36 for 12).

---

Under review as a conference paper at ICLR 2020
Table 1: MNIST classification and California Housing price prediction
MNIST
California Housing
Model
Architecture
Accuracy
Model
Architecture
RMSE
FCNN
784-300-100-10
0.984
FCNN
13-64-32-1
0.529
LO-FCNN
219-214-100-10
0.986
GAM
0.506
SENN (FCNN)
784-300-100
0.963
NIT
8-400-300-200-100-
0.430
Proposed
291-300s-10
0.985
Proposed
10-28-32*
0.477
The two objectives of our experiments are: 1) To test if our model is able to achieve competitive results, under the same initial architecture, compared to FCNN baseline and other recently proposed self-explaining models. This test is conducted by comparing model metrics such root mean square error (RMSE) for regression tasks, classification accuracy multi-class datasets. 2) Because our model can separately account for each layer's s contribution, we can apply the gradient with respect to each layer and get the level of features our model recognize for each part of the input.
Experiment implementation details are deferred to appendix A7-10.
5.1 DATASETS & PERFORMANCES
The MNIST hand writing dataset (LeCun et al. 2010) consists of pictures of hand written digits from to 9 in 28 28 grey scale format. We use 784 300 100 - 10 architecture for both FCNN baseline and the proposed model. This is the same architecture used the original implementations of (Louizos et al. 2017). Our model is able achieve similar result, with less number of layers, as those state-of the architectures using ReI.U activated FCNNs. The feature gates completely eliminated message passing to the 100 neuron layer, which implies that our model only need level and level 2 layers for feature extractions to learn the MNIST datasets effectively.
The California Housing dataset (Pace& & Barry 1997) is a regression task that contains various metries, such as longitude and owners' age to predict the price of a house. It contains 8 features and one of the features nominal. We converted the nominal feature into one-hot encoding and there are 13 features in total. Since California Housing dataset does not contain standard test set, we split the dataset randomly with 4:1 train-test ratio. Our proposed model could beat the FCNN baseline with the same initial architecture. Only out of 13 original features directly passed to the GLM layer, implying that California Housing's input features are mostly second and third level.
5.2 DISENTANGLING THE CONTRIBUTION OF EACH LEVEL OF FEATURES
4
Laya
5BhN
TOW
With digit 4 as an example, compared to FCNN and SENN-FCNN, our model's /1 identi- fies the contour of digit 4 and the corners of 4(lrrger in gradient) as first level feature for 4. shows concentrated negative gradients in the middle of the digit which corresponds to the "hole" in digit 4.
MNIST:
Proposed
NIT
Cal Housing: We compare with NIT in [22]. The left figure shows that our model with differ- colors indicating feature gradients from different layers, NIT's colors indicate different groups
7

---

Under review as a conference paper at ICLR 2020
Compared to [22]. our model's /1 identifies "longitude" (Ioon)) feature that linearly relates to housing price since in California, longitude is major determining factor for housing price com- paring to latitude. According the gradients, 12 and 13 emphasizes different parts of the input, justifying that our model could divide the features to different sets. However, for [22], gradients of most groups similar, indicating that the features not sufficiently disentangled among groups. In contrast, our model identifies most important features with stronger weight and zero or minimal weight for irrelevant ones.
5.3 SCALABILITY
During the training stage, our model requires more computation resources as features from higher layers are passed the final layer well to the final GLM layer. However, during the inference time when the gates are learned, each feature input to the neural layer is only computed once due to the mutual exclusion of our gating setup. The weight parameters related the "zeroed out" features can also be eliminated. In most cases our model results in lower parameter count. In the Appendix (A2) we show the number of parameters our framework needs for the reported inference models.
5.4 EXTEND CONVOLUTIONAL NEURAL NETWORKS
Our framework is also applicable to be applied to convolutional architectures. To modify, we simply apply the gate to the imput features. Appendix (A and Figure 5) shows our model can also clearly isolate features from different levels. To reduce the gated feature size, apply convolutions with no activation to reduce dimension while maintaining linearity.
STRENGTH IN PRUNING REDUNDANT HIDDEN LAYERS
Due to proposed model's ability to encourage linearity, model is also able to reduce its net- work complexity automatically by decreasing the number of hidden layers. MNIST classification, example, when the dataset feature level is less than the number of hidden layers, our proposed model can learn to prune excess hidden layers automatically as the network learns not to pass in- formation further hidden layers. As a result, the number of hidden layers are effectively reduced. Therefore, we believe that our framework is helpful for architectural design by helping researchers probe the ideal number of hidden layers to use as well as understanding the complexity of a given task.
0.
400
0.4
200
50
300
155
200
250
300
50
200
150
200
250
Thamand Interationss
Tixasand io
Figure 4: MNIST training performance curve and number of inputs passed the following hidden layer (blue denotes the number of features passed to the firs hidden layer. Orange curve denotes the second).
DISCUSSION
In this work we propose novel architecture that could perform feature leveling automatically to boost interpretability. We use a toy example to demonstrate the fact that not all features are equal in complexity and most DNNs take mixed levels of features as input, decreasing interpretability. We then characterize absolute feature complexity by the number of layers requires to extracted to make GLM decision. To boost interpretability by isolating the h level features. We propose feature leveling network with a gating mechanics and an end-to-end training process that allow the tthh level features to be directly passed to the GLM layer. We perfrom various experiments show that our feature leveling network is able to successfully separate out the 4tth level features without compromising performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. Tensorflow: system for large- scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI} 16), pp. 265-283. 2016.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7775-7784. Curran Associates, Inc., 2018.
Oshert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Advances in Neural Information Processing Systems, pp. 2494-2504, 2018.
Tom B Brown. Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 1712.09665, 2017.
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Mar- tin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpreta- tion. In 2018 IEEE Symposium on Security and Privacy (SP). pp. 3-18. IEEE, 2018.
Federico Girosi, Michael Jones, and Tomaso Poggio. Regularization theory and neural networks architectures. Neural computation, 7(2):219-269 1995.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, volume 3, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 102.59180,.05983, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. toronto. edu/kriz/cifar himl, 55, 2014.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [On- line]. Available: http://yann. lecun. com/exdb/mnistt, 2:18, 2010.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv: 11060066, 2015.
Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Thirty-Second AAAI Con- ference Artificial Intelligence, 2018.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through lo regularization. arXiv preprint arXiv:1712.01312,, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Seyed-Mohsen Moosavi- Dezaail,,, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 2574-2582, 2016.

---

Under review as a conference paper at ICLR 2020
Andrew Y Ng. Feature selection, 1 vs. 12 regularization, and rotational invariance. In Proceedings of the twenty-firs. international conference on Machine learning, pp. 78, ACM, 2004.
Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33 (3):291-297, 1997.
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in neural information processing systems, pp. 2377-2385, 2015.
Michael Tsang, Hanpeng Liu, Sanjay Purushotham, Pavankumar Murali, and Yan Liu. Neural in- teraction transparency (nit): Disentangling learned interactions for improved interpretability. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Ad- vances Neural Information Processing Systems 31, pp- 5804-58 -8133. Curran Associates, Inc., 2018.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. arXiv preprint arXiv: 024777 2018.
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems, pp. 6367- 6377, 2018.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082. 2016.
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Inter- pretable transformations with encoder-decoder networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Deep supervised feature selection using stochastic gates. arXiv preprint arXiv:/810.044 04247, 2018.
Radosiaw R 7akrzewski. Verification trained neural network accuracy. IJCNN'01. Interna- fional Joint Conference Neural Networks. Proceedings (Cat. No. OICH37222), volume 3, pp- 1657-1662. IEEE, 2001.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818 833. Springer, 2014.
Quanshi Zhang, Ying Nian Wu, and Song- Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827- 8836, 2018
A APPENDIX
A.1 ADDITIONAL EXPERIMENT
Table 2: CIFAR-10 Binary
Model
Architecture
AUC
FCNN
3072-2048-1024-2
0,855
GAM
(Tang et 2018
0.829
NIT
sang et al. 2018
3072-400-400-1
0,860
SENN
(FCNN)
1072-2004-10244
0.856
Proposed
3072-130- 1024*-2
0.866
The CIFAR-10 Dataset Krizhevsky et al., 2014) consists of 32 x 32 RGB images of 10 different classes. We test our model's 's ability to extract abstract concepts. For comparison, we follow the experiments in the NIT' paper and choose the class cat and deer perform binary classification.
10

---

Under review as a conference paper at ICLR 2020
The resulting architecture shows that for FCNN networks, most of the the two chosen classes mainly differentiated through their second level features. None of the raw imputs are used for direct classification. This corresponds to the assumption that RGB images of animals are relatively high level features.
A.2
NUMBER OF PARAMETERS REQUIRED FOR THE REPORTED MODELS
Table 3: MNIST
Model
Number of parameters
FCNN
266,200
LO-FCNN
240,962
SENN (FCNN)
532,400
Proposed
97,098
Table Cal-Housing
Model
Number of parameters
2,912 Proposed
FCNN
1,247
Table 5: CIFAR-10 Binary
Model
Number of parameters
FCNN
8,390,656
SENN (FCNN)
16,781.312
Proposed
6,430,460
Due to unable to reproduce a result reported by NIT paper on CIFAR-10, we used the original architecture that the authors used in the original NIT paper. As result, did not include the number parameters in our table.
A.3 REVISIT GLM FOR INTERPRETATIONS
DEEP NEURAL NETWORKS
Consider training linear model with dataset {X, (} where X' is the set of features and y is the corresponding set labels. The goal is to learn a function f(x) from (I,, Us) E (X, (/ujeee to criteria function La (z,, 34) with parameter set 6.
In a classical setting of Linear Models, 0 usually refers to a matrix w such that:
f(x) T(w +
Here, refers to the predicted label given sample instance of of feature and refers to the set of functions such as Logictic, Softmax and Identity. GLM is easy to interpret because the contribution of each individual dimension of to the decision output y by its corresponding weight. Therefore, we hope to emulate GML's interpretability in a DNN setting by creating method to efficiently back-trace the contribution of different features.
We argue that our proposed architecture is similar to GLM in that our final layer makes deci- sion based on the weights assigned to each level of input features. Our model is linear in rela- tionship to various levels of features. Given k levels of features, our model makes decision with y = [w 21, w. l2,, wk] each weight parameter w; indicates the influence of that layer. With this construction, This insight can help us to understand whether the given task is more "low level" or "high level" and thus also help to understand the complexity of given task with precise characterization.
easily interpret how each levels of feature contribute to decision making.
11

---

Under review as a conference paper at ICLR 2020
A.4 THE LAST LAYER OF COMMON NEURAL NETWORKS IS GLM LAYER
The "classical" DNN architecture consists set of hidden layers with non-linear activations and final layer that aggregates the result through sigmoid, softmax, or linear function. The final layer is in fact similar to the GLM layer since it itself has the same form and optimization objective.
A.5 OUR NOVELTY COMPARED TO RESNET AND OTHER SIMILAR ARCHITECTURE
Compared to Residual Networks, our model is able to explain features at different levels and their contributions separately due the linear nature of GLM. ResNet and DenseNet cannot isolate each level as their skip features are further entangled by polling, non-linear activation and the following hidden layers. Different from ResNet with full connection to all features, we propose learn which feature to pass to GLM from probabilistic perspective. Specifically, we introduce the 10 regularization for the purpose of performing effective feature leveling. In contrast, Resnet and Dense Net do not perform such layer wise regularization.
A.6 POWER TO EXTRACT FEATURE COMPLEXITY THROUGH PRUNING
To demonstrate that our network achieves effective pruning and can help practitioners determine the complexity of a given problem, we use Cal Housing as an example and train our models with 2-5 hidden layers. Each intermediate hidden layer has 32-32 structure. To prove that model can find the optimal structure, we first run the baseline model (without gating) with 2-5 hidden layers separately. We observe that the mse is 0.2364 for 3 layer, 0.2618 for and 0.4807 for 5. Thus, the layer model is sufficient to make accurate prediction. Then we train our model with gate selections and observed that when started with more than 4 hidden layers, our model would completely be reduced to a layer model after training and this is indeed the best structure for the Cal housing task as verified with the complete models. Thus, we argue that our model is able to discover optimal number of hidden layer to make accurate prediction of housing price. This further proves that our model would helpful for architecture engineers decide on optimal number of layers for any given task.
A.7 REPRODUCING EMPIRICAL RESULTS: GENERAL CONFIGURATION
All models are implemented in TensorFlow (Abadi et al., 2016 and hyperparameters configurations could be found our public repository or supplemental code. Model name with citation denotes that the result is obtained from the original paper. SEEN's architecture listed is the prototyping network while we use similar architecture for autoencoder parts. All SENN models are re-implemented with fully connected networks for comparison purposes.
A.8 DATASET AND PREPROCESSING
MNIST is dataset that contains 60000 training and 10000 testing of handwriting digits from 0 to 9. Experiment results were tested against the allocated testing set.
CIFAR-10 10 is dataset consists of 10 classes of images each with 10000 training and 2000 testing. We used the allocated testing set for reporting results.
For MNIST, CIFAR-10, rescaled the color channel with divisor of 255., to make pixel values from 0 to
For Cal Housing, dropped all samples with any empty value entry. Normalize all numerical values with mean and standard deviation.
The IXOR dataset is generated with the script attached in the supplemental material under src/independent xor.
A.9 HYPERPARAMETER
The only tunable hyperparameter in our model is the À which we usually consider values from 0.5 to 0.01. All the À values to display result is in the model scripts of the attached folder. Generally, lower
12

---

Under review as a conference paper at ICLR 2020
À are better for training more complicated dataset such CIFAR-10 to prevent many Gating at early stage.
A.10 EXACT NUMBER OF ITERATION RUNS
MNIST 280000 CIFAR- 10 680000
California Housing 988000
A.11 RESULT OF OUR MODEL ON CONVOLUTIONAL NEURAL NETWORKS
13

---

Under review as a conference paper at ICLR 2020
1
2
3
1
6
Figure 5: MNIST model trained with layers of convolutional neural network structure. We use the gradients to show different level of features our model extract from each layer. We see that each class is emphasized
features on different levels. 5, and 8 are more sensitive lo feature. Some other class are more sensitive to other level of features except from the last level. At the same time, due feature gating. the highest level feature in the last column is obvious.
14


Reviews:
[{"review_id": "rJgZ18JcFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a feature leveling technique to improve the self-explaining of deep fully connected neural networks. The authors propose to learn a gated function for each feature dimension for whether to directly send the feature to the final linear layer. The gated function is trained with L0 regularization technique proposed in (Louizos et al., 2017) to encourage more low-level features passed to the final layer. Experimental results on MNIST, California Housing, CIFAR10 show that the proposed method can achieve comparable performance with existing algorithms on sparse neural network training.\n\nQuality:\n\nOverall, the paper is well written with some minor formatting errors. The toy example demonstrates the idea of this paper clearly. However, the novelty of this paper, when compared to NIT, is that the L0 regularization is used to pass the feature to last layer. Considering the self-explaining feature, this work can only explain that some of the input features are suited for final layer, while there is no explanation on the other features since they are used to construct higher level features.\n\nClaririty:\n\nSome parts of this paper are not clear:\n1.\tWhy l_k and h_k has to be disjoint? A feature suited for final classification does not suggest that it can’t be used to construct higher-level feature.\n2.\tIn (4), should not B() be an inverse binary activation function: (1-z)?\n3.\tIs g(.) the Bernoulli distribution?\n4.\tIn section 5.2, why compare the gradients of a specific input example while one can directly look at z_k, the gated function?\n5.\tI assume the fully connected layers have bias term. If so, (4) suggests that the gated location will also be added with a learned bias, which is different than what the paper proposes.\n\nNovelty:\n\nThe novelty of this paper lies in the sparse training objective becomes passing as many lower-level features to final layer as possible instead of zeros out the intermediate weights. However, the key technique, L0 regularization, has been proposed and used as stated in the related work. While the authors state the application L0 to a novel context to select features is different from prior work, the novelty is rather incremental.  \n\nSignificance:\n\nThis work demonstrates that the L0 regularization technique for sparse neural network training can also be applied to learn a skip-layer connection. However, from both novelty, performance, and self-explaining perspectives, this work does not introduce much to the field.\n\n\nPros:\n\n1.\tThe paper is well written.\n2.\tThe toy example showcases the issue that this work tries to tackle.\n3.\tThe experimental results show the comparable performance to existing works.\n\nCons:\n1.\tThe novelty is not sufficient considering the prior works on sparse neural network training.\n2.\tThere are some clarification issues as mentioned before.\n3.\tThe performance is only comparable to existing works.\n4.\tThe self-explaining contribution is not clear since only a few input features can be explained if they are passed to the final layers.\n5.\tThere is no experiment on how \\lambda would affect the resulting network architecture.\n\n\nMinor corrections:\n1.\tFirst paragraph on sec. 5: three datasets: two datasets (or mention it’s in appendix).\n2.\t5.2 compare to NIT: the citations are in wrong format. Also the reference for NIT is corrupted.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skl_RbuoKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a neural network architecture to separate low-level features and high-level features. At the k-th hidden layer, 1) the k-th level features, defined as the set of features that requires k-1 hidden layers to extract, are directly passed to the final GLM layers; 2) the remaining features are further processed in the subsequent layers. This separation is achieved by applying the gating mechanism. The model can be interpreted by the weights associated with each of those k-th level features in the final GLM layer. Experimental results on the MNIST classification dataset and the California Housing regression dataset demonstrate the proposed approach can 1) achieve competitive performance (in terms of classification and regression) compared to the FCNN baseline; and 2) has better interpretability performance.\n\nThis paper is clearly written. The description of the model architecture is easy to follow. The introduction of the related works and background material are well organized.\n\nMy major concern with this work is how to interpret those k-th level features. Since the weights of those k-th level features in the final GLM model are used to indicate the importance of those features, interpreting the meaning of those k-th level features seem necessary. In practice, how to interpret the meaning of those higher-level features? For example, for the California Housing dataset, how to interpret the meaning of those features learned at level 2?\n\nThe interpretation of the MNIST classification examples seems difficult to understand. Compared to inspecting the raw pixels, would it be easier to interpret through learning a few prototypes, similar to the approach described in Alvarez-Melis and Jaakkola NIPS 2018?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkgEmy9zsS", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes a novel architecture that is able to separate different types of features learned at each layer of a neural network through a gating structure -- features that are sufficiently passed through the network are immediately sent to the final output layer. In addition, they provide reasonable definitions of levels of features, in contrast to the standard \"low\" to \"high\" descriptions. Lastly, in order to make the model more interpretable, they utilize an L0 loss on the gates of each layer to prioritize lower level features being used in the final layer.\n\nSignificance:\n\nAlthough gating is not novel, their use to send kth-level features to the final GLM layer is. Other than that, not much is contributed, as their differentiability trick, as mentioned, has already been done. The motivation to separate different types of features is interesting and definitely an issue that should be studied more. \n\nQuality:\n\nThe paper is easy to follow and nicely written, but with a few minor typo issues:\n\n1. Page 1, refers to appendix A.3 but should be for A.4\n2. Page 2, \"section\" is inconsistently capitalized \n3. Page 6, mentions three commonly used datasets but only mentions MNIST and California Housing. \n4. Page 8, mentions Appendix A.11 for CNN but this section is empty.\n\nIn regards to content quality, a few things stand out that could be improved:\n\n1. A major issue is that the interpretability of features with k > 1 are still not explained -- all we know is that they don't need to be sent further through the network. (i.e. solves the separation issue but leaves gaps in interpretability)\n2. Since the gates themselves can be studied, rather than finding gradients, wouldn't a simpler way to explain the network be to look at which features are passed to the GLM layer? This would especially be helpful in the first layer when looking at the original input features. \n3. Currently it is not clear if the architecture learns when features (l_k) are directly \"useful\" for classification, or if they are just not compatible with the features passed on to the next layer (h_k). \n4. In terms of interpretability, only a few other methods are tested, and gradients are the only way they compare. An exploration of other attribution methods could have further supplemented their claims.\n5. Claims are made about how many layers a certain dataset needs for sufficient classification through heuristic experiments; however they are not thorough enough in terms of ablation to fully make this claim. Width of layers are chosen but not analyzed; how is gating affected by the width of the network? For example, in MNIST, would only 3 layers be needed if the width is increased or decreased? This isn't immediately clear.\n6. Extensiveness of experiments -- I do like the toy dataset as an example, but to show effectiveness of this framework, a larger breadth of datasets could have been used. As an example, in the SENN paper, they utilize breast cancer and COMPAS but these were not tested on this architecture.  In addition, the results from convolutional layers would be much more preferred, since the best performing architectures on large vision datasets such as ImageNet primarily use convolutions.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xlfwiujS", "reviewer": "AnonReviewer7", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a categorization of inner layer weights to be linearly or non-linearly correlated with the output. The motivation on why this is important is somewhat weak in the paper. But I could see cases where this is important, if there is supporting evidence that it helps with interpretability. However, I did not see that in this draft unfortunately. \n\n“GLMs … interactions of non-linear activations are not involved” – the link function in all GLMs except linear regression is non-linear. So I am not sure what this statement means. \n\nThe definitions of \\mathbb{L_k} and \\mathbb{H_k} are not clear when they are introduced. The example given talks about l_1 and h_1, not about \\mathbb{L_k} and \\mathbb{H_k}.\n\nI didn’t get the point of the function B(.). Why not directly use z \\in {0,1} ?  Similarly, I did not get the point of introducing g(.), why not just push the entire mapping into \\phi ? The notation is made unnecessarily complicated. \n\nThe sentences after eq 7 are unclear to me. What does \\epsilon \\sim p(\\epsilon) even mean ? Is \\epsilon the parameter or the random variable here ? What is \\epsilon used for ? What is m(.) ?\n\nThe toy example is simple. But it does not address why such a classification into h_k and l_k is helpful and how it can be used. Can the authors motivate the usecases where such an interpretation (using the toy example) is useful ?\n\nThe experiments are not convincing. The MNIST experiment regarding identification of contours is very vague. There are tons of other methods that give better interpretation. The identification of “long” feature to be linearly correlated with the output can be done in a much faster and easier way, by simply checking individual feature correlations. The strength of such a method that the authors are proposing would be to extract useful information for cases where the input features are /not/ linearly correlated, while there are some inner layer features which /are/ linearly correlated. This could help in understanding the landscape of the classification better. But I did not see that happening here. Happy to be proved wrong by the authors or other reviewers if I am missing something here. Similarly, there is lot of work on sparsification/pruning of NNs, in light of which I am not sure what Section 6 adds.  \n\nAppendix A.4 is redundant, softmax/sigmoid being a glm is well-known. Also, in the main text (e.g. last para page 1), the reference is to Appendix A.3, while it should be to A.4. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to learn self-explaining neural networks using a feature leveling idea.  Unfortunately, the reviewers have raised several concerns on the paper, including insufficiency of novelty, weakness on experiments, etc. The authors did not provide rebuttal. We hope the authors can improve the paper in future submission based on the comments.  

Decision: Reject