Neural networks behave as hash encoders: An empirical study
Authors: Fengxiang He
Shiye Lei
Jianmin Ji
Dacheng Tao
Venue:  (2021)
Keywords: Explainability of deep learning

Abstract: The input space of a neural network with ReLU-like activations is partitioned into multiple linear regions, each corresponding to a specific activation pattern of the included ReLU-like activations. We demonstrate that this partition exhibits the following encoding properties across a variety of deep learning models: (1) {\it determinism}: almost every linear region contains at most one training example. We can therefore represent almost every training example by a unique activation pattern, which is parameterized by a {\it neural code}; and (2) {\it categorization}: according to the neural code, simple algorithms, such as $K$-Means, $K$-NN, and logistic regression, can achieve fairly good performance on both training and test data. These encoding properties surprisingly suggest that {\it normal neural networks well-trained for classification behave as hash encoders without any extra efforts.} In addition, the encoding properties exhibit variability in different scenarios. {Further experiments demonstrate that {\it model size}, {\it training time}, {\it training sample size}, {\it regularization}, and {\it label noise} contribute in shaping the encoding properties, while the impacts of the first three are dominant.} We then define an {\it activation hash phase chart} to represent the space expanded by {model size}, training time, training sample size, and the encoding properties, which is divided into three canonical regions: {\it under-expressive regime}, {\it critically-expressive regime}, and {\it sufficiently-expressive regime}.

Under review as a conference paper at ICLR 2021
NEURAL NETWORKS BEHAVE AS HASH ENCODERS: AN EMPIRICAL STUDY
Anonymous authors Paper under double-blind review
ABSTRACT
The input space of neural network with ReL.U-like activations is partitioned into multiple linear regions, each corresponding to specific activation pattern of the included ReI.U-like activations. We demonstrate that this partition exhibits the following encoding properties across a variety of deep learning models: (1) deter- minism: almost every linear region contains at most one training example. We can therefore represent almost every training example by unique activation pattern, which is parameterized by neural code; and (2) categorization: according to the neural code, simple algorithms, such as K K-Means, K K-NN, and logistic regression, can achieve fairly good performance on both training and test data. These encod- ing properties surprisingly suggest that normal neural networks well-traines for classification behave as hash encoders without any extra efforts. In addition, the encoding properties exhibit variability in different scenarios. Further experiments demonstrate that model size, training time, training sample size, regularization, and label noise contribute in shaping the encoding properties, while the impacts of the first three are dominant. We then define an activation hash phase chart to represent the space expanded by model size, training time, training sample size, and the encoding properties, which is divided into three canonical regions: under- expressive regime, fiftically-expressiit regime, and weapcciennty-epresstttte regime.
INTRODUCTION
Recent studies have highlighted that the input space of a rectified linear unit (RcLU) network is partitioned into linear regions by the nonlincarities in the activations (Pascanu et al. [013 Montufar ct al. 2014; Raghu et al., 2017), where ReLU networks refer to the networks with only ReLU- like (two-piece linear) activation functions (Glorot al., 2011 Maas et al., 2013 He et al. 2015 AArjvssy et al. 2016). Specifically, the mapping induced by a ReLU network is linear with respect the input data within linear regions and nonlinear and non-smooth in the boundaries between linear regions. Intuitively, the interiors of linear regions correspond to the linear parts of the ReLU activations and thus corresponds the boundaries are induced by the turning points. Therefore, every example can be represented by the corresponding activation pattern of the linear region where it falls In this paper, we parameterize the activation pattern as a (01 matrix, which is termed as neural code. Correspondingly, a neural network induces an activation mapping from every input example to neural code. For the detailed definition of neural code, please refer to Section 3 This linear region partition still holds the neural network contains smooth activations (such as sigmoid activations and tanh activations) besides ReLU-lik activations, in which the interiors are no longer linear but still smooth.
specific activation pattern of the ReLU-like activations, while
Through a comprehensive empirical study, this paper shows:
A well-trained normal neural network performs a hash encoder without any extra effort, where the neural code is the hash code and the activation mapping is the hash function.
Specifically, our experiments demonstrate that the neural code exhibits the following encoding prop- erties shared by hash code (Knuth) 1998) in most common scenarios of deep learning for classifica- tion tasks:

---

Under review as a conference paper at ICLR 2021
Determinism:
When neural network has been well trained, the overwhelming majority of the linear regions contain at most one training example per region. Thus, almost every training example can be represented by unique neural code. To evaluate this determinism property quantitatively, we propose a new term redundancy ratio, which is defined to be и-m where is the sample size and m is the number of the linear regions containing the sample. Experimental results show that the redundancy ratio is near zero in almost every scenario.
Categorization: The neural codes of examples from the same category are close to each other in the neural code space under the distance whereon (such as Euclidean distance and Hamming distance), while the neural codes are far away from each other if the corre- sponding examples are from different categories. We conduct clustering and classification experiments on the neural code space. Empirical results suggest that simple algorithms, such as K-Means [Looyd 1982). K-NN (Cover & Hart, 1967, Duda et al., 1973), and lo- gistic regression can achieve fairly good training and test performance which is at least comparable with the performance of the corresponding neural networks the raw data.
The two encoding properties collectively measure the expressivity of the activation mapping. For the brevity, we term this expressivity as goodness-of-hush.
It is worth noting that our study is different to the efforts of employing neural networks learn hash functions, where the outputs are the hash codes of the input examples (Wang et al. 2017)
Specifically, Xia et al. (014); Lai al. (2015); Zhu al. (2016); Cao et al. (2017; 2018 design hash layers to neural networks for learning hash functions of images; and Simonyyan an & Zisserman
2014 Donabue et al. (2013); Wang al. (2016); Varol et al. (2017); Chao et al. (218))))) Yuan al.
2019 extend the applicable domain to video data. Surprisingly. this paper reports that the activation
pattern (or neural code) is already fairly good hash code.
The encoding properties also exhibit some variabilities in different scenarios. We then conduct comprehensive experiments to investigate which factors would influence the encoding properties. The empirical results suggest that model size, training time, training sample size, regularization, and label noise contribute in shaping the encoding properties, while the first three have dominant influences. Specifically, larger model size, longer training time, and more training data lead to stronger encoding properties.
We evertually define an activation hash phase chart to characterize the space expanded by model size, training time, sample size, and the srddness-of-hassee According to the discovered correlations, this space is partitioned into three canonical regions:
Under-expressive regime. The redundancy ratio is considerably higher than zero while the categorization accuracy considerably lower than 100%. However, both redundancy ratio and categorization accuracy exhibit significantly positive correlations with model size, training time, and training sample size.
eextically-exrresiive regime. This a transition region between the under-expressive and sufficiently-exprssiii regimes. The goodness-of-haid changes considerably as model size, training time, and sample size change, while the correlations become insignificant.
Sufficienll-expressive regime. The redundancy ratio is almost zero while the categoriza- tion accuracy has become fairly good. One can hardly observe them change when model size, training time, and training sample size change. This regime covers many popular scenarios in the current practice of deep learning, especially those in classification.
It is worth noting that our partition is different from the one proposed by Nakkiran et al. 22200) which characterizes the the expressivity (or expressive power) of the input-output mapping induced by neural network. By contrast, our the partition in activation hash phase chart characerizes goodness-on -hahh..
Our results are established on empirical results of multi-layer perceptrons (MLPs), VGGs (Simonyan & Zisserman 2013), ResNets (He et al. 2016aa b). ResNeXt (Xie et al.) 2017). and DenseNet (Huang et al. 2017) trained for classification
the datasets MNIST (LeCun al. 1998) and CIFAR-10 (Krizhevsky & Hinton| 2009). Our code is available in the supplementary material. The code,
obtained models, and collected data will be released publicly.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
Many works have also studied the number of linear regions (linear region counting) in neural net- works containing ReLU activations. Pascanu al. (2013): Montufar et al. (2014) propose an upper bound exponential with the network's & depth and polynomial to the width. Montufar et al. (2014); Arora et al. (2016); Hu & Zhang (2018); Hanin & Rolnick (2019)); Zhu et (2020) improve the upper bounds and lower bounds for the linear region counting. Xiong et al. (2020) study the linear region counting of convolutional neural networks. [Serra et al. (2018) theoretically show that one can obtain larger linear region counting when the layer monotonously decreasing from the early layer to the final layer. Poole et al. 2 how the linear region counting would change as the training progresses. Raghu et al. (2017) define a trajectory length based on the activation patterns to measure the expressive powers of neural net- works. Kumar et al. (2019) empirically demonstrate that large proportion of the ReLU activations are always either activated or de-activated for all training examples in a well-trained and fixed net- work. Zhhan&& & Wu (2020) report that optimization methods would also significantly influence the geometry property of linear regions.
Novak et al. (2018); Hanin & Rolnick (2119)) investigate
A partition in the loss surfaces of neural networks has also been observed. [Soudry & Hoffer (2018) highlighted that the loss surfaces of neural networks with piecewise linear functions are partitioned into multiple smooth and multilinear open cells, while the boundaries are non-differentiable. He et al.] (2200) discovered three other properties: (1) every local minimum in a cell is the global minimum in the cell; (2) local minima in cell are interconnected; and (3) all local minima a cell are in an equivalence class. This paper focuses on another partition observed in the data space.
PRELIMINARIES
Suppose ReLUU network A is trained to fit dataset [11, v:), I....,n) for elassification, where v; E X' C R2dx, dx is the dimension of x, yi E E (1....,d) d]. d is the number of potential categories, and n is the training sample size. Additionally, we assume that all examples (x4, Vi) are independent and identically distributed (i.i.d)) random variables drawn from a data distribution D. Moreover, we denote the well-trained model as M. Here, "well-trained" refers to the training procedure has converged.
Recent works have shown that the input space of a ReLU network A partitioned into multiple linear regions, each of which corresponds to specific activation pattern of the ReLU activation functions. In this paper, we represent the activation pattern as matrix P E P C {0, 1 where I and w are the depth and the largest width of this neural network N. respectively. Specifically, the (i, j)th component characterizes the activation statue of the j-th ReLU neuron in the i-th layer. The (i,j)-th component equals 1 represents that this neuron is activated, while it equals 0 means this neuron is deactivated or invalid' The matrix P is termed as neural code. We can also re-formulate the neural code as vector if no confusion of the depth and width occurs.
is worth noting that the volume of boundaries between linear regions is zero, because the bound- aries correspond to at least one turning point in the activations, which are of measure zero. Corre- spondingly, the probability that some examples fall in the boundaries is zero. We thus assume no example is in the boundaries. Therefore, fixing the weight of the model M, every example æ X can be indexed by the neural code P of the corresponding linear region. It is worth noting that the instance x can be either seen in the training sample set or the test sample set.
NEURAL NETWORKS PERFORM AS HASH ENCODERS
Through an empirical study, this paper discovers that (1) the neural code is a hash code of the corresponding datum; and (2) correspondingly, the mapping X + P from datums to their neural codes is a hash function, which is termed as activation mapping. In contrast to the learning-to- hash methods, we find well-trained normal neural networks for normal tasks (such classification) already perform hash encoding without any extra efforts. Specifically, the neural code exhibits
Different layers may have different numbers of neurons. Therefore, there might he some indices (+) are invalid. We represent the activation patterss of these neurons as since they are never activated.
3

---

Under review as a conference paper at ICLR 2021
two major encoding properties: (1) determinism, and (2) categorization; please see details in Section [] Similar to the works in learning-tor to-ai,,,, we adopt the following two measures to quantitatively evaluate the activation mapping hash mapping.
Redundancy ratio. We define the following rechndancy ratio to measure the determinism property. Generally, smaller redundancy ratio is preferred when evaluating the activation mapping hashing function.
Definition 1. (redundancy ratio) Suppose there are n examples dataset S. If they are located in activation regions, the redundancy ratio is defined to be n_
Categorization accuracy. Hash code is usually employed for nearest neighbor searching. In this paper, we perform simple algorithms, such K-Means, K-NN, and logistic regression, to the neural codes. The training accuracy and test accuracy are employed to evaluate the encoding properties. Specifically, a higher accuracy corresponds to a better encoding property. K --Means is designed for unsupervised learning. Here, we use it to verify our encoding properties in the context of supervised learning. The pipeline is modified and given in Appendix
worth noting that
We investigate the activation mappings induced by MLPs, VGG-18, ResNet-18, ResNet-34, ResNeXI-26, DenseNet-28 trained the MNIST dataset and VGG- 19, ResNet-18, ResNet-20, and ResNet-32 trained on the CIFAR-10 dataset. Details of the implementations are given in Appendix [ due to the space limitation. The redundancy ratio is almost zero and the categorization ac- curacy is fairly good in all cases, as presented in Tables and 2 These results verify the determinism and cat- egorization properties. Moreover, we visualize the neu- ral codes employing t-SNE (Maaten Hinton| 20088, as presented in Figure This visualization suggests that ex- amples from the same category are concentrated together while clear boundaries are observed between examples
t-SNE visualization of the neural codes of MNIST generated by onc-hidden-layer MLP with width 100.
Figure
from different categories, which coincide with the categorization property.
Table 1: Accuracy of K-Means and K-NN on the neural code of CNNs trained on MNIST.
Table 2: Accuracy of logistic regression (LR) on the neural code of CNNs trained on CIFAR-10.
Architecture K-Means ace K-NN acc
Architecture
LR acc
Test acc
VGG-18
99.95%
99.33%
ResNet-18
98.96%
99.32%
VGG-19
92.19%
91.43%
ResNet-34
99.66%
99.49%
ResNet-18
89.55%
90.42%
ResNeXt-26
98.31%
99. 99.24%
ResNet-20
88.76%
90.44%
DenseNet-28
69.87%
98.59%
ResNet-32
89.05%
90.45%
FACTORS THAT SHAPE OF ENCODING PROPERTIES
The results presented in Tables 1 and 2 also suggest that the encoding properties exhibit variability in different scenarios. Through comprehensive experiments, we investigate which factors would influence the encoding properties. The investigated factors include model size, training time, sample size, three popular regularizers, random data, and noisy labels. Some details of the experiment implementations are given in Appendix A due to the space limitation.
5.1 RELATIONSHIP BETWEEN MODEL SIZE AND ENCODING PROPERTIES
We first study how model size would influence the encoding properties. We trained 115 one-hidden- layer MLPs on the MNIST dataset and 200 five-hidden layer MLPs on the CIFAR-10 dataset with different widths (please the full list of widths involved in experiments in Appendix(A) while all irrelative variables are strictly controlled. The experiments are repeated for 5 trials on MNIST and 10 trials CIFAR-10, respectively.
4

---

Under review as a conference paper at ICLR 2021
MNIST
CIFAR-10
Categorization accurary
MNIST
CIFAR-10
Redundancy ratio
Test
9:3
Train Test
Aooonzzzai accurcC
6.
0/00
Redundancy
0.00
KoMfans
04
K-Meaan are
KNNN act
KINN
LR
acc
test wnc
1.
022
width
Layer witth
width
Layer width
(a) Determinism vs. layer width
(b) Categorization vs. layer width
MNIST
MNIST
MNIST
MNIST
03
500 opechs
10 spocra
Rundom data Truk data
Redundancy ratio
Fondon
Diameter
Frequency
Withh 42
52
0.4
84
0.5
04
Layer width
Diameter
Diameter
Training time (epoch)
(c) D vs. layer width
(d) D in different time
(c) D for random/truc data (f) R vs. random/true data
Figure 2: (a) Plots of redundancy ratio as a function of the layer width of MLPs on both training set (blue) and test set (red). (b) Plots of test accuracies of K-Means (blue), K-NN (red), and logistic regression (LR, range) as functions of the layer widths of MLPs. (c) Plots of the average stochas- tic activation diameter (D) as a function of the layer width of MLPs on MNIST. (d) Histograms of stochastic diameters (D) calculated on MNIST for an MLP of width 50 trained on MNIST for 10 epochs (blue) and 500 epochs (red), respectively. (e) Histograms of stochastic diameters (D) calcu- lated on MNIST (blue) and randomly generated data with the same dimension (red), respectively, for an MLP of width 50 trained on MNIST. The two red histograms are identical. (f) Plots of redun- dancy ratio (R) calculated on MNIST ("True data", solid lines) and randomly generated data (dotted lines) as functions of training time for MIPs of widths 40 (blue). 50 (red), and 60 (orange). The dotted lines show networks trained on unaltered data, evaluated with random data. The models are trained for 5 times on MNIST and 10 times on CIFAR-10 10 with different random seeds. The darker lines show the average over seeds and the shaded area shows the standard deviations.
Measure model size by width. In the context of MLPs, a natural measure for the model size is the layer width? We then calculate the redundancy ratio and categorization accuracy in all cases, as presented in Figures 2(a) and 2(b) From the plots, we can observe clear correlations between the encoding properties and the width: (1) the redundancy ratio starts at relatively high position (nearly 1 on both training and test sets of MNIST, around 0.1 on the training set of CIFAR- 10, and around 0.04 on the test set of CIFAR-10). Then, it decreases to almost 0 in all cases as the layer width increases; and (2) the categorization accuracy starts at relatively low position (about 25% on MNIST, and 32-45% on 10). Then, as the width increases, the accuracy monotonically increase in all cases to relatively high position (around 70% for K-Means both datasets, higher than 90% for K-NN and logistic regression on MNIST, and around 50% for K-NN and logistic regression on CIFAR-10, 10, similar to the test accuracy the raw data).
Measure model capacity by the diameters of linear regions. We design an average stochastic activation diameter
measure for evaluating the model capacity, which is calculated in three steps: (1) we sample a random direction via the uniform distribution; (2) we define the stochastic diameter of linear region to be the length of the longest intersected line segment of the linear region and the line along the sampled direction; and (3) we define the average stochastic activation diameter to be the mean of the stochastic diameters of all the linear regions containing data. Intuitively, a smaller average stochastic activation diameter means that the input space has been divided into
'Depth is also a natural measure for model size. However, the optimal training protocol (especially training time) for networks of different depths significantly differs. Thus, it is hard to conduct experiments depth while controlling other factors.
5

---

Under review as a conference paper at ICLR 2021
MNIST
CIFAR-10
CIFAR-10
CIFAR-10
CIFAR-10
Desundancy ratto
Taxt Thai
Wedd 40
CoMeo RCCuRCT
oth cth
Iedin 50
With
recurdaancy
wioth-00
(llww))
wotth111
w6theet
w000m00
w9020002
Training time (epoch)
training (spoch)
tainnngg time (spocl))
training time tepoch!
training time (opoch)
(a) R ratio vs. time
(b) R ratio vs. time
(c) Categorization accuracy vs. time
Figure 3: (a) Redundancy ratio (R ratio) as a function of training time on MNIST. (b) Redundancy ratio (R ratio) as function of training time on CIFAR-10. (c) Test accuracy of K-Means (left), K-NN (middle), and logistic regression (right) as functions of training time. The models are MLPs of widths 10 (blue), 20 (red), 40 (orange), and 80 (green). The dotted lines show networks trained on unaltered data, evaluated with random data. The darker lines show the average over seeds and the shaded area shows the standard deviations.
smaller linear regions, and thus can represent more sophisticated data structures. Therefore, it can serve as a measure of model capacity. Correspondingly. negative correlation between the layer width and the average stochastic activation diameter is observed, as illustrated in Figure 2(c)
Hanin Rolnick (2019b) also define a 'the typical distance from a random input to the boundary of its linear region.' In contrast, our diameter is intuitively the longest distance between two points in a linear region. When the linear region is an ideal ball, their distance is equal to or smaller than the radius of the ball, the half of our diameter. However, linear regions irregular in practice. Please refer to a visualization of the linear regions in Figure 1. Haainn & Rolnick (219b). Given this, the distances of Hanin & Rolnick (2019b) would be significantly smaller than our diameter. Overall, these two definitions would exhibit a significant discrepancy depending on the irregular level; one be even fixed when the other is significantly changed. Moreover, their distance can yield lower bound for the linear region volume, while ours can deliver an upper bound.
usually extremely
We also studied the encoding properties beyond the data generating distribution. A set of examples is generated according to the uniform distribution over the unit ball centered at the original point. The original data also normalized so that every pixel is in the range [0, 1]. Therefore, the scales of the random data and the original are comparable. We observe that the redundancy ratio is larger than 0.8 on the randomly generated data; see Figure 2(() This result suggests that the determinism property no longer stands, and correspondingly, one cannot represent randomly generated data by unique neural codes. Therefore, the categorization property also becomes clusive. We further propose the following hypothesis explain our findings.
Hypothesis 1.
The diameters of linear regions in the support of data distribution becomes smaller as the training progresses, while the diameters of regions far away do not change much.
We then collect the average stochastic activation diameters for each scenario, as illustrated in Figure 2(d) and Figure 2 We observe that the stochastic diameters are more concentrated when the training time is longer; see Figure Moreover, we observe interesting result that the stochastic diameters for true data is more concentrated at lower values than the stochastic diameters for random data. Figure 2 shows diagram for stochastic diameters. The diagrams for other scenarios are given in Appendix B These results fully support our hypothesis.
5.2
RELATIONSHIP BETWERN TRAINING TIME AND ENCODING PROPERTIES
We next investigate the influence of training time the encoding properties. The experiments are conducted based on one-hidden-layer MLPs with three different widths on MNIST and five- hidden-layer MLPs with four different widths on CIFAR-10. Totally, 810 models tested. The experiments are repeated for 5 trials on MNIST and 10 trials on 100 respectively.
We collect the redundancy ratio and the categorization accuracy of every epoch in all the scenarios, as presented in 3 Most results for MNIST are given in Appendix(B.2Jdue to space limitation. The plots clearly suggest positive correlation between the encoding properties and the training

---

Under review as a conference paper at ICLR 2021
CIFAR-10
CIFAR-10 10 (Test)
CIFAR-10 (Test)
CIFAR-10 (Test)
Test
Train
Width
wisth-10
Wieth
Width
wi5th-20
Width 20
witth
Wiath
wisthi00
width 40
Witth 40
Redundancy
2.3
Whath
0:35
wisth=B0
Witth BD
Wipth BD
62
Means
NN-X
0.24
UP
331
10
231
30
3M
23'
137
33'
Sample size
sample size
Sample size
Sample size
(a) R ratio vs. sample size
(b) Test accuracy of K M-Mans, K-NN, and logistic regression sample size
Figure 4: (a) Redundancy ratios (R ratios) on training set (dotted lines) and test set (solid lines) of CIFAR-10 as functions of sample size. (b) Test accuracy of K K-Means (left), K-NN (middle), and logistic regression (right) as functions of sample size. The models are MLPs of widths 10 (blue), 20 (red), 40 (orange), and 80 (green). All models are trained on CIFAR-10 for classification for 10 times with different random seeds. The darker lines show the average over seeds and the shaded area shows the standard deviations.
time: when the training time goes longer, (1) the redundancy ratio monotonically decreases; and (2) the categorization accuracy monotonically increases.
We also observe that the redundancy ratio of an untrained MLP on MNIST is almost O; see Figure 3(a) Our explaination is as follows. When a neural network is randomly initialized, the input space is randomly partitioned into multiple activation regions. If these activation regions are sufficiently small, almost every training datum has its own activation region. However, the mapping from put data to the output prediction is meanless at random initialization, because the neural network may output two completely different predictions to two datums from neighboring activation regions. Therefore, the categorization accuracy is poor, which is consistent with your understanding. This phenomenon also suggests that only determinism not sufficient measure the encoding proper- ties. It coincides with the reservoir effects (Jaeger 2001 Maass et al. 2002).
worth noting that our finding is different from the result in Hanin & Rolnick (2019b) that the linear region counting increases as training progressing. We reported that the encoding properties do not apply beyond the training data distribution, no matter how the linear region counting change; see Figure 20 This suggests that the increase of linear region counting would just happen in small part of the input space which is usually extremely large. Therefore, an increasing linear region counting cannot guarantee a decreasing redundancy ratio.
5.3 RELATIONSHIPP BETWEEN SAMPLE SIZE AND ENCODING PROPERTIES
We then investigate how sample size impacts the encoding properties. We trained 210 one-hidden- layer MLPs with three different widths and 480 five-hiddenllyyer MLPs with four different widths on training sample sets of different sizes randomly drawn from of MNIST and CIFAR-10, respec- tively, while all irrelevant variables are strictly controlled. We adopt the number of iterations rather than epochs to measure the training time because the number of iterations in one epoch grows pro- portionally with the sample size. The experiments are repeated for trials on MNIST and 10 trials on CIFAR-10. 10, respectively
We calculated the redundancy ratio and categorization accuracy in all cases; see Figures 4(a) a and
4(b) respectively. We only present here the results collected on CIFAR-10 due to space limitation. The results for MNIST are given in Appendix [B3 The plots suggest that (1) the redundancy ratio calculated on either the training sample set the test sample starts at considerably high position at initialization, and then decreases monotonically to near zero as the training sample size increases; and (2) the test accuracies of all the three algorithms have clear positive correlations with the sample size: the K-Means accuracy increases from 20% to 40%, the K-NN accuracy increases from 10% to 45%, and the logistic regression accuracy increases from 15% 45% respectively.
Surprisingly, we observe that the encoding propertics on the test set are also stronger when the training sample size goes larger. Our hypothesis is as follows. Intuitively, larger training sample
7

---

Under review as a conference paper at ICLR 2021
layer
Redundancy ratio
KM accuracy
Myye layar 5
Sayer5
844
accuracy
0.40
All layers
laye
9.00
gett Inpot
K-NN
byer2 2
apar3
18 1yyy 0.0
Wilayers
Training time tepoch)
Training time (epoch)
Training time (epoch)
Training time (epoch)
(a) Determinism vs. time
(b) Categorization accuracy vs. training time for different single layers
Categorization accuracy
Calegorizzlio accuracy
Redundancy ratto
Redundangy ratio
2.35
0.00
8.30
K-Mfens
0400
123 2034
19346
345 2345 29346
123
3004
12045
2044 bas4s
Layer indice
Layer indice
Layer indice
Layer indice
(c) Determinism for different layers
(d) Categorization accuracy for different layers
Figure 5: (a) Plots of redundancy ratio of neural codes formed by different single layers of MLPs trained on CIFAR-10 as a function of training time. (b) Test accuracy of K-Means (left), K-NN (middle), and logistic regression (right) functions of training time. (c) Redundancy ratios of neural codes formed by multiple layers of MLPs as functions of sample size. (d) Test accuracy of K-Means (left), K-NN (middle), and logistic regression (right) as functions of sample size. The models are MLPs of width 40 on CIFAR-10.
size supports the neural network to attain higher expressive power, i.e., the linear partition in the input space is finer. Meanwhile, sample of larger size requires finer linear partition to yield the same redundancy ratio. Our experiments show that the first effect is stronger than the second one. Thus, a larger sample size can help reduce the redundancy ratio.
5.4 HAYER-WISE ABLATION STUDY
We next study how different layers impacts the encoding properties. We conducted layer-wise ablation study based on five-hidden-layer MLPs on the CIFAR-IO dataset, where every layer of width 40.
We calculate the redundancy ratios and the categorization accuracy in all epochs; see Figure 5 Our results show that (I) the redundancy ratio of neural code formed by the first layer is almost always 0, while the categorization accuracy is relative poor; (2) the redundancy ratio gradually increases while the categorization accuracy gradually goes better when neural codes formed by higher single layers; (3) the impact of the training time on the encoding properties formed by single layer is similar that on the neural code formed by all layers; (4) the redundancy ratio monotonically decreases when the neural code is formed by more layers; (5) the categorization accuracy gradually increases when the neural code is formed by from the first layer gradually to the while network; (6) the previous property does not hold when the neural code formed by from the last layer gradually to the while network; and (7) the categorization accuracy of the neural code formed by the last layer is comparable with that for the whole network, which coincides with the previous two properties. The property (2) (especially the part on the redundancy ratio) reconciles the hashing property and the good generalizability of deep learning; the data is gradually concentrated to a smaller number of cells from the first layer towards the last layer, which helps neural networks generalize.
test the encoding properties of
5.5 IMPACT OF REGULARIZATION RANDOM DATA, AND RANDOM LABELS
We also studied the impact of regularization on the encoding properties. We trained 345 MLPs on the MNIST dataset with without batch normalization, gradient clipping, and weight decay. The
8

---

Under review as a conference paper at ICLR 2021
App Mandon clppinng
whb 300
wih gradient dipping
with weight decay
Wi wio gradient dipping
wib weight decay
(a) Impact of different regularizers determinism
(b) Impact of different regularizers on LR accuracy
Figure 6: (a) Scatter of redundancy ratios of MLPs trained on MNIST with (w/, y-axis) or without (w/o, z-asix batch normalization (BN, left), gradient clipping (middle), or weight decay (right). (b) Scatter of logistic regression (LR) accuracy of MLPs trained on MNIST with (w/, y-axis) or without (w/o, w-asix) batch normalization (BN, left), gradient clipping (middle), weight decay (right). Every point drawn from a model. Totally, 345 models are involved.
results suggest that regularization has an impact the encoding properties but relatively smaller than model size, training time, or sample size; see Figure We omit the accuracies K-Means and K-NN to the appendices due to space limitation.
We also generated random data via uniform distribution and trained MLPs and CNNs on it. Unfor- tunately, the training does not converge. We then added label noise with different noise rates (0.1, 0.2, 0.3) to MNIST. The encoding properties still stand though become relatively worse. Our results suggest that the structure of the input-data can drive the organization of the hashed space. Please refer to Table 5 in Appendix B.5 and Figure 12 in Appendix B.6 for more details.
5.6 ACTIVATION HASH PHASE CHART
We eventually can define an activation hash phase chart that characterizes the space expanded by redundancy ratio, categorization accuracy, model size, training time, and sample size. Summarizing the relationships discovered above, the activation hash phase chart is divided into three canonical regions: wnder-expressive regime, criticalll-exxprssivve regime, and sufficientll-exxressiie please see more details in Section T This chart can help us for typer-paramete tuning, novel algo- rithm designing, and algorithms diagnosis. We would also like to note that the thresholds between the three regimes are currently unknown. Exploring them is a promising future direction.
regime;
6 CONCLUSION
This paper studies the linear partition in the linear spaces neural networks with ReLU-like activa- tions. In this partition, every region corresponds activation pattern of the ReLU-like activations, which is parameterized by neural code this paper. We discover that the neural code behaves as the hash code of the corresponding example. Specifically, the neural code possesses the following encoding properties: (1) determinism: almost every linear region contain at most one example. Cor- respondingly, almost every example can be represented by one unique neural code. This property can be quantitatively evaluated by redundancy ratio, which is defined to be the proportion of the examples sharing neural code with others: and (2) categorization: simple classification and clus- tering algorithms, such as K-NN, logistic regression, and K K-Means can achieve fairly good training accuracy and test accuracy on the neural code space. These properties also exhibit variabilities in different scenarios. We then find that model size, training time, training sample size, regularization, and label noise contribute shaping the encoding properties, while the impacts of the first three are dominant. Accordingly, we define activation hash phase chart represent the space spanned by model size, training time, sample size, and the encoding properties, which divided into three canonical regions: under-expressive regime, regime.
regime, and sufficiently-expresse
ACKNOWLEDGEMENTS
The authors sincerely appreciate the anonymous reviewers for their constructive comments, which contributed significantly in improving the quality of this paper.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128, 2016.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01499 2016.
Yue Cao, Mingsheng Long, Bin Liu, and Jianmin Wang. Deep cauchy hashing for hamming space retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1229-1237, 2018.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Hashnet: Deep learning to hash by continuation. In IEEE International Conference on Computer Vision, pp. 5608-5617, 2017.
Yu-We Tui Chao, Sudheendra Vijayanarasimhan, Bryan Scybold, David A Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster r-cnn architecture for temporal action localization. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1130-1139,000
Thomas Cover and Peter Hart, Nearest neighbor pattern classification. IEEE Transactions on Infor- mation Theory, 1 13(1):21-27, 1967.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In IEEE Conference Computer Vision and Patlern Recognition, pp. 2625-2222222 2015.
Richard O Duda, Peter E Hart, and David G Stork. Pattern Classification and Scene Analysis, volume 3. Wiley New York, 1973.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 315-323, 2011.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, pp. 2596-2604, 2019a.
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. In Advances in Neural Information Processing Systems, pp. 361-370, 2019b.
Fengxiang He, Bohan Wang, and Dacheng Tao. Piecewise linear activations substantially shape the loss surfaces of neural networks. International Conference on Learning Representations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance imagenet classification. In IEEE International Conference Computer Vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016b.
Qiang Hu and Hao Zhang. Nearly-tight bounds on linear regions of piecewise linear neural networks. arXiv preprint wtth.1.811111199 2018
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In IEEE Conference Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13, 2001.
Donald E Knuth. The art of computer programming: Volume Sorting and Searching. Addison- Wesley Professional, 1998.
10

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Abhinav Kumar, Thiago Serra, and Srikumar Ramalingam. Equivalent and approximate transfor- mations of deep neural networks. arXiv preprint arXiv:1905.11422 2019.
Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan. Simultaneous feature learning and hash coding with deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3270-3278, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (((1)):2278232244 1998.
Stuart Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2): 129-137, 1982.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In International Conference on Machine Learning, pp. 3, 2013.
Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based perturbations. Neural Computation, 14(11):2531-256) 2002.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (Nov))25599-2605 2008.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems, pp- 2992-2933, 2014.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learn- ing Representations, 2020.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization neural networks: an empirical study. In International Conference Learning Representations, 2018.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint arXiv:1312.6098, 2013.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Expo- nential expressivity deep neural networks through transient chaos. Advances in Neural Information Processing Systems, pp. 3360-3368, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the ex- pressive power of deep neural networks. In International Conference on Machine Learning, pp- 2847-2854, 2017.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In International Conference on Machine Learning. pp- 4558- 4566, 2018
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pp. 568-556866660 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks. International Conference on Learning Representations Workshop, 2018.
Gül Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recogni- tion. IEEE Transactions Pattern Analysis and Machine Intelligence, 1500-151) 2017.

---

Under review as a conference paper at ICLR 2021
Jingdong Wang, Ting Zhang, Nicu Sebe, and Heng Tao Shen. A survey learning to hash. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):769-790, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European Conference on Computer Vision, pp. 20-36. Springer, 2016.
Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. Supervised hashing for im- age retrieval via image representation learning. In AAAI Conference on Artificial Intelligence, volume 1, pp. 2, 2014.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 1492-15000 2017.
Huan Xiong. Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. In International Conference on Machine Learning, 2020.
Li Yuan, Eng Hock Francis Tay, Ping Li, and Jiashi Feng. Unsupervised video summarization with cycle-consistent adversarial Istm networks. IEEE Transactions Multimedia, 2019.
Xiao Zhang and Dongrui Wu. Empirical studies the properties of linear regions in deep neural networks. In International Conference on Learning Representations, 2020.
Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. Deep hashing network for efficient simi- larity retrieval. In AAAI Conference on Artificial Intelligence, 2016.
Rui Zhu, Bo Lin, and Haixu Tang. Bounding the number of linear regions in local area for neural networks with relu activations. arXiv preprint arXiv:2007.06803, 2020.
12

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL EXPERIMENT IMPLEMENTATION DETAILS
Dataset. Our experiments based the MNIST dataset (LeCun et al. 1998) and CIFAR-10 dataset (Krizhevsky & Hinton 2 (1)MNIST has 60, 000 training examples and 10. 000 test examples are from 10 classes. One download this dataset
(2) CIFAR-10 is consisted of 50, 000 training images and 10, 000 test images which belong to 10 classes. One can download CIFAR-IO httpss//www.cs..c www.sssstooooo....oooo .duu kriz/cifar.html The splits of training and test sets follow the official versions. All the images are normalized so that every pixel value is [0. 1].
Training settings. (1) For MNIST: MLPs are trained by Adam for 2, 000 epochs with batch size of 128 and constant learning rate. VGG, ResNets, ResNeXt, and DenseNet are trained by Adam for 500 epochs with batch size of 128. Learning rate is initialed 0.01 and decays to the 1/10 of the previous value per 100 epochs. For all models, the hyperparameter /3 set as 0.9 and the hyperparameter $2 is set to 0.999. (2) For -IFAR-10: MLPs with 5 hidden layers are trained by Adam for 200 epochs with batch size of 64. Learning rate is initialed as 0.01 and decays to 1/10 of the previous value per 20 epochs. VGG and ResNet are trained by SGD for 200 epoch with batch size of 64. Learning rate is initialed as 0.01 and decays to the 1/10 of the previous value per 50 epochs. MLPs MNIST trained for five times with different random seeds. MLPs CIFAR-10 10 are trained for ten times with different random seeds.
Average stochastic diameter. We first trained MLPs with width {5, 10, 15, 20, 25, 30, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 100) on MNIST. Then, we randomly select 600 examples from the test set and calculate the mean of their corresponding stochastic diameters.
Network architectures. The network architectures involved in Section 4 are shown in the following Tables 3 and
Table 3: The detailed architectures of neural networks on MNIST.
VGG-18
ResNet-18
ResNet-34
ResNeXi-26
DenseNet-28
3 3, 32, stride maxpool, 3 x
x 3, 32, stride 2
3 x 32, stride
x 3, 32, stride maxpool,
x 3, 6, stride maxpool, x
maxpool, 3 3
maxpool, 3 x
x 3,32
3 3,32
1 1,32
1 1,
(3 3,33)
3 3,32,
x 3,32
3 3,32
3 3,3
1 1,64
x 3, 64
3 3 3,64
coner x
3, x
x
x
x
3,
avypool,
3 x 128
128
1,64 x C
1 1, 12 3 x 3,3 x
(3 x 3, 128) x
2
X
x
3 x 3, 128
x 3, 128
1x 1, 128
3 3, 256
x 3, 256
como, 1 x
(3 x 256)
3 256
3 3, 256
augpool, x
128
1 1, 12
3 3, 128, C= x 1, 256
x
3 3,3
avgpool
avgpool
avgpool
avgpool
avgpool
fc-10, softmax
fc-10, 10, sofimax
fc-10, sofimax
fc-10, sofimax
fc-10, softmax
Experimental designing for K-Means. The pipeline for the experiments on K-Means is as fol- lows: (1) we set K as the number of classes: (2) run K-Means on the neural codes and obtain K clusters; (3) every cluster can assigned a label from {1,2,...,10) label) pairs; (4) for every (cluster, label) pair, assign the label to all datums from the cluster and calculate the accuracy; and (5) we select the highest accuracy as the accuracy of the K- K-Means algorithm.
Thus, there are 90 (cluster,
13

---

Under review as a conference paper at ICLR 2021
Table 4: The detailed architectures of neural networks on CIFAR-10.
VGG-19
ResNet-18
ResNet-20
ResNet-32
(3 x 3, 32) x maxpool, 2 x 2
3 x 3, 64
3 x 3, 16
3 X 3, 16
(3 x 3, 128) x 2
3 3, 64
3 x 3, 16
3 x 3, 16
x
X 3
5
maxpool, 2 x (3 x 3, 256) X 1
x 3,64
x 3, 16
3 x 3, 16
x 3,88
3 x 3,32
3 x 3,32
X 2
3
5
maxpool, 2 x 2
3 3,
x 3,32
3 x 3,32
(3 x 3, 512) x 4
3 x
3 x 3, 64
3 x 3, 64
x 2
X 3
x 5
maxpool, 2 x 2
3 x 256
3 x 3,64
3 3,
(3 x 3, 512) x 4
3 x 3, 512
2
maxpool, 2 x
3 x 512
fc 1096
avgpool
avgpool
avgpool
fc 4096
fc-10, softmax
fc-10, softmax
fc-10, softmax
fc-10, softmax
Experiments concerning the relationship between model size and encoding properties. We trained MLPs of widths {3. 10, 15, 20, 23, 27, 27,30, 30, 33, 37, 40, 43, 47, 50, 53. 57, 60, 65, 70, 75, 80, 90, 100} on MNIST and 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200). on CIFAR-10
Experiments concerning the relationship between training process and model size. (1) For MNIST, We trained MLPs with widths of {40,50 50, 60). Redundancy ratio and the test aceu- racy of K-Means, K-NN, and logistic regression are calculated when the training epoch is in the list (1,3, 6, 10, 30, 60, 100,30 300, 600, 1000, 1200, 1500, 1800, 2000]. (2) For CIFAR-10, We trained MLPs with widths of {10, 20, 40, 80}. Redundancy ratio and the test accuracy of K- Means, K-NN, and logistic regression are calculated when the training epoch is the list of (1, 6, 10, 20, 30, 40, 60, 80, 100. 120, 140, 160, 180, 200}
Experiments concerning the relationship between sample size and model size. (1) For MNIST: We trained MLPs with widths of {40, 50, 60) training sample sets of size {10, 30, 60, 100, 300, 600, 1000, 2000, 3000, 6000, 10000, 20000, 30000, 60000) randomly drawn from the training training sample sets of size {10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 40000) ran- domly drawn from the training set.
(2) For CIFAR-10: We trained MLPs with widths of {10, 20, 40, 80} on
Experiments concerning the relationship between regularization and model size. Three regu- larizers are involved our experiments:
Batch normalization: we add a batch normalization layer before every ReLU layer.
Weight decay: we utilize L2 weight regularizer with hyperparameter À = 0.01.
Gradient clipping: we set clip norm as
Layer-wise ablation study. We trained MLPs with width of 40 on CIFAR-10, 10. The training strategy is the same as the one previously used on MLPs with CIFAR-10.
Experiments concerning random data All of the pixels of random data generalized from the uniform distribution (O0 1), individually. The shape random example is 28 x 28, i.e., the same MNIST images.
Experiments concerning noisy labels. Specified number of training examples of MNIST are as- signed random labels according the label noise ratios of 0.1, 0.2, and 0.3, respectively. Then,
14

---

Under review as a conference paper at ICLR 2021
trained one- seo-hidden-layee MLPs of widths f3, 7, 10, 20, 23, 33, 40, 43, 47, 50, 53, 60, 65, 70, 75, 80, 90, 100} on the noisy training set.
B ADDITIONAL EXPERIMENTAL RESULTS
This appendix collect experimental results omitted from the main text due to the space limitation.
B.1
ADDITIONAL RESULTS FOR THE DIAMETERS
The following figure is for the study the diameters. Please refer to Section 5.1
MNIST
MNIST
MNIST
MNIST
Random
p spocks
500 spachs
Frequency
Frequency
Frequency
Frequency
61
Diameter
Diameter
Diameter
Diameter
(a)
(b)
(c)
(d)
Figure 7: (a) Histograms of diameters of stochastic diameters calculated MNIST for MLP of width 60 trained on MNIST for 10 epochs (red) and 500 epochs (blue), respectively. (b) Histograms of stochastic diameters calculated MNIST (blue) and randomly generated data with the same di- mension (red), respectively. The model is an MLP of width 60 trained on MNIST. (c) Histograms of diameters of stochastic diameters calculated MNIST for an MLP of width 70 trained on MNIST for 10 epochs (red) and 500 epochs (blue), respectively. (d) Histograms of stochastic diameters cal- culated on MNIST (blue) and randomly generated data with the same dimension (red), respectively, The model is an MLP of width 70 trained on MNIST.
15

---

Under review as a conference paper at ICLR 2021
B.2
ADDITIONAL RESULTS FOR TRAINING TIME
The following figure shows the encoding properties about training time on MNIST. Please refer to Section 5.2
MNIST
MNIST
MNIST
MNIST
058
Redundancy ratio
Text
Train
K-Means accura))
3.5
Widh
B4M
Width 80
004
62
Width
Width
With
Width Width
Width
Width 60
Width
078
12"
1o
Training time (opoch)
Training time (epoch)
Training time (epoch)
Training time (epoch)
(a) Determinism vs. time
(b) Test accuracy of K-Means, K-NN, and logistic regression vs. time
Figure 8: (a) Redundancy ratio of MNIST as a function of training time. (b) Test accuracy of K-Means (left), K-NN (middle), and logistic regression (right) functions of training time. The models are MIPs of depths 40 (blue), 50 (red), and 60 (orange) on MNIST. The dotted lines show networks trained on unaltered data, evaluated with random data. All models are trained on MNIST for classification for 5 times with different random seeds. The darker lines show the average over seeds and the shaded area shows the standard deviations.
B.3 ADDITIONAL RESULTS FOR SAMPLE SIZE
The following figure shows the encoding properties about sample size on MNIST. Please refer to Section 5.3
MNIST Train
MNIST (Test)
MNIST (Test)
MNIST (Test)
Test
Redumdancy ratio
Width 40
3.63
Widhh 60
Width
4
Width
Wiah-50
Witth 60
Wisk-60
Witth 60
6.0
Sample size
Sample
Sample
Sample size
(a) Determinism vs. sam- ple size on hoth training and test sets
(b) Test accuracy of K MMeens, K-NN, and logistic regression vs. sample size
Figure 9: (a) Redundancy ratios on training set (dotted lines) and test (solid lines) of MNIST as functions of sample size. (b) Test accuracy of K-Means (left), K-NN (middle). and logistic regression (right) as functions of sample size. The models are MLPs of widths 40 (blue). 50 (red), and 60 (orange) on MNIST. All models are trained on MNIST for classification for 5 times with different random seeds. The darker lines show the average over seeds and the shaded area shows the standard deviations.
16

---

Under review as a conference paper at ICLR 2021
B.4 ADDITIONAL RESULTS FOR REGULARIZATION
The following figure shows the impacts of regularizers, gradient clipping and weight decay, on the redundancy ratio and the test accuracy of K-Means, K-NN, and the logistic regression.
Redundancy Ratio
K-Meass Accuracy
K-NN Accuracy
LR Accuracy
7-2: -0.00
2.00
Zi 2,0000
7-2 5.2
W/ B/
W BN
WM BN
wV BN
w/o BN
w/o BN
w/o
W/ gradient clipping
w/ gradient clipping
w/ gradient clipping
gradient clipping
9-2 0.63
w/o gradient clipping
w/o gradient clipping
w/o gradient clipping
w/o gradient clipping
W/ weight decsy
/-2=0.17
w/ weight decay
y
W/ weight decay
weight decay
5- 10 x0
xx
w/a weight: decay
w/o weight decay
w/o weight decay
w/o weight decay
First row: Scatter of redundancy ratios and test accuracy of K-Means (blue), K-NN (violet), and logistic regression (LR, orange) for MLPs of depth from to 100 with batch normal- ization (y-axis) and without gradient clipping (æ-assix). We perfer smaller ÿ F in redundancy ration, and larger ones in the test accuracies of K-MMans, K K-NN, and logistic regression. Totally, 115 models are involved in scatter. Second row: Scatter of redundancy ratios and test accuracy of K-Means (blue), K-NN (violet), and logistic regression (I.R, orange) for MIPs depth from to 100 with gradient clipping (y-axis) and without gradient clipping (æ-asix).. We perfer smaller - E in redundancy ration, and larger ones in the test accuracies of K K-means, K-NN, and logistic regression. Totally, 115 models are involved in one scatter. Third row: Scatter of redundancy ratios and test accuracy of K-Means (blue), K-NN (violet), and logistic regression (LR, orange) for MLPs of depth from 3 to 100 with weight decay (y-axis) and without weight decay (æ-asix). We perfer smaller F in redundancy ration, and larger ones in the test accuracies of K -Means, K-NN and logistic regression. Totally, 1 115 models are involved one scatter.
Figure 10:
17

---

Under review as a conference paper at ICLR 2021
B.5 ADDITIONAL RESULTS FOR RANDOM DATA
Figure B.5 is an example of random data and every pixel of it is generated from the uniform dis-
tribution (0, 1). Table reveals one-hidden layer MLP with width 100 fails fit random data.
Table 5: Training accuracy and loss on training random data with one- Ame-hidden-layer MLPs
Epoch
0
100
300
500
Training acc (%)
10.92
11.24
11.24
11.24
Figure 11: An example of random data
Loss
230.56
230.13
230.13
230.13
B.6 ADDITIONAL RESULTS FOR RANDOM LABEL
The following figures show the impacts of random label on redundancy ratio and the test accuracy of -Means, K-NN, and logistic regression. Please refer to Section 5.3
Label nose 0.1
K-Meens s03urao
Labelnore
LR accuracy
Redundancy
KNNN
Labelinoks) 0011
Label noise
Label nois
Labelinots 0.2
nolce 0.2
Label 02
Label
Label
Leyer wicth
Layer width
Layer width
Layer width
(a) Determinism in differ- (b) Test accuracy of K-Means, K-NN. and logistic regression in different label ent label noises
noises
Figure 12: (a) Redundancy ratios of MNIST functions of layer width in different label noises.
(b) Test accuracy of K-Means (left), K-NN (middle), and logistic regression (right) as functions of layer width in different label noises. The models are MLPs trained in different label noises (blue), 0.1 (red), 0.2 (orange) and 0.3 (green) MNIST. All models are trained on MNIST with noise labels for classification for times with different random seeds. The darker lines show the average over seeds and the shaded area shows the standard deviations.
18


Reviews:
[{"review_id": "bIOWQHLFaJN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This draft proposes to use the relu activation pattern of the neurons in the neural network as the hash code for the input. Essentially the input features are bucketized into small piecewise linear regions. The authors show empirically that the proposed hash code has small collision and high accuracy given certain conditions including\n1. The features are around the sample manifold \n2. the training time is long enough\n3. the network is wide enough\n4. the training sample size is large enough\nThe authors also found empirically the effect of regularization is relatively small on the encoding properties.\n\nI feel this is an interesting thought but I am not sure if this is the first work on it. Some other possible issues:\n1. Figure 4(a) somehow shows the redundancy is the smallest at epoch 0, then it goes high after 1 epoch and decreases slowly as the number epochs grows. Can authors provide some explanation on this? Does it suggest the random initialization of the neural network gives a good hash code in terms of the redundancy metric? (of course the accuracy will be bad)\n2. The authors used K-means as another benchmark to compare. To me k-means is an unsupervised clustering algorithm. How do you get the accuracy from k-means? How do you match the cluster id to the labels?\n3. My biggest complaint is that only the MNIST data set is investigated in the experiment. MNIST is too easy to show any conclusive results. You may need to work on other data sets such as image net, cifar 100, or NLP related data set to draw a convincing conclusion.\n4. All the conclusions are purely empirical. Can authors provide some explanation or intuition on why the redundancy ratio decreases as the training time grows? Is this related to the type of optimizer being used? Why can a larger sample size also help reduce the redundancy ratio?\n\n\nOverall I think this draft has some really good ideas but the empirical result is not quite conclusive due to the lack of extensive experimentation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "W30ojOnalj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after rebuttal:** I thank the authors for their detailed responses and the additional experiments. The responses addressed most of my concerns. I noticed that I had the wrong notion of redundancy ratio in my mind (I'm glad the authors now give a more formal definition of this concept as I think this would trip up many other readers). I'm also glad that the authors have clarified the difference between their results and those reported in Hanin and Rolnick (ICML, 2019). Given these, I'm happy to increase my score to a weak accept (a weak accept, because I'm still not quite sure about the significance of the results reported in this paper). \n\n--------------------------------------------------------------\nThis paper reports some observations about properties of linear regions in deep ReLU networks. Unfortunately, I have several major issues with the paper. \n\n(1) First of all, I am a bit confused about the motivation behind this work. The motivation is initially couched in terms of hash codes, so one gets the impression that the authors are going to propose a new hash coding scheme using neural networks. But this is clearly not the case. The proposed coding scheme is practically useless as a hash code because of the enormous dimensionality of the codes (I estimate this to be on the order of millions even for the toy MNIST case studied in the paper, but it would actually be very helpful for the reader if the authors explicitly mentioned the dimensionality of the proposed hash codes--at least the order of magnitude--). This is also why the authors are stuck with the toy MNIST dataset throughout the paper, because the proposed scheme is completely impractical for any reasonably large dataset and model.\n\n(2) This begs the question: what exactly is the significance of the observation that linear regions satisfy some properties of hash codes, if they’re not going to be used as hash codes? It’s not meaningful to just point out that something satisfies some properties of hash codes. One can point to a million different things that satisfy some properties of hash codes. What exactly is the significance of linear regions having these properties?\n\n(3) This brings me to the related works section. This section is written very shallowly, the authors do not do a good job of situating their contributions in the context of prior works. You cannot just say: “Other advances in linear region counting include …” (p. 3) and then cite a bunch of references. You have to tell us what each of these papers did and how what you’re doing in this paper differs from these earlier works and makes a meaningful and novel contribution to the prior literature.\n\n(4) The concept of redundancy ratio seems to play a central role in the paper, but it is not defined formally, just a verbal (potentially ambiguous) definition is given. Please define this concept formally to avoid any ambiguities. I’m also not sure this concept is the right one for quantifying the goodness of a code: consider a case where each point in a dataset shares its linear region with exactly one other point in the dataset vs. a case where all points belong to the same linear region. It seems that the redundancy ratio will be 100% in both cases, however intuitively the code in the first case should be much better than in the second case (for example for retrieval).\n\n(5) Relatedly, Figure 3c suggests that the redundancy ratio is close to zero for an untrained random network. Then, by the authors’ own definition, the encoding is actually very good before training. Please consider what this means (also see point 7 below).\n\n(6) Unfortunately, I don’t think classification results for MNIST only are very meaningful. Almost anything will get above 99% accuracy on MNIST. Moreover, no effort is made by the authors to understand what drives good test accuracy in these experiments. A very straightforward explanation is that accurate classification is primarily driven by higher layers, so one actually doesn’t need most of the dimensions in the hash code for good classification performance (similar “cache” models using high layer features have been proposed before: e.g. Dubey et al., CVPR 2019; Orhan, NeurIPS 2018; Khandelwal et al., ICLR 2020).\n \n(7) Most importantly, one of the main phenomena observed in this paper (the change of redundancy ratio over training) has already been reported in Hanin and Rolnick (ICML, 2019): they note that the number of linear regions in a deep ReLU net first decreases and then increases during training (please read their section 3 carefully). This would easily explain the trajectory of the redundancy ratio observed in Figure 3c (and in Figure 4a) in this paper. Moreover, the concept of diameter is also rigorously defined (and diameters of linear regions studied) in Hanin and Rolnick (ICML, 2019), but this is not acknowledged at all by the authors. This is a pretty serious omission. \n\n(8) Typos: should be: “Mapping induced by a relu network” (p. 1), “it is worth noting” (p. 2)\n“Geometric properties of linear regions” (p.3). “Another diameters of linear regions”? (p. 5)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UXdEFurD4yE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after authors' response**\nI am very happy to see the additional results on CIFAR-10, and the layer-wise ablations and other control experiments. To me, these results have shed a lot of light onto how the observed hashing effect can be explained. These explanations mostly confirm intuitions. Nonetheless I think it's worth reporting the empirical verification of these intuitions that tie in with earlier results reported in other papers. To me, the most interesting aspect of the work is how the hashing properties change over training (and across layers). The paper could still be improved by experiments on e.g. CIFAR-100, ImageNet and non-vision tasks, as well as more mathematically sophisticated definitions of some of the measures (e.g. average stochastic activation diameter). I personally think the results are now sufficient (and sufficiently backed up) for a publication and most of the criticism raised by the other reviewers has been addressed sufficiently (for me). I would now rate the paper as a 6.5 - but to facilitate the reviewer's discussion I will take a clear stance and have thus raised my score to 7.\n---\n\n**Summary**\nThe paper investigates activation patterns in ReLU networks, which are known to be piecewise linear. The main finding is that trained MNIST classifiers produce unique activation patterns for most points from the data-distribution (but not for random data). To be precise, an activation pattern is a binary matrix where each entry corresponds to one ReLU unit, and is 1 if the unit has non-zero activation and 0 otherwise. This means that trained MNIST classifiers can be viewed as “hashing” each data-point into a unique activation pattern. Importantly, simple clustering/classification algorithms (K-means, K-nearest-neighbors, and logistic regression) on these hashed patterns lead to good classification accuracy, implying that the hashed patterns follow some underlying geometric regularity (which is not trivially expected from an arbitrary hashing function).\n\n---------------------------------------------------------------------\n**Main contributions, Novelty, Impact**\n1) Systematic study of the “hashing” behavior of MNIST classifiers. To the best of my knowledge this particular type of analysis is novel. The findings are interesting, though perhaps not totally surprising. The impact of the results is currently limited by 2 important factors: (i) results are shown on MNIST only, and (ii) it is currently unclear whether the hashing effect is mostly explained by early layer activations, or not. To address (ii) layer-wise ablations would be crucial (see improvements for more details).\n\n2) Various ablations and control experiments to identify the impact of certain hyper-parameters and architectural variations. The ablations and control experiments are interesting to further characterize the observed hashing effect. Two important control experiments are currently missing: (i) results for untrained networks, and (ii) results when fitting and random labels (to determine whether the observed effects are mainly input-data-driven, or whether the hashed landscape is mainly shaped by label information).\n\n3) Definition of an “Activation Hash Phase Chart” with three qualitatively distinct reasons. The main idea behind this is interesting, but the paper never shows such a diagram, and (importantly!) how the chart is meant to be used (hyper-parameter tuning, or rather as a diagnostic tool, …?). It is thus currently unclear whether such a chart could have wider impact in the community.\n\n---------------------------------------------------------------------\n**Score and reasons for score**\nThe main observation in the paper is interesting and supported by a number of experiments. Unfortunately the generality of the findings is currently fairly limited since evaluation happened for MNIST classification only, and some control experiments are missing. I think the ingredients for a strong and interesting paper are there, but it does not quite come together yet. I therefore currently suggest a major revision of the work, which means resubmission at another conference. I personally want to strongly encourage the authors to spend a bit more time and work on the paper to turn it into the strongest version possible - in which case I think the paper could be quite impactful.  I am of course happy to reconsider my final verdict based on the other reviews and the authors' response.\n \nPerhaps the most important unaddressed point is how the observed hashing property can be reconciled with the good generalization performance: if each test-datapoint is mapped to a unique pattern, how does the classifier generalize well? I suspect that the hashing property is mostly a property of early layers that gets washed out towards the output of the classifier (but it might also be the case that the hashing property holds essentially until the final layer). I think the hashing property itself is not very surprising, but what’s surprising is that the “goodness-of-hash” changes quite a bit over training - not only for training points but also test data, but not random data. Shedding more light on this would make the paper much stronger in my opinion.\n\n---------------------------------------------------------------------\n**Strengths**\n * Observed hashing effect and hypothesis stated fits quite well with the wider literature (which is also well cited in the paper).\n * The experiments that are conducted are thorough (multiple repetitions per run, covering a broad range of hyperparameters).\n * To me Fig 3c is the most convincing result that the hashing effect is (at least partly) an caused by training and not simply a generic feature neural networks with a certain width.\n---------------------------------------------------------------------\n**Weaknesses**\n * Results are on MNIST only. Historically it’s often been the case that strong results on MNIST would not carry over to more complex data. Additionally, at least some core parts of the analysis does not require training networks (but could even be performed e.g. with pre-trained classifiers on ImageNet) - there is thus no severe computational bottleneck, which is often the case when going beyond MNIST.\n * The “Average stochastic activation diameter” is a quite crude measure and results must thus be taken with a (large) grain of salt. It would be good to perform some control experiments and sanity checks to make sure that the measure behaves as expected, particularly in high-dimensional spaces.\n * The current paper reports the hashing effect and starts relating it to what’s known in the literature, and has some experiments that try to understand the underlying *causes* for the hashing effect. However, while some factors are found to have an influence on the strength of the effect, some control experiments are still missing (training on random labels, results on untrained networks, and an analysis of how the results change when starting to leave out more and more of the early layers).\n\n---------------------------------------------------------------------\n**Correctness**\nOverall the methodology, results, and conclusions seem mostly fine (I’m currently not very convinced by the “stochastic activation diameter” and would not read too much into the corresponding results). Additionally some claims are not entirely supported (in fullest generality), based on the results shown, see comments for more on this.\n\n---------------------------------------------------------------------\n**Clarity**\nThe main idea is well presented and related literature is nicely cited. However, some of the writing is quite redundant (some parts of the intro appear as literal copies later in the text). Most importantly the writing in some parts of the manuscript seems quite rushed with quite a few typos and some sentences/passages that could be rephrased for more fluent reading.\n\n---------------------------------------------------------------------\n**Improvements (that would make me raise my score) / major issues (that need to be addressed)**\n1) Experiments on more complex datasets.\n\n2) One question that is currently unresolved is: is the hashing effect mostly attributed to early layer activations? Ultimately, a high-accuracy classifier will “lump together” all datapoints of a certain class when looking at the network output only. The question is whether this really happens at the very last layer or already earlier in the network. Similarly, when considering the input to the network (the raw data) the hashing effect holds since each data-point is unique. It is conceivable that the first layer activations only marginally transform the data in which case it would be somewhat trivially expected to see the hashing effect (when considering all activations simultaneously). However that might not explain e.g. the K-NN results.\nI think it would be very insightful to compute the redundancy ratio layer-wise and/or when leaving out more and more of the early layer activations (i.e. more and more rows of the activation pattern matrix). Additionally it would be great to see how this evolves over time, i.e. is the hashing effect initially mostly localized in early layers and does it gradually shape deeper activations over training? This would also shed some light on the very important issue of how a network that maps each (test-) data-point to a unique pattern generalize well?\n\n3) Another unresolved question is whether it’s mostly the structure of the input-data or the labels driving the organization of the hashed space? The random data experiments answers this partially. Additionally it would be interesting to see what happens when (i) training with random data, (ii) training with random labels - is the hashing effect still there, does the K-NN classification still work?\n\n4) Clarify: Does Fig 3c and 4a show results for untrained networks? I.e. is the redundancy ratio near 0 for training, test and random data in an untrained network? I would not be entirely surprised by that (a “reservoir effect”) but if that’s the case that should be commented/discussed in the paper, and improvement 3) mentioned above would become even more important. If the figures do not show results for untrained networks then please run the corresponding experiments and add them to the figures and Table 1.\n\n5) Clarify: Random data (Fig 3c). Was the network trained on random data, or do the dotted lines show networks trained on unaltered data, evaluated with random data?\n\n6) Clarify: Random data (Fig 3). Was the non-random data normalized or not (i.e. is the additional “unit-ball” noise small or large compared to the data). Ideally show some examples of the random data in the appendix.\n\n7) P3: “It is worth noting that the volume of boundaries between linear regions is zero” - is this still true for non-ReLU nonlinearities (e.g. sigmoids)? If not what are the consequences (can you still easily make the claims on P1: “This linear region partition can be extended to the neural networks containing smooth activations”)? Otherwise please rephrase the claims to refer to ReLU networks only.\n\n8) I disagree that model capacity is well measured by layer width. Please use the term ‘model-size’ instead of ‘model-capacity’ throughout the text. Model capacity is a more complex concept that is influenced by regularizers and other architectural properties (also note that the term capacity has e.g. a well-defined meaning in information theory, and when applied to neural networks it does not simply correspond to layer-width).\n\n9) Sec 5.4: I disagree that regularization “has very little impact” (as mentioned in the abstract and intro). Looking at the redundancy ratio for weight decay (unfortunately only shown in the appendix) one can clearly see a significant and systematic impact of the regularizer towards higher redundancy ratios (as theoretically expected) for some networks (I guess the impact is stronger for larger networks, unfortunately Fig 8 in the appendix does not allow to precisely answer which networks are which).\n\n---------------------------------------------------------------------\n**Minor comments**\nA) Formally define what “well-trained” means. The term is used quite often and it is unclear whether it simply means converged, or whether it refers to the trained classifier having to have a certain performance.\n\nB) There is quite an extensive body of literature (mainly 90s and early 2000s) on “reservoir effects” in randomly initialized, untrained networks (e.g. echo state networks and liquid state machines, however the latter use recurrent random nets). Perhaps it’s worth checking that literature for similar results.\n\nC) Remark 1: is really only the *training* distribution meant, i.e. without the *test* data, or is it the unaltered data generating distribution (i.e. without unit-ball noise)?\n\nD) Is the red histogram in Fig 3a and 3b the same (i.e. does Fig 3b use the network trained with 500 epochs)?\n\nE) P2 - Sufficiently-expressive regime: “This regime involves almost all common scenarios in the current practice of deep learning”. This is a bit of a strong claim which is not fully supported by the experiments - please tone it down a bit. It is for instance unclear whether the effect holds for non-classification tasks, and variational methods with strong entropy-based regularizers, or Dropout, ...\n\nF) P2- The Rosenblatt 1961 citation is not entirely accurate, MLP today typically only loosely refers to the original Perceptron (stacked into multiple-layers), most notably the latter is not trained via gradient backpropagation. I think it’s fine to use the term MLP without citation, or point out that MLP refers to a multi-layer feedforward network (trained via backprop).\n\nG) First paragraph in Sec. 4 is very redundant with the first two bullet points on P2 (parts of the text are literally copied). This is not a good writing style.\n\nH) P4 - first bullet point: “Generally, a larger redundancy ratio corresponds a worse encoding property.”. This is a quite hand-wavy statement - “worse” with respect to what? One could argue that for instance for good generalization high redundancy could be good.\n\nI) Fig 3: “10 epochs (red) and 500 epochs (blue),” does not match the figure legend where red and blue are swapped.\n\nJ) Fig 3: Panel b says “Rondom” data.\n\nK) Should the x-axis in Fig 3c be 10^x where x is what’s currently shown on the axis? (Similar to how 4a is labelled?)\n\nL) Some typos\nP2: It is worths noting\nP2:  By contrast, our the partition in activation hash phase chart characerizes goodnessof-hash.\nP3: For the brevity\nP3: activation statue\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SDqOQqU29av", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the problem of linear partition in the linear spaces of neural networks with ReLU-like activations is studied. It demonstrates that such a partition exhibits two properties, including determinism and categorization, across a variety of deep learning models \n\n————————————————————————————————————————————\n\nThis paper presents an interesting work, however, there are a few issues/comments with the work:\n\n1.This paper mentions that “simple classification and clustering algorithms, such as K-NN, logistic regression, and K-means can achieve fairly good training accuracy and test accuracy on the neural code space”,  About “achieve fairly good training accuracy and test accuracy”, maybe some comparisons with existing methods will strength this statement.\n\n2.The paper concludes that model capacity, training time, and sample size play important roles in shaping the encoding properties, while several popular regularizers have little impact on the encoding properties.  As for the conclusion of the regularization technique, it is better to try other methods, such as drop-out, to verify gradient clipping and weight decay not only on MLP but also on RNN, before reaching such a conclusion. Moreover, only for MLP on MNIST, it seems unconvincing.  It will be better to try out some existing/well-known NN models on more datasets. \n3.I was confused about the sentence below Section 5.4 “we trained 345 MLPs on the MNIST dataset” and the last sentence in the caption of Figure6 “Totally, 115 models are involved in one scatter. “. If possible, could you please explain that?\n\n4.In Figure 2 (c) and (d), “Diameter“ on the horizontal axis correspondings to the “Layer width” in Figure 2 (a) and (b)\n\n5.What is the meaning of “Frequency” on the vertical axis?\n\n6.In Figure 6, the third and fourth subfigures mean that for some of MLPs with depths between 3 and 100, “ without BN” would be better than  “with BN”, BN sometimes harms the performance of these models? is that so?\n\n7.It needs some space between the second and the third paragraphs below Figure 2.\n\n8.In deep models, depth generally is more important than width. Why only analyze the model capacity by width?\n\n9.For the definitions of three regions “\tUnder-expressive regime”, “Critically-expressive regime”, and “Sufficiently-expressive regime”, may not need to be repeated on page 8, as similar definitions are explained on page 2. Also, is that possible to quantify/formalize them? If so, it would be more interesting and useful.\n\n10.The format of references is inconsistent, for example, references “Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al.. 2017” and “Li Yuan, Eng Hock Francis Tay, Ping Li, and Jiashi Feng. 2019”.\n\n==========================================================================================\n\n\"In addition, I am not sure the description would be enough to reproduce and no code seems to provide.\" In the beginning, I did not find the codes related to this paper, but later, the author(s) uploaded codes. Thanks.  For the codes,  if setting random seed program-wide in codes maybe it will be more helpful for reproducing. \n\n==========================================================================================\n\n————————————————————————————————————————————\n\nOverall, I think this is an interesting piece of work that may be of interest to people to explore/understand (deep) neural networks, but I think the results/conclusions require more careful/extended analysis.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the behavior of the intermediate ReLU-like activations of trained neural networks and show empirically that the intermediate activation can be used as a hashing function for the examples with some key advantages, including almost no collisions and that there are desirable geometric properties (i.e. can use k-means, k-nn, logistic regression on these embeddings).

Pros:
- The experimental analysis was solid and thorough investigating the effects of model size, training time, training set, regularization and label noise. 

Cons:
- An overall lack of novelty. It is already quite well-known throughout the ML community that in many cases, using the intermediate embeddings serve as useful features to apply more classical methods such as kNN and clustering.

Overall, the reviewers appreciated the solid and thorough investigation into the hashing properties of neural network activation patterns, which convincingly confirms some intuitions about the behavior of activation patterns in neural networks. However, the reviewers also agreed that there was no significant new finding. There have already been many studies on clustering and kNN on the embeddings of a network.
Thus, the core novelty of the paper appears to be the finding that almost every linear region has at most one datapoint after training (which does not seem too surprising given Hanin & Rolnick (ICML 2019)); however, without further novel implications of this finding, the impact of the paper is limited.
Decision: Reject