Under what circumstances do local codes emerge in feed-forward neural networks
Authors: Ella M. Gale
Nicolas Martin
Venue:  (2020)
Keywords: localist coding
emergence
contructionist science
neural networks
feed-forward
learning representation
distributed coding
generalisation
memorisation
biological plausibility
deep-NNs
training conditions

Abstract: Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCs emerging in response to invariant features, and this finding is robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, increasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep networks (2 hidden layers) which have many LCs lose them when common aspects of deep-NN research are applied (large training data, ReLU activations, early stopping on training accuracy and softmax), suggesting that LCs may not be found in deep-NNs. Switching to more biologically feasible constraints (sigmoidal activation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are not found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by a lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise of LCs be required, this work suggests how to tune a NN so they emerge. 

Under review as a conference paper at ICLR 2020
UNDER WHAT CIRCUMSTANCES DO LOCAL CODES EMERGE IN FEED-FORWARD NEURAL NETWORKS.
Anonymous authors Paper under double-blind review
ABSTRACT
Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCS emerging response to invariant features, and this finding robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, in- creasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep net- works (2 hidden layers) which have many LCs lose them when common aspects deep-NN research applied (large training data, ReLU activations, early stop- ping on training accuracy and softmax), suggesting that LCs may not be found deep-NNs. Switching to more biologically feasible constraints (sigmoidal ac- tivation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise LCs required, this work suggests how to tune NN they emerge.
INTRODUCTION
With neural networks (NNs) being widely deployed in various tasks it is essential to understand how they work and what data is used make their decisions. NNs used to be viewed as 'black boxes', but recent results (Nguyen et al. (016)) have started to open that box. NNs came from the field of psychology as simple bio-inspired models, has been debated whether information is represented in the brain in distributed manner (from parallel distributed processing, PDP) or via a localist coding scheme. Although the distributed approach was the most popular, there are some results in neuroscience (Quiroga al. 2005) and psychology (McClelland & Rumelhart 1981) that are commensurate with a localist coding scheme, including a report of LCs in RNNS [Bowers al., 2014). Recently, there has been an explosion of interest in NNs, especially deep- NNs, as these algorithms are now commercially relevant, and this increase in their accuracy has been credited sources of vastly more labelled data and novel training techniques like dropout (Srivastava et al. 2014). Many newer researchers in NNs were perhaps unaware of the distributed- localist coding debate within psychology. and thus looked for localist-like codes in their NNs, and found indicative (of L.C coding scheme) evidence of detectors for objects (Zhou et al. 2018 2015).
concepts (Karpathy et al.) 20166] Lakretz al. 2019). features (Nguyen et al. 2019 Erhan et al. 2009), textures (Olah et al. (017). single directions (Morcos al. 2018 etc., (Bowers 2017 for a review.
With faster and larger computers, it is possible, even with the increase in input data size, for deep- NNs to 'memorise' the data-set (an extreme form of overfitting): a process where the NN has simply learned mapping between input and output vectors, opposed to learning rule which will allow it to generalise to unseen data that follows the underlying rule. Generalisation performance is often improved if NN training is stopped early, often when validation set loss (val loss) stops improving.

---

Under review as a conference paper at ICLR 2020
as the NN is prevented from further minimising its loss function by memorising the input (training) data. Single directions (Morcos et al. 2018)) have been implicated in memorization the data-set.
Localist codes (coding for class A) are defined as units which are activated at a high (low) level for all mehb the NN gets correct) and low (high) level for all members of the other classes (class -A), i.e. the set of activations for class A is disjoint from the activations for class -A (see figure [ in the appendix), and these codes are very strict measure of selectivity. As such, LCs are very easy to interpret, and the presence of them in NNs would make it easy to understand how the NN is working.
This paper takes no position on whether not localist codes exist in the brain in deep-NNs. instead we take the constructionist science approach of asking when would we expect LCs to appear, and what aspects of the system, data-set and training conditions favour or disfavour their emergence. As NNs are considered (simplified) models for the brain, we can also take into account biological plausibility. We hypothesized that LCs should emerge when there was an invariant in the data-set. As deep-NNs take a long time to train, it is hard to get representative statistics, so we look at very simple networks (shallow: 3-layer and pseudo-deep: 4-layer) where possible to do hundreds of repeats and thus get resilient trends.
The main insight of this work that L.C do emerge when there is an invariant in the data. To set up system with such 'short-cut" we use simple binary vectors inputs, which built from prototypes, such that there are 1/10 input bits that are always for each class and these are the invariant bits, the Os of each prototype are then filled in with a random mix of and 0 of known weight, see figure [ thus, a given bit is always on for a given class, and maybe on or of for other classes. Note also, that in this up, if the proportional weight of the prototype exceeds that of the random vector, then vectors belonging to the same class are 'closer" to each other than those of separate classes, i.c. there is larger between-class variance than within-clatt variance. The prototypes are also perturbed to increase the variance of the 'invariant' bits. If one views a deep conv- NN a feature extraction machine (lower and convolutional layers) with a feature classification NN on top (the higher fully connected layers), then it is reasonable to suppose that given class is likely to share features the top convolutional layer, which would result the activation vectors at that layer having higher between group variance than within group, possibly even invariant features for a class (perhaps object detectors), and so these experiments could give insight into the representation of data in the 'fc' layers of deep-NNs.
1.1 FINDINGS
1. LCs related to lower within-class variance than between-class variance in input
2. LCs related to a NN internalising rule
3. No. of LCs related to difficulty of the problem and the computing power of the NN, with different behaviour for under- and over-resourced NNs.
4. Large values of dropout increases L.C
5. LCs correlate with generalisation performance
6. Large data-sets, softmax and aggressive early stopping reduce the number of LCs
7. Monitoring the number of LCs can be useful for figuring out when to stop training
METHODOLOGY
Data design Data input a neural network can be understood as code, {Ca, ). with each trained input data vector designated as a codeword, Cz. The size of the code is related to the number of codewords (i.e. the size of the training set), nzz, Lg is the length of the codeword, generally 500bits in this paper. We used a binary alphabet, and the number of 1s in codeword is the weigh w. of that codeword.
1 'There are two metrics that are relevant measuring the distance between these vectors, the Hamming distance which is the number of bits that have to he switched to turn one vector into another and the cosine similarity, which is the angle hetween the vectors, we use Hamming distance here. "his weight definition is the same as connection weights in the neural network.
2

---

Under review as a conference paper at ICLR 2020
Py Pa
Figure 1: Schematic for building a random code with known properties. Black circles represent ones, white circles represent zeros. Class prototypes (Pi, P22. and P3) were made with length Lp; the number of prototypes are np, which is three in this example, their weight is four, with sparseness number, Sp of 3. Random vectors, Rar were made, as shown, these have length LR and there are na of them; they have weight, w/2 of two and sparseness number, Sn, of 1 To assemble new codeword, a prototype is chosen, in this example, P2. and 'perturbation" errors are applied, in this example, the perturbation rate is 3. 80 a single one is turned to a zero in the modified prototype (P²). A random vector, in this example R1. then generated, split into blocks and added the parts of the prototype (P1) that were zero (the random vectors cannot overwrite a random decay in P). The process is repeated to create an input 'code' with n, codewords of length Iur, where V2 n and 1 Lp. If the decay values is sufficiently low, members of each class, as they are based on the same prototype are more similar to each other than codewords in other classes.
To create a set of classes with a known structural similarity, the procedure in figure was followed. We start with a set of np prototypes, {Pa,] 1 < x S np), with blocks of is of length Lp ip/np, called prototype blocks, which code for a class. For example, if L, were 12 and np were P3 = [00000000111]], and this would gives
3: Pi = [111 100000000] and P- = prototypes that are Hamming distance of 8 apart, and thus we know that our prototypes span input- data space. To create members of each class, the prototype is used as a mask, with the 0 blocks replaced by blocks from random vector, R.. The weight of the random vectors, wn can be tuned to ensure that a set of vectors randomly clustered around the prototype vector are generated, such that members of the same category are closer to each other than those of the other categories (N.B. the prototypes are not members of the category), A more realistic data-set is created by allowing the prototypes be perturbed so that percentage of the prototype block is randomly switched to Os each time a new codeword is created, in accordance with the perturbation rate (see P/2 in figure []. This method creates a code with a known number of invariant bits for codewords in the same category. For example, in figure I codewords Cj and C2 were both derived from Pi, and have a Hamming distance of 6, where as Ci and C4 are different classes and have a Hamming distance 8. Note, the difference between these numbers is larger in experiments as L 500 and there are 10 categories. We define 'sparseness' of a vector, Sz, as the fraction of bits that are 'I's LCs were highly unlikely to appear in the input code, and none were observed in random checks.
Neural network design We have three-layer feed-forward network with L, input neurons, NLLN hidden layer neurons (HLN) and L, output neurons, using a sigmoidal activation function and no softmax on the output. For experiment L is 500bits, mapped to 10 output classes, so the weight of prototype vectors Wp. is 50, and the wR is 150 (so S,=1/3), the output vector is a 50bit-long distributed vector (w,=25). NNs were trained for 45,000 epochs, and each plotted point is a number of repeats between 10 and 15. Experiment 2-7 varied thus: 2: ng {250, 500]: 3: SR (²/9- 2/e. //3::40 4: L. = {300, 700, 1000); 5: activation function=(ReLU, sigmoid} J: 6: output vector is distributed or 1-hot; 7: takes NHEN = 500, 1000 and decays the wr from 50 to 25 (P from 1 0.5).
Experiment 8 is ~50 repeats of experiment 1, with values of dropout in {0, 0.2, 0.5, 0.7, 0.9}. Ex- periment is a repeat 1, with activation noise added for networks with n HLN = {100, 500, 1000). Experiment 10 measures the number of LCs over training time for NHEN {250, 500, 1000, 2000); }. To do generalisation tests, a new test set is built with the same parameters as the training set, with Meröin 10000 and applied to pre-run results (from experiments 8 and 10). Experiment 11: for the layer neural networks, LL of the first hidden layer is varied, the second is set to 250, the output vectors are 1-HOT and different training parameters and values are given in table 11 Experiment
3

---

Under review as a conference paper at ICLR 2020
12: For the MLP experiments, use MNIST data-set, with added 20 pixel invariants that code for the class which are either non-varying ( ('invariant') ) or drawn from a Gaussian distribution ('Gaus- sian'), see table 8 in the appendix. The invariant was either not applied ('standard") or applied to 50% of 100% of all images, or applied to the whole of 2, 5 or categories, and data is from 10 repeats. Experiment 13: To see if LCs were associated with memorising the dataset, trained NNs where the codewords were shuffled before being assigned to targets. NNs were run in Keras with TensorFlow backend.
Experiment
Activation function sigmoid
Dropout 20% yes sigmoid no no
Softmax no
Size of
Stopping condition
training set
A
nr 100
60,000 steps
B
100
validataion set loss
C
sigmoid yes no
nr 100
60,000 steps
D
sigmoid yes yes
100
validation set loss
E
sigmoid
no no
fhe 100
training accuracy
F
sigmoid yes
n 100
training accuracy
G
ReLU
yes
no
fg 100
validation set loss
H
ReLU
yes
Yhr 100
60,000 steps
I
ReLU
yes
no
Yhr 100
training accuracy
ReLU
no
yes
fh 100
60,000 steps
K
ReLU
no
no
fh 100
validation set loss
L
ReLU
no
nh 100
training accuracy
M
ReLU
n0
no
1000
60,000 steps
N
ReLU
n0
no
n5 100
60,000 steps
O-S
ReLU
1000
Table 1: Training conditions for experiment 11. *O-S included tests for softmax, dropout and the three training conditions.
Accuracy and generalisation. The accuracy reported by Keras counts an output vector correct each bit is within 50% of the correct value, c.g., the output vector [0.4, 0.6] would map to the target vector [O, 1] (the outputs binary), and in standard classification neural networks with 1- hot target vectors these outputs would be very close to the target after the softmax operation. Thus, label this accuracy 'classification accuracy'. If one wants to machine, then one could put an arbitrary limit on how big an error between the output values and the targets, chose 10%, and the target [0, 1] would need an output vector of [< 0.1, 2 0.9] to be considered correct. We call this more stringent condition the 'pattern matching accuracy." As the codes are built rule, it is possible generate an arbitrarily large code, thus our test sets are at least 10 times larger than the training sets.
NN
pattern matching
3 RESULTS
As the chance that all the members of A would emerge disjoint from the members of -A is (50) (500) tiny (4.32 10-71), finding a single interpretable local code, such those shown in figure 8 in the appendix, refutes the idea that neural networks do not have interpretable or locally encoded units.
Local codes are seen in response to an invariant As shown in fig. 2a the number of local codes tuned by the size the hidden layer, with a peak the number local codes seen at 10000=1000 for the standard data set, and a peak in the percentage of HLNs which are local codes seen at NHLN 500 (data not shown): dashed and dot-dashed grey lines are drawn at these points in all relevant figures. The shape of the graph (in fig(2 appears to be an interaction between the difficulty of the problem and the amount computing power (i.e. the NHLN) available, with those networks to the left of the peak ('under-powered" with respect to LC emergence NNs) not having enough spare units to give over to localist code, and those to the right ( foverr-pwereed" Note that, in the results that follow, the data for NHLN is often qualitatively different for over- and under-powered NNs; for example, the number of LCs decreases over time for longer times or the underpowered NNs in experiment 10 and figure 3a and increases for over-power NNs.
4

---

Under review as a conference paper at ICLR 2020
Increasing the difficulty of the problem (by increasing Rz, see fig Za increasing Le see fig. g in appendix. or decreasing the relative information in the prototype by decreasing sparsity of SR. see fig. decrreass the number of local codes. The proportionately larger the invariant is in the imput code, the more likely LCs are to emerge in the NN, presumably as a 'rule' that attends only to the invariant is correspondingly more efficient. In the MLP experiments, the presence of invariant features in pixel space did cause small number of LCs to emerge (see table 9 in the appendix), and these systems trained much faster (these were trained to a given accuracy), see tables I0 in the appendix).
60
No. L.C.5
S 9
2,=250
150
40
550
S
30
D2- 1000
100
Spme
0
1000 2000 3000 4000 5000 6000 7000
(a) Experiment and 2: The number of local codes (LCs) against the number of HLNs, NHLN. for different numbers of training ex- amples (1a)
2000 4000
6000
5000 1000
No. HINss
(b) Experiment 3: The number of local codes against LNLN for varying the sparse- ness of the random blocks of the vector, SR.
Figure 2: Input data that is few in number and sparse exhibits more local codes. As the prototype vector all these cases has a sparseness of 1 10- the weight of the random wR and prototype, wpp, parts of the codeword and the codewords are 500bits long, note that purple: S. 0.2, wR=50, Wp = 50; green: S₂ = 0.3, wn=1000, wp = 50; black dashed: S₂ = 0.4, wn=150, wp = 50 Gray dashed and dot-dashed lines are drawn at RH 5500 and 1000 respectfully.
LCs increase with Training time and may relate to generalisation Figure 3a shows that the number of LCs in a NN increases with training time all across the range of TLLNN The corner of the LC-epoch curve (Fig. 3b) was roughly correlated with the system achieved 100% classification accuracy ("easy' criteria) on the training data and the asymptote on the test data (around 95%), see figure [e] Interestingly, this point is also when the pattern-matching accuracy ('hard' criteria) on both the train and test data-sets rises above O (fig suggesting that the additional LCS emerging are caused by the separation of is and Os in the NN output. Thus, monitoring the rate of LC increase can give measure of when to stop training.
To see if LCs were related to memorisation, we investigated the experiment with the greatest range in LCs-the 90% dropout shown in figure and and the training and test accuracy is plotted in fig. 4a There were many solutions that had an accuracy > 99%, but where the solutions were lower accuracy, the accuracy on both train and test data many times larger than the training set (n.= 10,000 for test, n = 500 for training) this shows that the NNs have clearly internalised the rule, the rough positive correlation between the number of LCs and the accuracy (in the accuracy range of 20-100%) demonstrates that LCs are part of the internalisation of this rule. Similarly, plotting the test accuracy against the number of LCs for experiment 10 (fig 4b) shows a positive correlation between the no. of LCs and classification accuracy (similar results are seen with the mateehing accuracy, see figure [2 in the appendix). Shuffling the input code to force the NN to memorise the data resulted in a reduction in the number of LCs, but only by a maximum 47% (see fig. 13 in the appendix).
related to the number of LCs. Note that, as the test set was
Why might LCs not be seen in modern deep NNs? Figure [ shows the effect of changing the system setup and output vectors. There is little difference between ReLU and sigmoid, except that the peak of the curve is shifted little to the left in the ReL.U data, likely as ReLU is more ppoerfull
[ activation function than sigmoid. An alternative explanation is that as ReLUs train faster and the number of local codes increases with training time (see fig 3a). the ReLU networks are
"No ICCs were seen in the standard MNIST runs. i... its range is not limited at the positive end so can use larger number space than sigmoid
5

---

Under review as a conference paper at ICLR 2020
No. L.C.s
300
L.C.s
250
0.020
250
500
200
150
0.010
1000
2000
100
0.005
50
0.000
10000
20000
30.000
40000
Repoohs
10000
20000
30000
40000
(a) Local codes increase with training time
(b) Taking differences identify the turning points of the curves. Accuracy
(Repoeha Accuracy
100
100
80
250
250
500
500
1000
40-
1000
2000
2000
20
20:
10000 20000
30000
40000
Popccha
10000
20000
30000 40000
Pepocta
(c) Classification accuracy on train (solid)
(d) Pattern matching accuracy on train (solid) and test (dashed).
and test (dashed).
Figure 3: Experiment 10. The number of local codes increases with training time. The point which the rate of LC code addition decreases correlated with both when the NN achieves ~99% classification accuracy and when it achieves classification accuracy of more than Key: red: NELN 250; blue; L = 500; black: nHLN 1000; purple: THLN 2000.
Acc.
No. of LCs
350
300
60
Test
200
Train
40
150
20
50
LCs
Acc
20
60
100
120
80
(a) Relation between classification accuracy and train for the 90% dropout exper- iment (8).
(b) Relationship between the number of LCs and classification accuracy on from periment 10.
Figure 4: The number of local codes increases with improvement generalisation accuracy. Fig over 250 repeats trained for the same amount of time, Fig 4b are repeated measurements on the same NNs as they trained.

---

Under review as a conference paper at ICLR 2020
effectively trained for longer compared to sigmoidal networks. We chose to distributed output code in the 3-layer networks as we thought the local codes at the output layer might encourage the emergence of LCs in the hidden layer: the opposite happened, likely because simple 3-layer network set up with prototypes like this, the problem of mapping input to output is linearly separable and much easier, hence LCs not needed. Note that, in deeper 4-layer HLNs with 1-hot output vectors there are many local codes (see B in figure 6bb.
No. L.C.s
No. L.S.s
40
30
ReLU
25
H-Hot
30
Sigmoid
20
Distributed
20
15
10
1000 2000 3000 4000 5000 6000 7000
1000 2000 3000 4000 5000 6000 7660
(a) Experiment Effect of changing the ac- tivation function.
(b) Experiment 6. Effect of changing the output form.
Figure 5: Network architecture parameters can drastically effect the emergence of local codes. 5a Switching from HLNS with a sigmoidal activation function to a rectified linear (ReLU) units; 5b Switching from a distributed 1-hot output encoding.
To test NNe we compared 4-layer NNs with our standard set-ups (A, B) that exhibited LCs with similar architecture and standard CNN training techniques added (e.g. L,, O-S). NNs with all the settings commonly found in deep-NN literature do not produce LCs. Figure (auure/aassoww that there is little difference between stopping on val_loss or after 60K epochs (as does the overlap between bars A and in fig. 6b L, M, O-S had ReLU and large data-sets, which completely inhibited local codes. H to S used ReLU rather than sigmoid,
in these systems it seems that ReLU-based NNs do not learn local codes, although they did in our earlier experiments (fig. (). That C and D are so similar shows real effect of dropout in these NNs. That C<A, D<B, (and fig. (6a) shows that softmax reduces the number of local codes slightly. We propose this is because softmax does some extra 'processing' on the final layer, shifting 50% accurate output vectors up 90%, and this extra processing makes the problem easier for the hidden layers (so the 'rule' is less useful). Alternatively, due to this extra "processing' using softmax reduces the time required train the network, which is expected to reduce the number of LCs, see figure Ba (N.B. with early stopping conditions training time now varied). Thus, the reason that LCs are not found deep-NNs could be the use of large training data sets, early stopping, ReLU and softmax in those systems, or alternatively it could be because there are no feature invariants associated with output classes at the top convolutional layer (i.e. the layer before the fully connected classification layers).
100
80
No. L.C.s
Soltra
60
40
0
ABODEFGH
Dataset
(a) Changing stopping conditions.
(b) Results for experiment 11
Figure 6: 'Pseudo-deep' NNs that exhibit LCs lose them if modern deep-learning training techniques are applied.

---

Under review as a conference paper at ICLR 2020
3.1 BIOLOGICAL CONSTRAINTS ENCOURAGE THE EMERGENCE OF LCS
As the positive activation range of the ReLU is unlimited, it is less biologically plausible than the sigmoid activation function (although these NNs are still very far from being model of biology). In this section we investigate noise on sigmoidal, non-softmax NNs. As real biological systems are (possibly) noisy we looked several different ways of introducing noise into the system: decaying the invariant, dropping out units and adding in activation noise.
Perturbing the prototype block is shown in figure 7al and LCs are still seen if the invariant is not always present, but simply more likely to be present for given class. However, the number of LCs disappears when the decay is 0.4, i.e. weight of the (50-bit long) prototype section is 20bits, and this is roughly equal to the weight of a 50bit long section of the random vector (16.67bits), at this point the signal is indistinguishable from the noise. The curves are not well fit by an exponential, the fits shown here are: (for N = 500) 20.62 30.65 VP, with R²² = 0,984; (for HNLN = 1000) 19.72 - 32. 29.29 / with R² 0.977, and these data were fit up to P=0.4. Comparing to the MLP case, there were no LCs if the invariant feature was perturbed, see table 9
Contrary to our expectations, adding dropout and increasing it to very large values increases both the average and maximum number of LCs in solutions found by the neural network, although lower values of dropout seemed to overlap, with two competing possible underlying distributions perhaps between 0 and 40% (see fig. i0 appendix). Summary statistics and Kolmogorov-Smingg hypothesis tests reported in tables ?? in the appendix, and above 50% dropout there is signifi- cant difference in the underlying distributions with an increase in number of LCs. Repeated copies of local codes may be more resilient noise than distributed code: a fully distributed code will always be incomplete but local code only has chance of being not present under dropout.
Adding in activation noise changes the number of LCs that are found (see fig. II in the appendix), increasing them in overpowered NNs and decreasing them in underpowered networks.
No. LC.s
PDF
20
8.10
0.6
15
500
10
THIN= 1000
0.04
0.38
No. LfS
8.0
20
40
60
00
100
120
0.1
0.2
0.3
0.5
(b) Probability Density Function (PDF) and har charts for NNs with different values of dropout (in legend).
(a) Local codes decrease with 'perturbation' of the prototype.
Figure 7: How noise affects local codes. Fig 7a perturbing the prototype part of the codeword decreases the drive to learn local codes. Perturbation. P, is measured as the number of bits in prototype block code that are randomly flipped. Fig.7b] increasing dropout increases the number of local codes. As the dropout percentage increases, generally, the mean and range local codes found increases, suggesting that localized encoding by the network offers some advantage against noise
4 DISCUSSION
The number of local codes is not large, usually between 0-10% of the hidden layer, but this is not insignificant and the fact that they emerge in response to invariance in the data-set and some train- ing conditions suggest that they have a function (presumably detecting the presence of an invariant which can be used as a 'short-cut'). These results are resilient to a small perturbation of the in- variant (in the NNs), the invariant feature does not need to be always present, merely more likely
'It is commonly suggested that living neural networks are noisy and thus that neurons and synapses must be resilient noise, however there is some debate whether this is true and some neurons are highly un-noisy.
8

---

Under review as a conference paper at ICLR 2020
to be present for specific class. LCs emerge in correlation with generalisation performance, so monitoring the rate of change of LC numbers can give feedback on when to stop training to avoid overfitting, As LCs were found in response to added invariants in 2D graphical data in MLPs, we expect that the use of LCs to identify and code for found 'rules' based on invariant features will be a general phenomenon. MMrcos al. (2018) found that 'single direction' codes were associated with the harder problem of memorising the dataset (as compared to learning the simpler underlying rule), here we found that LCs emerged in response learning a simpler rule, and were inhibited by switching to memorisation task. Further work is needed to tease out which aspects of the difference architecture and problem choice between Morcos et (2018)'s 's study and ours is the cause.
It's an open question whether there are localist codes in deep-CNNs, PDP dogma suggests not, but there have been some highly selective units found in other types of NNs. Our results suggest that there might not be LCs in the fully connected layers if typical current deep-CNN training methods are used. Furthermore, there is the question of whether there are invariant features coming from the conv/pooling layers into the fully connected feed-forward layers (of typical current deep-CNN) which is not currently known and an area of active research for us.
As LCs were seen when the NN has slightly more biologically plausible training conditions like noisy system, it might be possible that LCs might emerge in the human brain, although even the more biologically plausible NNs are perhaps too far removed from actual biology.
REFERENCES
Istvan SN Berkeley, Michael RW Dawson, David A Medler, Don P Schopflocher, and Lorraine Hornsby. Density plots of hidden value unit activations reveal interpretable bands. Connection Science, 7(2):167-187, 1995.
Jeffrey S Bowers. Parallel distributed processing theory in the age of deep networks. Trends in cognitive sciences, 21(12):950 961, 2017.
Jeffrey S Bowers, Ivan Vankov, Markus F Damian, and Colin J Davis. Neural networks learn highly selective representations order to overcome the superposition catastrophe. Psychological review, 121(2):248-261 2014.
Dumitru Erhan, Yoshua Bengio. Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. Workshop Track International Conference on Learning Representations, 2016.
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in 1stm language models. arXiv preprint arXiv: 100.007333 2019.
James L McClelland and David Rumelhart. An interactive activation model of context effects in letter perception: I. an account of basic findings. Psychological review, 88(5):375, 1981.
Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representattinss 2018. URL URL httptttss///peeeeiiiii http://w...e...eeeeeeeeeeeee fevet/forummiddee. i
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pp. 3387-3395, 2016.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualiza- tion: A survey. arXiv preprint arXiv: 0,99399,999 2019.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.239915ddisttll.0007 http:///dsillllllo00/0///aeeeeai
Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 435(7045): 100- 2005.

---

Under review as a conference paper at ICLR 2020
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 191):1929-1958 2014.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene CNNs. In International Conference on Learning Representations, 2015.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 2018.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.I LOCAL CODE EXAMPLES
Jitterplots are common analysis techniques in neuroscience, but can also be applied to neural net- work et al. (1995) by plotting the activation of unit on the x axis and randomly jittering the y-axis value. Examples local codes found in our networks are given figure g these were sigmoidal activation units. As it takes more energy to spike than not spike, in biological neurons the localist codes are generally 'on' codes where the unit is on more strongly in the presence of the class it codes for than not, figure figute In our system there is energy penalty for using high activations (in fact, energetic considerations are not modelled at all, and this one the reasons NNs are not biologically plausible) so also 'off" codes: figure iight.
12
15
0.8
Figure 8: Examples interpretable local codes found in a distributed network. Left: a selectively on unit with selectivity of ~ +0.12 Right: selectively off unit with selectivity of ~ -0.2. Red circles belong to single category, blue stars are all the members of all other categories, the x-axis the
the activation of hidden layer neuron (HLN) and points jittered randomly around y-axis for case of viewing. There is clear separation between activations for the class depicted in red (A) and all other activations (not-A), thus examination of the activations of these units would reveal the presence or absence of the red class.
A.2 EXPERIMENT 4: CHANGING INPUT VECTOR LENGTH (Lz)
As shown in figure there is little difference between different lengths of vectors for smaller lengths, although the number of LCs decreased at Lı 1000.
No. L.C.s 40
30
L,=300
L, 550
20
700700
L,=1000
10
0
7000 HLN
HLNN
1000 2000 3000 4000 5000 6000
Figure 9: Experiment 4. The effect of changing the input vector length.
11

---

Under review as a conference paper at ICLR 2020
A.3 FURTHER RESULTS FROM EXPERIMENT 8: DROPOUT
Dropout (see Srivastava et al. 2014)) is a common training technique where a percentage of a layer's neurons are "dropped out' of the network (their connections are zero) during training to prevent over-fitting, however, dropout can also be viewed as a type of training noise. Results are shown in figure expected (mean) and maximum number roughly increases with an increasing percentage of dropped out neurons.
?? and table 2 There are always solutions with close to zero local codes, but the
Generally, dropout percentages in the range of 20-50% are used in training, and these probability distribution functions (PDFs), like 0% peak, are also joint peaks, with the 20% data having more of the lower solution and the 50% having more of the higher one. Dropping out more than 50% of the network not generally used as slows down training. However, with these higher values, the range of solutions is much higher (as evidence by a higher variance and range of the number of local codes), which is expected as dropout forces the network to adopt a range of solution sub-networks, the increase in local codes suggest that localised encoding offers some protection against noise.
At first glance, this might seem unlikely, as distributed patterns are claimed to be more resilient against failure. However, say we had 20% dropout rate, fully distributed encoding, would be affected by dropout 100% of the time, losing 20% of its information, whereas a localised encoding would be unaffected 80% of the time (although 20% of the time it would lose all data), and further resilience can be provided if duplicate local codes were used for the same class. Note that, as only 10 classes were used, the large number of local codes, especially for the high dropout values, suggests there are multiple LCs for each category. These results suggest that, for a noisy network, solutions involving some duplicate localised codes are useful methods for dealing with uncertainty.
Table 2: Various quantities associated with the distribution of local codes (LCs) in with dropout (given percentage) applied during training
No. LCs
0%
20 %
50%
70%
90%
Minimum
8
15
0
Maximum
34
29
41
125
Mean
18.44
16.13
20.65
34.54
73.80
Standard deviation
4.55
4.19
4.96
8.05
24.53
To test if the different datasets could have come from the same distribution we did Kolmogorov- Smirnov hypothesis tests with the hypothesis being that the data were from the same distribution. As can be in seen tables ?? and figure ?? low values of dropout are likely the same distribution, but the distributions are different for the high values of dropout. The expanded set of experiments is shown in figure [0] and there seems be underlying distributions between and 50% with the 10%, 20% 30% dropout NNS having more of the distribution with less LCs, and 0%, 40% and 50% having more of the distrubtion with more LCs.
Dropout
Dropout
Result
p-value 1.
0%
0%
Do not reject
0%
10%
Reject
0.0000489944
0%
20%
Reject
0%
30%
Reject
0. 0.0000247196
0%
40%
Do not reject
0.137226
0%
50%
Reject
0.000251174
0%
60%
Reject
2.424097737663309881000011
0%
70%
Reject
1.3379086295029556x100
0%
90%
Reject
5.31281662567244 662567248x000
Table 3: Kolmogorov-Smimov hypothesis tests for the dropout experiments compared to a run with
no dropout.
12

---

Under review as a conference paper at ICLR 2020
PDF
0%
0.10
10%
0.08
20%
30%
0.06
40%
50%
0.04
60%
70%
0.02
90%
No. of LCs
0
20
60
100
120
PDF
0%
0.10
10%
0.08
20%
30%
0.06
40%
50%
0.04
60%
70%
0.02
90%
0
10
20
30
40
50
60
Figure 10: Extra data for the dropout tests.
Dropout
Dropout
Result
p-value
10%
0%
Reject
0.0000489944
10%
10%
Do not reject
1.
10%
20%
Do not reject
0.690898
10%
30%
Do not reject
0. 0.142326
10%
40%
Reject
0.0134828
10%
50%
Reject
3. 3.93507749988555406 10-11
10%
60%
Reject
1.227 x 10-46
10%
70%
Reject 1.9931398445589927 10-100
10%
90%
Reject
L.65492879805339366 x 10-133
Table 4: Kolmogorov-Smimno hypothesis tests for the dropout experiments compared to a run with
10% dropout.
13

---

Under review as a conference paper at ICLR 2020
Dropout
Dropout
Result
p-value
20%
0%
Reject
20%
10%
Do not reject
0.690898
20%
20%
Do not reject
1.
20%
30%
Do not reject
0.786929
20%
40%
Reject
0.000721937
20%
50%
Reject
4.515284336852291
20%
60%
Reject
205608442999664427110-
20%
70%
Reject 2.295194833448 01999199988388977771000--11
20%
90%
Reject
L.24300433932266433 x 10-146
Table 5: Kolmogorov-Smirrov hypothesis tests for the dropout experiments compared to a run with
20% dropout.
Dropout
Dropout
Result
p-value
30%
0%
Reject
0.0000247196
30%
10%
Do not reject
0.142326
30%
20%
Do not reject
0.786929
30%
30%
Do not reject
1.
30%
40%
Reject
0.0012839
30%
50%
Reject
77366477097586233111-
30%
60%
Reject
3.04037573854777xx004
30%
70%
Reject
5.3922880922299991110091
30%
90%
Reject
Table 6: Kolmogorov-Smimoy hypothesis tests for the dropout experiments compared to with
30% dropout.
Dropout
Dropout 0%
Result
p-value
40%
Do not reject
0.137226
40%
10%
Reject
0.0134828
40%
20%
Reject
0.000721937
40%
30%
Reject
0.0012839
40%
40%
Do reject
40%
50%
Reject
2.47677224452160033xx1999
40%
60%
Reject
944066207-2296775xx10-88
40%
70%
Reject
1.222853347997046xxx1-77
40%
90%
Reject
1.780277971174446x10-
Table 7: Kolnogorov-Smirmov hypothesis tests for the dropout experiments compared to a run with
40% dropout.
14

---

Under review as a conference paper at ICLR 2020
A.4 RESULTS FROM EXPERIMENT 9: ACTIVATION NOISE
We added in Gaussian distributed activation noise around a mean of 0.5 to all HLNs (which were sigmoidal, the activation varied between 0 and 1), the results for three networks shown in figure The addition of noise changes the distribution LCs seen. For the overpowered NNs, those above the RHLN peak in experiment 1, the number of local codes increases significantly in most cases. For the underpowered NNs (NHLN = {100, 500}) the number of local codes decreases, and this is not significant for tightly distributed noise.
No. L.C.s
25
1000 HLNs
500 HLNs
20
100 HLNs
15
10
5
1.0 0
0.2
0.4
0.6
0.8
Figure 11: Including activation noise affects the number of local codes, a is the standard deviation of the Gaussiancy distributed noise. For NN with 1000 HLNs adding noise generally increases the number of local codes, for NN with 500 HLNs there is generally no effect or decrease with high sigma. Lines are drawn to mark the standard error around the mean of LCs at O, i.c. no added noise.
A.5 FURTHER DATA FOR EXPERIMENT 12: MULTI-LAYER PERCEPTIONS.
Examples of the modified MNIST input data given in In the invariant case block of 20 (2.5%) hot pixels (fully on) are positioned around the edge of the image, thus a NN could learn to attend the hot pixels in order to correctly assign the category instead of learning letter shapes. the Gaussian case the prototype code perturbed by having the pixel value drawn from different Gaussian distribution for each image (this is the equivalent of the perturbation of prototypes in experiment 7. The networks were trained with no regularisation and no validation set.
identify the
The results for the MLP experiment (11) is given in table There is small, but significant emer- gence of LCs in response to categories with added invariants, but it has to be for all images in a category for the NN develop local code to look for the invariant. In the Gaussian examples, no local codes emerge. Given that the invariants were less <2.5% of the input data, and the result that perturbation removes local codes above a certain level, we suspect that were the prototypes perturbed less or bigger, LCs might have been seen with the perturbed invariants.
reliable code, which the NN can learn instead of learning to identify the number shapes. The NN trains much faster the problem is solving a simpler one. Thus, the presence of invariant adds 'short-cut' to the task which the NN can find.
Table 10 shows the impact on training times of the presence
A.6 FURTHER CENERALISATION RESULTS FOR EXPERIMENT 10: TRAINING TIME
Figure shows the correlation between the number of local codes and classification accuracy and pattern matching accuracy the test data. For NNs with HLNSs to the left of the peak (i.e. NHLN < 1000) show a slight negative correlation, those at the peak or above show a slight positive correlation, although there is lots of spread in this data.
15

---

Under review as a conference paper at ICLR 2020
Invariant
Gaussian
Invariant
Gaussian
S
5
6
6
4D
2
2
7
7
3
3
8
8
4
9
9
Table 8: Example hot-pixel invariant MNIST input data images (after normalisation from the mean image). 'Invariant" images have hot-pixel short-cut code (of white pixels) that is the same for all examples of that class, 'Gaussian' invariant images have greyscale hot-pixel code that is drawn from a Gaussian distribution and different for each image.
data-set
Invariant 100units
Gaussian
48units
500units
48units
100units
500units
Standard
0::0
0::0
0=:0
0.00
04:0
0.:0
50% all images
0+:0
0+:0
0=:
0+:0
0+:0
0.:0
All images
1.1+0.26
1.5=0.32
0.6:0.255
0+:
0+:0
01:0
2 categories
0.140.09
0.1=0.09
0.1:00.99
0+0
0+:0
0+:0
categories
0.2--0.19
0.8++0.31
0.6.-0.25
0+-0
0++0
0++0
8 categories 1.4++0.03
0.9+0.26
0.9+-0.22
0+-0
0+00
0++
Table 9: Average numbers of LCs in 4-layer MLPs trained on MNIST (standard) and modified MNIST with added hot pixel invarients.
16

---

Under review as a conference paper at ICLR 2020
data-set
Average no. epochs
Average no. epochs Gaussian distribution
Invariants
Standard
1090.8
1090.8
50% all images
983.4
979.9
all images
221.7
460.9
2 categories
1040.9
1025.0
5 categories
494.3
537.0
8 categories
327.5
462.3
Table 10:
Training times for MLP networks with different MNIST training Note that the presence of a invariant reduces training time because the NN finds the shortcut (all NN are trained >99% acc), and introducing GAussian distribution over that invariant (i.e. making it more variable) generally increases the training time.
Accuracy
NHLN
gradient
intercept
R
pattern matching
250
25.7
-0.18
0.23
pattern matching
500
68.8
-0.01
0.0002
pattern matching
1000
130.7
0.82
0.26
pattern matching
2000
174.6
2.43
0.41
classification
250
26.5
-0.08
0.23
classification
500
68.7
-0.001
6.6x 10
classification
1000
123.0
0.42
0.34
classification
2000
144.0
1.33
0.50
Table 11: Fits for the trend lines drawn in figures
and 12
A.7 RESULTS FOR EXPERIMENT 13: SHUFFLING THE INPUT DATA
We tried shuffling the input data which should force the NN to memorise the dataset, rather than relying on the short-cut. We expected the number local codes to drop to zero, but as shown in figure 13 this did not happen, instead the number of local codes only reduced.
ACKNOWLEDGMENTS
An. Author would like acknowledge funding from Leverhulme on grant no.
17

---

Under review as a conference paper at ICLR 2020
No. of LCs
No. of LCs
350
350
300
300
250
250
200
200
150
150
100
50
Acc
Acc
10
20
50
20
40
60
80
100
(a) The number of local codes increases with general- isation accuracy.
(b) The number of local codes increases with training accuracy.
Figure 12: Local codes are generally positively correlated with accuracy. Key: red: NHEN 2000;
LLLN 1000; NUIN = 500; WHIN 250.
80
150
60
No. L.C.s
No. L.C.
100
20
50
0
100
100
500
500
1000
1000
100
100
500
500
1000
100
1000
nx
(a)
250
(b)
500
Figure 13: Shuffling input data to create 'memorising networks' reduces the number of LCs but does not completely inhibit them. red: unshuffled data (control); blue shuffled input data.
18


Reviews:
[{"review_id": "rygJbaI6cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the emergence of local codes in neural networks on a synthetic dataset. From my understanding, a neuron is counted as a local code if there is a class A such that its activations of data points from A are linear separable from its activations of data points from all other classes. However, is this definition for data points in the training set, or in the test dataset, or union of them? I did not find the exact definition in the paper.\n\nIt designed experiments to study the number of local codes. It made 7 empirical findings by the experiments on a synthetic dataset, listed in Section 1.1. It's findings are purely empirical. The authors may clarify this work's novelty and importance.\n\nThis paper seems to be finished in rush, because there is question masks, e.g., \"Summary statistics and Kolmogorov-Smirnov hypothesis tests are reported in tables ?? in the appendix\" in Page 8, \"Results are shown in figure ?? and table 2.\" in Page 12, \"As can be in seen tables ?? and figure ?? low values of dropout are likely the same distribution\" in Page 12. The paper is very difficult to read for me, partly due to its writing in a language (local codes) that I'm not familiar with. I think that its presentation can be greatly improved for general audience. \n\nI'm not familiar with the concept of \"local codes\", and I do not understand part of the paper. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Syet1-9jtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have a lot of questions about the data used in the experiments. They are created according to the method explained in “Data design” (p.2). It is also summarized in the last paragraph of the first section as follows: ”there are 1/10 input bits that are always 1 for each class and these are the invariant bits, the 0s of each prototype are then filled in with a random mix of 1 and 0 of a known weight”. What is the intention behind this way of creating data? How general are the data created in this way as well as the analyses based on them? It seems to me that the data and thus the analyses lack the generality needed for the purpose of understanding behaviors of neural networks on real tasks/data. \n\nThe same is true for “Neural network design” (p.3), in which 13 experiments conducted in this study are explained. I think their explanations are too condensed; each explanation is very short and it is hard to understand the motivation and purpose of each experiment, i.e., what is the hypothesis to be verified and in what way it is verified? \n\nIn Experiment-12, MNIST is used as data unlike other experiments, and they are modified as “with added 20 pixel invariants”. What is the purpose of this modification? There is a statement in a footnote of p.5 “No LCs were seen in the standard MNIST runs”, which agrees with the above concern about the lack of generality.\n\nAdditionally, I do not understand the statement in p.5 “Increasing the difficulty of the problem (by increasing n_x, …”. Why does the use of more training data make the problem harder? It should usually be the opposite; the smaller, the harder. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByeN3hTwcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Overview: \n\nThis paper aims to study when hidden units provide local codes by analyzing the hidden units of trained fully connected classification networks under various architectures and regularizers.  The main text primarily studies networks trained on a dataset where binary inputs are structured to represent 10 classes with each input containing a subset of elements indicative of the class label.  The work also studies fully connected networks trained on the MNIST dataset (with the addition of some pixels indicating each class label).  After enumerating the number of local codes observed under these different settings, the authors conclude the following: (1) \"common\" properties of deep neural networks & modern datasets seem to decrease the number of local codes (2) specific architectural choices, regularization choices & dataset choices seem to increase the number local codes (i.e. increasing dropout, decreasing dataset size, using sigmoidal activations etc.).   The work then state that these insights may suggest how to train networks to have local codes emerge.  \n\nReview: \nI particularly liked the simple dataset the authors construct in determining whether local codes emerge in hidden units, especially since deep networks and dataset used in practice are overly complex to gain insight for this behavior.  However, I find the overall message to be a bit confusing, especially in regard to using the analysis to construct networks with emergent local codes.  In particular, I feel that the authors could strengthen this work greatly by using their findings to train a deeper neural network for which local codes do emerge on a more realistic dataset.  Furthermore, this work would be significantly more impactful if a network with more local codes does generalize better, but that is unclear as of now (especially since local codes seem to not emerge in practical settings even though these networks are state of the art).  \n\nCriticisms/Questions:\n(1) Main:  I'm somewhat confused about the main takeaway from this work in terms of understanding when local codes actually emerge in deep neural networks.  The authors seem to have a number of very specific conditions that are both architecture and dataset dependent, and overall I feel the message would be much stronger if the authors were able to rigorously study perhaps just a few of these conditions across many more settings.  For example, even just studying the impact of activation and providing some conditions/theory or a clearer understanding of which nonlinearities lead to more local codes would be insightful.  The current work seems to be more broad instead of tackling one of these properties in depth. \n\n(2) I am a bit confused about the thresholds used by the authors in determining whether a hidden unit provides a local code or not. Do you just determine if there is some threshold given by the unit that separates out all points of one class from the rest? \n\n(3) After several experiments, there are some heavy conjectures trying to rationalize the result of the experiment.  As an example, the authors provide statements like \"ReLU is a more powerful activation function than sigmoid.\"  However, this statement in particular is not exactly correct, since given enough width, networks with either activation function should be able to interpolate the training data.  Another example of this is at the bottom of page 7, when the authors provide 5 possible explanations as to why local codes don't emerge in modern training settings.  It is unclear which of these explanations are true, but it would be great if the authors could actually provide a cleaner rationalization.  \n\nMinor criticisms:\n(1) I've seen a number of different conventions for how to refer to the depths of networks, and I believe what you refer to as 3 layer networks would conventionally be referred to as 2-layer networks for theory audiences (as there are 2 weight matrices involved) or 1-hidden layer networks for empirical audiences.  I think adding a figure in the appendix for your architecture would clear up any confusion immediately.  \n(2) Some of the formatting is a bit awry: there are references to figures that appear as ?? (see page 8 paragraph 3).  \n(3) It would be nice to provide a consistent legend in some of the figures.  For example, Figure 4b has no indication for which settings the colors represent.  \n(4) As there seem to be a lot of experiments numbered 1-12, I think it would be much more readable to have different subsections on the different settings and outline the experiments in the subsection more clearly.  Referring back to these numbers on page 3 & 4 constantly makes it less readable.  \n(5) I quite liked Figure 8 in the Appendix.  I feel that this would have been a great figure to put towards the front of the paper to provide an example of local codes emerging.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJeh3af0FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors studied the local codes in neural networks through a set of controlled experiments (by controlling the invariance in the input data, dimensions of the input and the hidden layers, etc.), and identified some common conditions under which local codes are more likely to emerge.\n\nThe fact that local codes tend to emerge as a response to invariance is interesting but not surprising, especially given that convolution operations are designed to capture location invariance. It would be useful if the authors can clarify their contributions and compare against existing works in the literature.\n\nExperiments are conducted at a relatively small scale: On a synthetic dataset with binarized vectors and on MNIST, which a predefined rule for noise injection (Figure 1). The controlled experiments conducted in the paper are still informative, but the overall message would be much stronger if the empirical analysis can be extended to common benchmarks such as CIFAR and/or ImageNet.\n\nAll of the experiments are based on very shallow networks (3-4 layers), and as the result, the study ignores batch normalization and skip connections which are common ingredients in state-of-the-art convolutional networks. It remains unclear whether the presence of those components would change the emergence behavior of local codes, and hence affect some of the conclusions in the paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies when hidden units provide local codes by analyzing the hidden units of trained fully connected classification networks under various architectures and regularizers. The reviewers and the AC believe that the paper in its current form is not ready for acceptance to ICLR-2020. Further work and experiments are needed in order to identify an explanation for the emergence of local codes. This would significantly strengthen the paper.
Decision: Reject