Year: 2019

Abstract:
Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCs emerging in response to invariant features, and this finding is robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, increasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep networks (2 hidden layers) which have many LCs lose them when common aspects of deep-NN research are applied (large training data, ReLU activations, early stopping on training accuracy and softmax), suggesting that LCs may not be found in deep-NNs. Switching to more biologically feasible constraints (sigmoidal activation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are not found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by a lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise of LCs be required, this work suggests how to tune a NN so they emerge.

heading: 1 INTRODUCTION
text: With neural networks (NNs) being widely deployed in various tasks it is essential to understand how they work and what data is used to make their decisions. NNs used to be viewed as ‘black boxes’, but recent results (Nguyen et al., 2016) have started to open that box. NNs came from the field of psychology as simple bio-inspired models, it has been debated whether information is represented in the brain in a distributed manner (from parallel distributed processing, PDP) or via a localist coding scheme. Although the distributed approach was the most popular, there are some results in neuroscience (Quiroga et al., 2005) and psychology (McClelland & Rumelhart, 1981) that are commensurate with a localist coding scheme, including a report of LCs in RNNs (Bowers et al., 2014). Recently, there has been an explosion of interest in NNs, especially deepNNs, as these algorithms are now commercially relevant, and this increase in their accuracy has been credited to sources of vastly more labelled data and novel training techniques like dropout (Srivastava et al., 2014). Many newer researchers in NNs were perhaps unaware of the distributedlocalist coding debate within psychology, and thus looked for localist-like codes in their NNs, and found indicative (of LC coding scheme) evidence of detectors for objects (Zhou et al., 2018; 2015), concepts (Karpathy et al., 2016; Lakretz et al., 2019), features (Nguyen et al., 2019; Erhan et al., 2009), textures (Olah et al., 2017), single directions (Morcos et al., 2018) etc., see (Bowers, 2017) for a review.
With faster and larger computers, it is possible, even with the increase in input data size, for deepNNs to ‘memorise’ the data-set (an extreme form of overfitting): a process where the NN has simply learned a mapping between input and output vectors, as opposed to learning a rule which will allow it to generalise to unseen data that follows the underlying rule. Generalisation performance is often improved if NN training is stopped early, often when a validation set loss (val loss) stops improving,
as the NN is prevented from further minimising its loss function by memorising the input (training) data. Single directions (Morcos et al., 2018) have been implicated in memorization of the data-set.
Localist codes (coding for a class A) are defined as units which are activated at a high (low) level for all members(that the NN gets correct) and low (high) level for all members of the other classes (class ¬A), i.e. the set of activations for class A is disjoint from the activations for class ¬A (see figure 8 in the appendix), and these codes are very strict measure of selectivity. As such, LCs are very easy to interpret, and the presence of them in NNs would make it easy to understand how the NN is working.
This paper takes no position on whether or not localist codes exist in the brain or in deep-NNs. instead we take the constructionist science approach of asking when would we expect LCs to appear, and what aspects of the system, data-set and training conditions favour or disfavour their emergence. As NNs are considered (simplified) models for the brain, we can also take into account biological plausibility. We hypothesized that LCs should emerge when there was an invariant in the data-set. As deep-NNs take a long time to train, it is hard to get representative statistics, so we look at very simple networks (shallow: 3-layer and pseudo-deep: 4-layer) where it is possible to do hundreds of repeats and thus get resilient trends.
The main insight of this work is that LCs do emerge when there is an invariant in the data. To set up a system with such a ‘short-cut’ we use a simple binary vectors as inputs, which are built from prototypes, such that there are 1/10 input bits that are always 1 for each class and these are the invariant bits, the 0s of each prototype are then filled in with a random mix of 1 and 0 of a known weight, see figure 1, thus, a given bit is always on for a given class, and maybe on or of for other classes. Note also, that in this set up, if the proportional weight of the prototype exceeds that of the random vector, then vectors belonging to the same class are ‘closer’1 to each other than those of separate classes, i.e. there is a larger between-class variance than within-class variance. The prototypes are also perturbed to increase the variance of the ‘invariant’ bits. If one views a deep convNN as a feature extraction machine (lower and convolutional layers) with a feature classification NN on top (the higher fully connected layers), then it is reasonable to suppose that a given class is likely to share features at the top convolutional layer, which would result in the activation vectors at that layer having a higher between group variance than within group, or possibly even invariant features for a class (perhaps object detectors), and so these experiments could give insight into the representation of data in the ‘fc’ layers of deep-NNs.

heading: 1.1 FINDINGS
text: 1. LCs related to lower within-class variance than between-class variance in input
2. LCs related to a NN internalising a rule
3. No. of LCs related to difficulty of the problem and the computing power of the NN, with different behaviour for under- and over-resourced NNs.
4. Large values of dropout increases LCs
5. LCs correlate with generalisation performance
6. Large data-sets, softmax and aggressive early stopping reduce the number of LCs
7. Monitoring the number of LCs can be useful for figuring out when to stop training

heading: 2 METHODOLOGY
text: Data design Data input to a neural network can be understood as a code, {Cx}, with each trained input data vector designated as a codeword, Cx. The size of the code is related to the number of codewords (i.e. the size of the training set), nx. Lx is the length of the codeword, generally 500bits in this paper. We used a binary alphabet, and the number of 1s in a codeword is the weight2, wx of that codeword.
1There are two metrics that are relevant to measuring the distance between these vectors, the Hamming distance which is the number of bits that have to be switched to turn one vector into another and the cosine similarity, which is the angle between the vectors, we use Hamming distance here.
2This weight definition is not the same as connection weights in the neural network.
To create a set of nP classes with a known structural similarity, the procedure in figure 1 was followed. We start with a set of nP prototypes, {Px, 1 ≤ x ≤ nP }, with blocks of 1s of length LP /nP , called prototype blocks, which code for a class. For example, if Lx were 12 and nP were 3: P1 = [111100000000] and P2 = [000011110000], P3 = [000000001111], and this would gives prototypes that are a Hamming distance of 8 apart, and thus we know that our prototypes span inputdata space. To create members of each class, the prototype is used as a mask, with the 0 blocks replaced by blocks from a random vector, Rx. The weight of the random vectors, wR can be tuned to ensure that a set of vectors randomly clustered around the prototype vector are generated, such that members of the same category are closer to each other than those of the other categories (N.B. the prototypes are not members of the category). A more realistic data-set is created by allowing the prototypes to be perturbed so that a percentage of the prototype block is randomly switched to 0s each time a new codeword is created, in accordance with the perturbation rate (see P ′2 in figure 1). This method creates a code with a known number of invariant bits for codewords in the same category. For example, in figure 1, codewords C1 and C2 were both derived from P1, and have a Hamming distance of 6, where as C1 and C4 are in different classes and have a Hamming distance of 8. Note, the difference between these numbers is larger in our experiments as Lx = 500 and there are 10 categories. We define ‘sparseness’ of a vector, Sx, as the fraction of bits that are ‘1’s. LCs were highly unlikely to appear in the input code, and none were observed in random checks.
Neural network design We have three-layer feed-forward network with Lx input neurons, nHLN hidden layer neurons (HLN) and Lo output neurons, using a sigmoidal activation function and no softmax on the output. For experiment 1 Lx is 500bits, mapped to 10 output classes, so the weight of prototype vectors wp, is 50, and the wR is 150 (so Sr=1/3), the output vector is a 50bit-long distributed vector (wo=25). NNs were trained for 45,000 epochs, and each plotted point is a number of repeats between 10 and 15. Experiment 2-7 varied thus: 2: nx = {250, 500}; 3: SR = {1/9, 2/9, 1/3}; 4: Lx = {300, 700, 1000}; 5: activation function={ReLU, sigmoid}; 6: output vector is distributed or 1-hot; 7: takes nHLN = 500, 1000 and decays thewP from 50 to 25 (P from 1 to 0.5). Experiment 8 is ∼250 repeats of experiment 1, with values of dropout in {0, 0.2, 0.5, 0.7, 0.9}. Experiment 9 is a repeat of 1, with activation noise added for networks with nHLN = {100, 500, 1000}. Experiment 10 measures the number of LCs over training time for nHLN = {250, 500, 1000, 2000}. To do generalisation tests, a new test set is built with the same parameters as the training set, with ntrain = 10000 and applied to pre-run results (from experiments 8 and 10). Experiment 11: for the 4-layer neural networks, nHLN of the first hidden layer is varied, the second is set to 250, the output vectors are 1-HOT and different training parameters and values are given in table 11. Experiment
12: For the MLP experiments, we use MNIST data-set, with added 20 pixel invariants that code for the class which are either non-varying (‘invariant’) or drawn from a Gaussian distribution (‘Gaussian’), see table 8 in the appendix. The invariant was either not applied (‘standard’) or applied to 50% of 100% of all images, or applied to the whole of 2, 5 or 8 categories, and data is from 10 repeats. Experiment 13: To see if LCs were associated with memorising the dataset, we trained NNs where the codewords were shuffled before being assigned to targets. NNs were run in Keras with a TensorFlow backend.
Accuracy and generalisation. The accuracy reported by Keras counts an output vector as correct if each bit is within 50% of the correct value, e.g.. the output vector [0.4, 0.6] would map to the target vector [0, 1] (the outputs are binary), and in standard classification neural networks with a 1- hot target vectors these outputs would be very close to the target after the softmax operation. Thus, we label this accuracy ‘classification accuracy’. If one wants to use a NN as a pattern matching machine, then one could put an arbitrary limit on how big an error between the output values and the targets, we chose 10%, and the target [0, 1] would need an output vector of [≤ 0.1, ≥ 0.9] to be considered correct. We call this more stringent condition the ‘pattern matching accuracy.’ As the codes are built to a rule, it is possible to generate an arbitrarily large code, thus our test sets are at least 10 times larger than the training sets.

heading: 3 RESULTS
text: As the chance that all the members of A would emerge disjoint from the members of ¬A is( 50 50 ) / ( 500 50 ) is tiny (4.32 × 10−71), finding a single interpretable local code, such as those shown in figure 8 in the appendix, refutes the idea that neural networks do not have interpretable or locally encoded units.
Local codes are seen in response to an invariant As shown in fig. 2a, the number of local codes is tuned by the size the hidden layer, with a peak in the number of local codes seen at nHLN=1000 for the standard data set, and a peak in the percentage of HLNs which are local codes seen at nHLN = 500 (data not shown): dashed and dot-dashed grey lines are drawn at these points in all relevant figures. The shape of the graph (in fig.2) appears to be an interaction between the difficulty of the problem and the amount of computing power (i.e. the nHLN ) available, with those networks to the left of the peak (‘under-powered’ with respect to LC emergence NNs) not having enough spare units to give over to a localist code, and those to the right (‘over-powered’). Note that, in the results that follow, the data for nHLN is often qualitatively different for over- and under-powered NNs; for example, the number of LCs decreases over time for longer times or the underpowered NNs in experiment 10 and figure 3a and increases for over-power NNs.
Increasing the difficulty of the problem (by increasing nx, see fig.2a, increasing Lx see fig. 9 in appendix, or decreasing the relative information in the prototype by decreasing sparsity of SR, see fig. 2b decreases the number of local codes. The proportionately larger the invariant is in the input code, the more likely LCs are to emerge in the NN, presumably as a ‘rule’ that attends only to the invariant is correspondingly more efficient. In the MLP experiments, the presence of invariant features in pixel space did cause a small number of LCs to emerge3 (see table 9 in the appendix), and these systems trained much faster (these were trained to a given accuracy), see tables 10 in the appendix).
LCs increase with Training time and may relate to generalisation Figure 3a shows that the number of LCs in a NN increases with training time all across the range of nHLN . The corner of the LC-epoch curve (Fig. 3b) was roughly correlated with the system achieved 100% classification accuracy (‘easy’ criteria) on the training data and the asymptote on the test data (around 95%), see figure 3c. Interestingly, this point is also when the pattern-matching accuracy (‘hard’ criteria) on both the train and test data-sets rises above 0 (fig.3d, suggesting that the additional LCs emerging are caused by the separation of 1s and 0s in the NN output. Thus, monitoring the rate of LC increase can give a measure of when to stop training.
To see if LCs were related to memorisation, we investigated the experiment with the greatest range in LCs-the 90% dropout shown in figure 7b-and the training and test accuracy is plotted in fig. 4a. There were many solutions that had an accuracy > 99%, but where the solutions were lower accuracy, the accuracy on both train and test data was related to the number of LCs. Note that, as the test set was many times larger than the training set (nx= 10,000 for test, nx = 500 for training) this shows that the NNs have clearly internalised the rule, the rough positive correlation between the number of LCs and the accuracy (in the accuracy range of 20-100%) demonstrates that LCs are part of the internalisation of this rule. Similarly, plotting the test accuracy against the number of LCs for experiment 10 (fig.4b) shows a positive correlation between the no. of LCs and classification accuracy (similar results are seen with the pattern-matching accuracy, see figure 12 in the appendix). Shuffling the input code to force the NN to memorise the data resulted in a reduction in the number of LCs, but only by a maximum 47% (see fig. 13 in the appendix).
Why might LCs not be seen in modern deep NNs? Figure 5 shows the effect of changing the system setup and output vectors. There is little difference between ReLU and sigmoid, except that the peak of the curve is shifted a little to the left in the ReLU data, likely as ReLU is a more ‘powerful’4 activation function than sigmoid. An alternative explanation is that as ReLUs train faster and the number of local codes increases with training time (see fig.3a), the ReLU networks are
3No LCs were seen in the standard MNIST runs. 4i.e. its range is not limited at the positive end so can use a larger number space than sigmoid
●● ●● ●● ●● ● ●● ● ●●● ● ● ●● ●●●
●
● ●● ●● ●● ●● ●
●
● ●●●● ●●● ●●
●
●● ● ●●
●
●
●
●
● ●
●● ●
●● ● ●● ●
●
● ●
●● ●
● ●●
● ● ● ● ●
●●
● ●● ●● ●
●
●● ●●
●●
●● ●
●
●
● ● ● ●● ● ● ●
●●
●
● ●
●
● ● ● ● ● ● ●
●
●●
● ● ●
●
●
●
●
● ●
●
● ●● ●●●● ●
●
●●
● ●
●
●
●
●
● ● ●●
●
● ● ●● ● ● ●● ● ●● ● ●● ●● ●●● ● ● ● ●●● ● ●● ●
●
●
● ●
● ● ●● ● ■■ ■■■ ■■ ■ ■■ ■ ■■■ ■■ ■ ■■■
■
■■ ■ ■■ ■ ■ ■ ■■■■ ■■ ■■ ■ ■■■ ■■ ■
■
■
■
■ ■
■■ ■ ■■ ■■ ■ ■ ■ ■ ■
■ ■ ■■ ■ ■■■ ■■■■ ■ ■■
■
■ ■ ■■ ■ ■ ■ ■ ■ ■■ ■■ ■■ ■
■
■
■
■
■
■
■ ■ ■■ ■■ ■■ ■■ ■ ■ ■ ■ ■ ■ ■
■
■ ■■ ■ ■ ■■ ■ ■■ ■■■■ ■■■■ ■■ ■
■
■
■■■
● Test ■ Train
� �� �� �� �� ��� ��� ���
��
��
��
��
��� ����
(a) Relation between classification accuracy on test and train for the 90% dropout experiment (8).
●●●●●● ●●●●●●●●●●● ● ●● ● ●●●●●● ●● ●●●● ●● ●●● ●●●● ● ● ● ● ●● ●● ●● ● ●● ●●●● ● ●●● ●●●● ●● ● ● ●● ●● ●●●●●● ●●● ●●● ●●●● ●● ●●● ●●● ● ● ●● ●●●● ●●●●●●● ●●● ●●● ● ●●●●■■
■■■■■ ■■■ ■■■■■ ■■■■■■ ■■■■ ■ ■■ ■ ■■ ■■■ ■ ■■ ■ ■ ■ ■■ ■ ■■ ■ ■■ ■■ ■ ■■ ■■■■ ■■ ■■■■ ■ ■■ ■ ■■ ■ ■ ■■ ■ ■■ ■■■ ■ ■ ■■ ■ ■ ■■ ■ ■ ■■ ■ ■■ ■■ ■ ■■■ ■■■ ■ ■ ■ ■■■■■ ■■ ■■ ■ ■ ■ ■ ■■■ ■ ■ ■ ■■■■■ ■ ■■ ■■ ■■■■ ■■ ■ ■ ■ ■ ■■■■■ ■■ ■■■■ ■■■ ■ ■■ ■■■■ ■■ ■■ ■ ■ ■■ ■ ◆◆ ◆◆◆◆◆◆◆ ◆◆◆ ◆◆◆◆ ◆◆◆ ◆◆◆◆ ◆◆ ◆◆ ◆ ◆◆
◆ ◆ ◆◆◆ ◆ ◆ ◆ ◆ ◆◆◆ ◆ ◆
◆ ◆
◆ ◆ ◆
◆
◆◆◆ ◆ ◆ ◆◆ ◆◆ ◆◆◆◆ ◆◆ ◆◆◆ ◆ ◆ ◆◆
◆ ◆◆ ◆ ◆ ◆ ◆◆ ◆◆ ◆ ◆ ◆◆◆ ◆ ◆ ◆◆ ◆◆ ◆◆◆ ◆ ◆ ◆ ◆◆ ◆ ◆
◆ ◆ ◆
◆ ◆ ◆◆
◆ ◆◆ ◆◆ ◆ ◆ ◆ ◆ ◆◆ ◆ ◆ ◆◆ ◆◆◆◆ ◆ ◆ ◆◆◆ ◆◆ ◆◆ ◆ ◆◆◆ ◆◆◆◆ ◆ ◆◆ ◆◆ ◆ ◆◆◆ ◆ ◆ ◆◆ ◆ ◆◆ ◆ ◆◆◆◆ ◆ ◆◆◆ ◆◆ ◆◆◆ ◆◆ ◆◆ ◆◆ ◆ ◆◆ ◆ ◆◆ ◆◆
▲▲▲ ▲▲▲▲▲▲▲▲ ▲ ▲▲▲▲▲ ▲▲▲▲▲ ▲ ▲▲▲▲▲ ▲▲ ▲ ▲▲ ▲▲▲ ▲ ▲
▲▲ ▲
▲
▲
▲ ▲ ▲ ▲▲ ▲ ▲
▲
▲ ▲
▲ ▲ ▲ ▲▲▲ ▲
▲ ▲ ▲ ▲
▲ ▲ ▲ ▲▲ ▲
▲ ▲ ▲ ▲▲
▲▲
▲▲
▲
▲▲ ▲
▲
▲ ▲
▲ ▲ ▲ ▲▲
▲ ▲ ▲▲
▲
▲ ▲▲ ▲▲
▲ ▲ ▲▲ ▲ ▲▲▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲▲ ▲ ▲ ▲ ▲
▲
▲
▲▲
▲ ▲▲ ▲▲ ▲
▲ ▲▲
▲ ▲▲
▲ ▲▲▲ ▲ ▲▲ ▲▲
▲
▲
▲
▲▲▲ ▲ ▲ ▲ ▲▲▲▲ ▲ ▲▲
▲ ▲▲
▲ ▲
▲▲ ▲ ▲ ▲
▲
▲ ▲ ▲▲ ▲▲ ▲ ▲ ▲ ▲▲ ▲ ▲
▲ ▲ ▲▲ ▲ ▲ ▲ ▲ ▲▲ ▲▲ ▲▲ ▲▲ ▲▲ ▲ ▲▲ ▲ ▲▲ ▲ �� �� �� �� ������ �� ��� ��� ��� ��� ��� ��� ��� �� ���
(b) Relationship between the number of LCs and classification accuracy on test from experiment 10.
Figure 4: The number of local codes increases with improvement on generalisation accuracy. Fig.4a is over 250 repeats trained for the same amount of time, Fig.4b are repeated measurements on the same NNs as they are trained.
effectively trained for longer compared to sigmoidal networks. We chose to use a distributed output code in the 3-layer networks as we thought the local codes at the output layer might encourage the emergence of LCs in the hidden layer: the opposite happened, likely because a simple 3-layer network set up with prototypes like this, the problem of mapping input to output is linearly separable and much easier, hence LCs are not needed. Note that, in deeper 4-layer HLNs with 1-hot output vectors there are many local codes (see ‘B’ in figure 6b).
●
● ●
●
●
● ● ● ● ● ●●
● ● ● ●●●
● ●
● ● ● ●■■■■
■ ■■ ■
■ ■ ■ ■ ■ ■
■ ■ ■ ■
● ReLU ■ Sigmoid
���� ���� ���� ���� ���� ���� ���������
��
��
��
�� ��� �����
(a) Experiment 5. Effect of changing the activation function.
● ●●●● ●●●●●●●●●●●●● ● ● ● ● ● ●■■■ ■ ■ ■■ ■
■ ■
■ ■ ■
■
■
■ ■ ■
● 1-Hot ■ Distributed ���� ���� ���� ���� ���� ���� ��������� � �� �� �� �� ��
��� �����
(b) Experiment 6. Effect of changing the output form.
Figure 5: Network architecture parameters can drastically effect the emergence of local codes. 5a: Switching from HLNs with a sigmoidal activation function to a rectified linear (ReLU) units; 5b: Switching from a distributed to a 1-hot output encoding.
To test deep-NNs, we compared 4-layer NNs with our standard set-ups (A, B) that exhibited LCs with similar architecture and standard CNN training techniques added (e.g. L, O-S). NNs with all the settings commonly found in deep-NN literature do not produce LCs. Figure 6a shows that there is little difference between stopping on val loss or after 60K epochs (as does the overlap between bars A and B in fig. 6b. L, M, O-S had ReLU and large data-sets, which completely inhibited local codes. H to S used ReLU rather than sigmoid, so in these systems it seems that ReLU-based NNs do not learn local codes, although they did in our earlier experiments (fig. 5). That C and D are so similar shows no real effect of dropout in these NNs. That C<A, D<B, F<E (and fig. 6a) shows that softmax reduces the number of local codes slightly. We propose this is because softmax does some extra ‘processing’ on the final layer, shifting 50% accurate output vectors up to 90%, and this extra processing makes the problem easier for the hidden layers (so the ‘rule’ is less useful). Alternatively, due to this extra ‘processing’, using softmax reduces the time required to train the network, which is expected to reduce the number of LCs, see figure 3a, (N.B. with early stopping conditions training time now varied). Thus, the reason that LCs are not found in deep-NNs could be the use of large training data sets, early stopping, ReLU and softmax in those systems, or alternatively it could be because there are no feature invariants associated with output classes at the top convolutional layer (i.e. the layer before the fully connected classification layers).

heading: 3.1 BIOLOGICAL CONSTRAINTS ENCOURAGE THE EMERGENCE OF LCS
text: As the positive activation range of the ReLU is unlimited, it is less biologically plausible than the sigmoid activation function (although these NNs are still very far from being a model of biology). In this section we investigate noise on sigmoidal, non-softmax NNs. As real biological systems are (possibly) noisy5, we looked a several different ways of introducing noise into the system: decaying the invariant, dropping out units and adding in activation noise.
Perturbing the prototype block is shown in figure 7a, and LCs are still seen if the invariant is not always present, but simply more likely to be present for a given class. However, the number of LCs disappears when the decay is 0.4, i.e. weight of the (50-bit long) prototype section is 20bits, and this is roughly equal to the weight of a 50bit long section of the random vector (16.67bits), as at this point the signal is indistinguishable from the noise. The curves are not well fit by an exponential, the fits shown here are: (for nHLN = 500) 20.62 - 30.65 √ P , withR2 = 0.984; (for nHLN = 1000)
19.72 − 32.29 √ x with R2 = 0.977, and these data were fit up to P=0.4. Comparing to the MLP case, there were no LCs if the invariant feature was perturbed, see table 9.
Contrary to our expectations, adding dropout and increasing it to very large values increases both the average and maximum number of LCs in solutions found by the neural network, although lower values of dropout seemed to overlap, with two competing possible underlying distributions seen perhaps between 0 and 40% (see fig. 10 in appendix). Summary statistics and Kolmogorov-Smirnov hypothesis tests are reported in tables ?? in the appendix, and above 50% dropout there is a significant difference in the underlying distributions with an increase in number of LCs. Repeated copies of local codes may be more resilient to noise than a distributed code: a fully distributed code will always be incomplete but a local code only has a chance of being not present under dropout.
Adding in activation noise changes the number of LCs that are found (see fig. 11 in the appendix), increasing them in overpowered NNs and decreasing them in underpowered networks.

heading: 4 DISCUSSION
text: The number of local codes is not large, usually between 0-10% of the hidden layer, but this is not insignificant and the fact that they emerge in response to invariance in the data-set and some training conditions suggest that they have a function (presumably detecting the presence of an invariant which can be used as a ‘short-cut’). These results are resilient to a small perturbation of the invariant (in the NNs), the invariant feature does not need to be always present, merely more likely
5It is commonly suggested that living neural networks are noisy and thus that neurons and synapses must be resilient to noise, however there is some debate as to whether this is true and some neurons are highly un-noisy.
to be present for a specific class. LCs emerge in correlation with generalisation performance, so monitoring the rate of change of LC numbers can give feedback on when to stop training to avoid overfitting. As LCs were found in response to an added invariants in 2D graphical data in MLPs, we expect that the use of LCs to identify and code for found ‘rules’ based on invariant features will be a general phenomenon. Morcos et al. (2018) found that ‘single direction’ codes were associated with the harder problem of memorising the dataset (as compared to learning the simpler underlying rule), here we found that LCs emerged in response to learning a simpler rule, and were inhibited by switching to a memorisation task. Further work is needed to tease out which aspects of the difference in architecture and problem choice between Morcos et al. (2018)’s study and ours is the cause.
It’s an open question whether there are localist codes in deep-CNNs, PDP dogma suggests not, but there have been some highly selective units found in other types of NNs. Our results suggest that there might not be LCs in the fully connected layers if typical current deep-CNN training methods are used. Furthermore, there is the question of whether there are invariant features coming from the conv/pooling layers into the fully connected feed-forward layers (of a typical current deep-CNN) which is not currently known and an area of active research for us.
As LCs were seen when the NN has slightly more biologically plausible training conditions like a noisy system, it might be possible that LCs might emerge in the human brain, although even the more biologically plausible NNs are perhaps too far removed from actual biology.

heading: A APPENDIX

heading: A.1 LOCAL CODE EXAMPLES
text: Jitterplots are common analysis techniques in neuroscience, but can also be applied to neural networksBerkeley et al. (1995) by plotting the activation of a unit on the x axis and randomly jittering the y-axis value. Examples of local codes found in our networks are given in figure 8, these were sigmoidal activation units. As it takes more energy to spike than not spike, in biological neurons the localist codes are generally ‘on’ codes where the unit is on more strongly in the presence of the class it codes for than not, see figure 8left. In our system there is no energy penalty for using high activations (in fact, energetic considerations are not modelled at all, and this is one of the reasons NNs are not biologically plausible) so we also see ‘off’ codes: see figure 8right.
A.2 EXPERIMENT 4: CHANGING INPUT VECTOR LENGTH (Lx)
As shown in figure 9, there is little difference between different lengths of vectors for smaller lengths, although the number of LCs is decreased at Lx = 1000.

heading: A.3 FURTHER RESULTS FROM EXPERIMENT 8: DROPOUT
text: Dropout (see Srivastava et al. (2014)) is a common training technique where a percentage of a layer’s neurons are ‘dropped out’ of the network (their connections are set to zero) during training to prevent over-fitting, however, dropout can also be viewed as a type of training noise. Results are shown in figure ?? and table 2. There are always solutions with close to zero local codes, but the expected (mean) and maximum number roughly increases with an increasing percentage of dropped out neurons.
Generally, dropout percentages in the range of 20-50% are used in training, and these probability distribution functions (PDFs), like 0% peak, are also joint peaks, with the 20% data having more of the lower solution and the 50% having more of the higher one. Dropping out more than 50% of the network is not generally used as it slows down training. However, with these higher values, the range of solutions is much higher (as evidence by a higher variance and range of the number of local codes), which is expected as dropout forces the network to adopt a range of solution sub-networks, the increase in local codes suggest that localised encoding offers some protection against noise.
At first glance, this might seem unlikely, as distributed patterns are claimed to be more resilient against failure. However, say we had a 20% dropout rate, a fully distributed encoding, would be affected by dropout 100% of the time, losing 20% of its information, whereas a localised encoding would be unaffected 80% of the time (although 20% of the time it would lose all data), and further resilience can be provided if duplicate local codes were used for the same class. Note that, as only 10 classes were used, the large number of local codes, especially for the high dropout values, suggests there are multiple LCs for each category. These results suggest that, for a noisy network, solutions involving some duplicate localised codes are useful methods for dealing with uncertainty.
To test if the different datasets could have come from the same distribution we did KolmogorovSmirnov hypothesis tests with the hypothesis being that the data were from the same distribution. As can be in seen tables ?? and figure ?? low values of dropout are likely the same distribution, but the distributions are different for the high values of dropout. The expanded set of experiments is shown in figure 10, and there seems to be two underlying distributions between 0 and 50%, with the 10%, 20% 30% dropout NNS having more of the distribution with less LCs, and 0%, 40% and 50% having more of the distrubtion with more LCs.

heading: A.4 RESULTS FROM EXPERIMENT 9: ACTIVATION NOISE
text: We added in Gaussian distributed activation noise around a mean of 0.5 to all HLNs (which were sigmoidal, so the activation varied between 0 and 1), the results for three networks are shown in figure 11. The addition of noise changes the distribution LCs seen. For the overpowered NNs, those above the nHLN peak in experiment 1, the number of local codes increases significantly in most cases. For the underpowered NNs (nHLN = {100, 500}) the number of local codes decreases, and this is not significant for tightly distributed noise.

heading: A.5 FURTHER DATA FOR EXPERIMENT 12: MULTI-LAYER PERCEPTIONS.
text: Examples of the modified MNIST input data are given in 8. In the invariant case a block of 20 (2.5%) hot pixels (fully on) are positioned around the edge of the image, thus a NN could learn to attend to the hot pixels in order to correctly assign the category instead of learning to identify the letter shapes. In the Gaussian case the prototype code perturbed by having the pixel value drawn from a different Gaussian distribution for each image (this is the equivalent of the perturbation of prototypes in experiment 7. The networks were trained with no regularisation and no validation set.
The results for the MLP experiment (11) is given in table 9. There is a small, but significant emergence of LCs in response to categories with added invariants, but it has to be for all images in a category for the NN to develop a local code to look for the invariant. In the Gaussian examples, no local codes emerge. Given that the invariants were less <2.5% of the input data, and the result that perturbation removes local codes above a certain level, we suspect that were the prototypes perturbed less or bigger, LCs might have been seen with the perturbed invariants.
Table 10 shows the impact on training times of the presence of a reliable code, which the NN can learn instead of learning to identify the number shapes. The NN trains much faster as the problem it is solving is a simpler one. Thus, the presence of an invariant adds a ‘short-cut’ to the task which the NN can find.

heading: A.6 FURTHER GENERALISATION RESULTS FOR EXPERIMENT 10: TRAINING TIME
text: Figure 12 shows the correlation between the number of local codes and classification accuracy and pattern matching accuracy on the test data. For NNs with HLNs to the left of the peak (i.e. nHLN < 1000) show a slight negative correlation, those at the peak or above show a slight positive correlation, although there is lots of spread in this data.

heading: A.7 RESULTS FOR EXPERIMENT 13: SHUFFLING THE INPUT DATA
text: We tried shuffling the input data which should force the NN to memorise the dataset, rather than relying on the short-cut. We expected the number of local codes to drop to zero, but as shown in figure 13 this did not happen, instead the number of local codes was only reduced.

heading: ACKNOWLEDGMENTS
text: An. Author would like acknowledge funding from Leverhulme on grant no.

Review summaries:
I have a lot of questions about the data used in the experiments. They are created according to the method explained in “Data design” (p.2). It is also summarized in the last paragraph of the first section as follows: ”there are 1/10 input bits that are always 1 for each class and these are the invariant bits, the 0s of each prototype are then filled in with a random mix of 1 and 0 of a known weight”. What is the intention behind this way of creating data? How general are the data created in this way as well as the analyses based on them? It seems to me that the data and thus the analyses lack the generality needed for the purpose of understanding behaviors of neural networks on real tasks/data. 

This paper aims to study when hidden units provide local codes by analyzing the hidden units of trained fully connected classification networks under various architectures and regularizers.  The main text primarily studies networks trained on a dataset where binary inputs are structured to represent 10 classes with each input containing a subset of elements indicative of the class label.  The work also studies fully connected networks trained on the MNIST dataset (with the addition of some pixels indicating each class label).  After enumerating the number of local codes observed under these different settings, the authors conclude the following: (1) "common" properties of deep neural networks & modern datasets seem to decrease the number of local codes (2) specific architectural choices, regularization choices & dataset choices seem to increase the number local codes (i.e. increasing dropout, decreasing dataset size, using sigmoidal activations etc.).   The work then state that these insights may suggest how to train networks to have local codes emerge.