Under what circumstances do local codes emerge in feed-forward neural networks

1 INTRODUCTION . With neural networks ( NNs ) being widely deployed in various tasks it is essential to understand how they work and what data is used to make their decisions . NNs used to be viewed as ‘ black boxes ’ , but recent results ( Nguyen et al. , 2016 ) have started to open that box . NNs came from the field of psychology as simple bio-inspired models , it has been debated whether information is represented in the brain in a distributed manner ( from parallel distributed processing , PDP ) or via a localist coding scheme . Although the distributed approach was the most popular , there are some results in neuroscience ( Quiroga et al. , 2005 ) and psychology ( McClelland & Rumelhart , 1981 ) that are commensurate with a localist coding scheme , including a report of LCs in RNNs ( Bowers et al. , 2014 ) . Recently , there has been an explosion of interest in NNs , especially deepNNs , as these algorithms are now commercially relevant , and this increase in their accuracy has been credited to sources of vastly more labelled data and novel training techniques like dropout ( Srivastava et al. , 2014 ) . Many newer researchers in NNs were perhaps unaware of the distributedlocalist coding debate within psychology , and thus looked for localist-like codes in their NNs , and found indicative ( of LC coding scheme ) evidence of detectors for objects ( Zhou et al. , 2018 ; 2015 ) , concepts ( Karpathy et al. , 2016 ; Lakretz et al. , 2019 ) , features ( Nguyen et al. , 2019 ; Erhan et al. , 2009 ) , textures ( Olah et al. , 2017 ) , single directions ( Morcos et al. , 2018 ) etc. , see ( Bowers , 2017 ) for a review . With faster and larger computers , it is possible , even with the increase in input data size , for deepNNs to ‘ memorise ’ the data-set ( an extreme form of overfitting ) : a process where the NN has simply learned a mapping between input and output vectors , as opposed to learning a rule which will allow it to generalise to unseen data that follows the underlying rule . Generalisation performance is often improved if NN training is stopped early , often when a validation set loss ( val loss ) stops improving , as the NN is prevented from further minimising its loss function by memorising the input ( training ) data . Single directions ( Morcos et al. , 2018 ) have been implicated in memorization of the data-set . Localist codes ( coding for a class A ) are defined as units which are activated at a high ( low ) level for all members ( that the NN gets correct ) and low ( high ) level for all members of the other classes ( class ¬A ) , i.e . the set of activations for class A is disjoint from the activations for class ¬A ( see figure 8 in the appendix ) , and these codes are very strict measure of selectivity . As such , LCs are very easy to interpret , and the presence of them in NNs would make it easy to understand how the NN is working . This paper takes no position on whether or not localist codes exist in the brain or in deep-NNs . instead we take the constructionist science approach of asking when would we expect LCs to appear , and what aspects of the system , data-set and training conditions favour or disfavour their emergence . As NNs are considered ( simplified ) models for the brain , we can also take into account biological plausibility . We hypothesized that LCs should emerge when there was an invariant in the data-set . As deep-NNs take a long time to train , it is hard to get representative statistics , so we look at very simple networks ( shallow : 3-layer and pseudo-deep : 4-layer ) where it is possible to do hundreds of repeats and thus get resilient trends . The main insight of this work is that LCs do emerge when there is an invariant in the data . To set up a system with such a ‘ short-cut ’ we use a simple binary vectors as inputs , which are built from prototypes , such that there are 1/10 input bits that are always 1 for each class and these are the invariant bits , the 0s of each prototype are then filled in with a random mix of 1 and 0 of a known weight , see figure 1 , thus , a given bit is always on for a given class , and maybe on or of for other classes . Note also , that in this set up , if the proportional weight of the prototype exceeds that of the random vector , then vectors belonging to the same class are ‘ closer ’ 1 to each other than those of separate classes , i.e . there is a larger between-class variance than within-class variance . The prototypes are also perturbed to increase the variance of the ‘ invariant ’ bits . If one views a deep convNN as a feature extraction machine ( lower and convolutional layers ) with a feature classification NN on top ( the higher fully connected layers ) , then it is reasonable to suppose that a given class is likely to share features at the top convolutional layer , which would result in the activation vectors at that layer having a higher between group variance than within group , or possibly even invariant features for a class ( perhaps object detectors ) , and so these experiments could give insight into the representation of data in the ‘ fc ’ layers of deep-NNs . 1.1 FINDINGS . 1 . LCs related to lower within-class variance than between-class variance in input 2 . LCs related to a NN internalising a rule 3 . No . of LCs related to difficulty of the problem and the computing power of the NN , with different behaviour for under- and over-resourced NNs . 4 . Large values of dropout increases LCs 5 . LCs correlate with generalisation performance 6 . Large data-sets , softmax and aggressive early stopping reduce the number of LCs 7 . Monitoring the number of LCs can be useful for figuring out when to stop training 2 METHODOLOGY . Data design Data input to a neural network can be understood as a code , { Cx } , with each trained input data vector designated as a codeword , Cx . The size of the code is related to the number of codewords ( i.e . the size of the training set ) , nx . Lx is the length of the codeword , generally 500bits in this paper . We used a binary alphabet , and the number of 1s in a codeword is the weight2 , wx of that codeword . 1There are two metrics that are relevant to measuring the distance between these vectors , the Hamming distance which is the number of bits that have to be switched to turn one vector into another and the cosine similarity , which is the angle between the vectors , we use Hamming distance here . 2This weight definition is not the same as connection weights in the neural network . To create a set of nP classes with a known structural similarity , the procedure in figure 1 was followed . We start with a set of nP prototypes , { Px , 1 ≤ x ≤ nP } , with blocks of 1s of length LP /nP , called prototype blocks , which code for a class . For example , if Lx were 12 and nP were 3 : P1 = [ 111100000000 ] and P2 = [ 000011110000 ] , P3 = [ 000000001111 ] , and this would gives prototypes that are a Hamming distance of 8 apart , and thus we know that our prototypes span inputdata space . To create members of each class , the prototype is used as a mask , with the 0 blocks replaced by blocks from a random vector , Rx . The weight of the random vectors , wR can be tuned to ensure that a set of vectors randomly clustered around the prototype vector are generated , such that members of the same category are closer to each other than those of the other categories ( N.B . the prototypes are not members of the category ) . A more realistic data-set is created by allowing the prototypes to be perturbed so that a percentage of the prototype block is randomly switched to 0s each time a new codeword is created , in accordance with the perturbation rate ( see P ′2 in figure 1 ) . This method creates a code with a known number of invariant bits for codewords in the same category . For example , in figure 1 , codewords C1 and C2 were both derived from P1 , and have a Hamming distance of 6 , where as C1 and C4 are in different classes and have a Hamming distance of 8 . Note , the difference between these numbers is larger in our experiments as Lx = 500 and there are 10 categories . We define ‘ sparseness ’ of a vector , Sx , as the fraction of bits that are ‘ 1 ’ s . LCs were highly unlikely to appear in the input code , and none were observed in random checks . Neural network design We have three-layer feed-forward network with Lx input neurons , nHLN hidden layer neurons ( HLN ) and Lo output neurons , using a sigmoidal activation function and no softmax on the output . For experiment 1 Lx is 500bits , mapped to 10 output classes , so the weight of prototype vectors wp , is 50 , and the wR is 150 ( so Sr=1/3 ) , the output vector is a 50bit-long distributed vector ( wo=25 ) . NNs were trained for 45,000 epochs , and each plotted point is a number of repeats between 10 and 15 . Experiment 2-7 varied thus : 2 : nx = { 250 , 500 } ; 3 : SR = { 1/9 , 2/9 , 1/3 } ; 4 : Lx = { 300 , 700 , 1000 } ; 5 : activation function= { ReLU , sigmoid } ; 6 : output vector is distributed or 1-hot ; 7 : takes nHLN = 500 , 1000 and decays thewP from 50 to 25 ( P from 1 to 0.5 ) . Experiment 8 is ∼250 repeats of experiment 1 , with values of dropout in { 0 , 0.2 , 0.5 , 0.7 , 0.9 } . Experiment 9 is a repeat of 1 , with activation noise added for networks with nHLN = { 100 , 500 , 1000 } . Experiment 10 measures the number of LCs over training time for nHLN = { 250 , 500 , 1000 , 2000 } . To do generalisation tests , a new test set is built with the same parameters as the training set , with ntrain = 10000 and applied to pre-run results ( from experiments 8 and 10 ) . Experiment 11 : for the 4-layer neural networks , nHLN of the first hidden layer is varied , the second is set to 250 , the output vectors are 1-HOT and different training parameters and values are given in table 11 . Experiment 12 : For the MLP experiments , we use MNIST data-set , with added 20 pixel invariants that code for the class which are either non-varying ( ‘ invariant ’ ) or drawn from a Gaussian distribution ( ‘ Gaussian ’ ) , see table 8 in the appendix . The invariant was either not applied ( ‘ standard ’ ) or applied to 50 % of 100 % of all images , or applied to the whole of 2 , 5 or 8 categories , and data is from 10 repeats . Experiment 13 : To see if LCs were associated with memorising the dataset , we trained NNs where the codewords were shuffled before being assigned to targets . NNs were run in Keras with a TensorFlow backend . Accuracy and generalisation . The accuracy reported by Keras counts an output vector as correct if each bit is within 50 % of the correct value , e.g .. the output vector [ 0.4 , 0.6 ] would map to the target vector [ 0 , 1 ] ( the outputs are binary ) , and in standard classification neural networks with a 1- hot target vectors these outputs would be very close to the target after the softmax operation . Thus , we label this accuracy ‘ classification accuracy ’ . If one wants to use a NN as a pattern matching machine , then one could put an arbitrary limit on how big an error between the output values and the targets , we chose 10 % , and the target [ 0 , 1 ] would need an output vector of [ ≤ 0.1 , ≥ 0.9 ] to be considered correct . We call this more stringent condition the ‘ pattern matching accuracy. ’ As the codes are built to a rule , it is possible to generate an arbitrarily large code , thus our test sets are at least 10 times larger than the training sets .

Summary: I have a lot of questions about the data used in the experiments. They are created according to the method explained in “Data design” (p.2). It is also summarized in the last paragraph of the first section as follows: ”there are 1/10 input bits that are always 1 for each class and these are the invariant bits, the 0s of each prototype are then filled in with a random mix of 1 and 0 of a known weight”. What is the intention behind this way of creating data? How general are the data created in this way as well as the analyses based on them? It seems to me that the data and thus the analyses lack the generality needed for the purpose of understanding behaviors of neural networks on real tasks/data.