Unsupervised Dynamic Routing Via Competition Over Network Loss

This paper proposes a novel neural network architecture that can simultaneously1 do normal network optimizing while attaining the ability of unsupervised learning.2 Almost all existing unsupervised learning algorithms are based on doing calcu-3 lations on the input space or feature space , this paper proposes a new possibility4 to discover a structure in the functional space without supervision . Using the5 self-organizing map over the competition of the loss of individual neural column,6 we route the input to the most appropriate modules dynamically , by doing this we7 separate the input functional space into different sub spaces which are represented8 by each individual neural column . At the end of the paper , we propose several9 possible architectures based on the philosophy of this paper that could build a10 neural network system block by block.11 1 Introduction12 The unsupervised learning ability of the natural structured biological neural network is always13 fascinating scientists . The structure of some primitive neural networks is generated from evolution14 [ 1 ] , while other superior networks have a base neural substrate that could adapt to the input neural15 signal and generate a suitable structure automatically [ 2 ] [ 3 ] [ 4 ] . Even computer simulation of this16 process gives the same result . By simulating the evolution process by genetic algorithm , a scientist17 could evolve the same network structure for the insect ’ s path integration [ 5 ] . On the same subject , one18 could also use a general purpose recurrent neural network ( RNN ) to optimize the path integration19 problem and still get the same network structure by interpreting the network weights [ 6 ] [ 7 ] . This20 method still works for more complicated problems , some scientists could replicate the neuron21 activation pattern for the grid cell in the hippocampus of the rat , which is used for rat ’ s navigation22 and path integration [ 8 ] [ 9 ] [ 10 ] . Although different in optimizing algorithm and network structure,23 RNN and hippocampus could get the same activation pattern eventually on the subject of navigation24 which means that to develop an artificial network structure to mimic the biological plausible structure25 is not impossible.26 Information has its own structure and the structure is important . Convolutional neural network ( CNN ) 27 works best in the domain of image information [ 11 ] while natural language sequence information28 works best under the structure of recurrent neural network ( RNN ) or deep self attention network29 ( Transformer ) [ 12 ] . Image information has a structural match with the structure of CNN , so even30 without training the structure of CNN could give us a relatively high accuracy in classification31 task which is better than random guessing [ 13 ] . So the hierarchical structure of image information32 and the hierarchical network structure of CNN is having a resonance . On the other hand , natural33 language sequence doesn ’ t have such a strict information structure , the `` pixels '' of the sequence has a34 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . correlation of each other some random distance away , so the self attention algorithm works best under35 this scenario [ 12 ] . Can any neural network learn such information structure automatically that it can36 adapt to any kind of information universally ? The biological neural network of vertebrate animals37 seems achieved this goal already by evolved a columnar structure [ 3 ] , so we continue to adopt this38 philosophy on the design of neural network architecture.39 Contribution We proposed a novel network architecture called functional self organizing map40 ( fSOM ) for making connection between network layers and also a method to discover the connection41 target automatically by dynamic routing among the neural columns between layers by mutual42 competition between network columns.43 2 Related Work44 Capsule networks For the area of columnar networks that can do dynamic routing , some method45 has been proposed [ 14 ] [ 16 ] . Hinton first introduced the concept of capsules which is a multi-layer46 columnar network which are trained to generated images with certain transformations . However in47 their method , if the feature size is large then the mutual connection matrix will get large quickly48 which will make the optimization slow . Actually if considering employing regularization to enhance49 the generalization of the network , most of the number in the feature vector is zero with only a small50 portion of the vector non zero after activation function [ 15 ] . As seen in Fig1 in appendix we can51 partition the feature vector into columns and only transfer this non-zero column to the next layer.52 And also , their method is based on a local routing rule of mutual agreement which although to some53 extend have a biological plausibility , doesn ’ t take into account the mutual competition of the columns54 involved so the network lacks of a global interpretability because the columns it generated don ’ t have55 a topological relationship.56 3 Functional Self Organizing Map57 3.1 Introduction to self organizing map58 Here we give an introduction to the unsupervised learning algorithm of the self organizing map59 ( SOM ) which is selected as the mutual competition algorithm of our method [ 18 ] . Each data from60 our dataset is a vector Xm ∈ Rm of dimension m. The SOM is composed of a group of vectors61 Wmi ( also called the weight of the SOM ) which have the same dimension m that is distributed on62 a two dimensional map . The index i of Wmi is the position of the weight vectors as seen in Fig 1.63 The input vector is then compared with each of the vectors on the map , the one with the minimum64 distance d from the input vector will be selected for update , which is called the best matching unit65 ( BMU ) :66 di = 1 m √∑ m ( Xm −Wmi ) 2 ( 1 ) 67 BMUm = Wmargmini ( di ) ( 2 ) where di is called distance function , BMUm is the best matching unit vector . To update the weight,68 we need to make the vector of the BMU closer to the input vector according to hebbian learning69 rule [ 19 ] . We also need to update the vectors that are close to BMU , otherwise , the same BMU will70 always be selected no matter what is input . To do this , we need to define a neighborhood function of71 the BMU and assign different rates when updating the weight.72 hi = α exp ( − ‖ri − rBMU‖2 2σ2 ) ( 3 ) Where hi is called the neighborhood function , ri ∈ R2 and rBMU ∈ R2 are the position coordinates on the SOM , σ is the radius of the neighborhood function . The weights of the SOM will be updated according to the following rule . Wm′i = W m i + hi ( X m −Wmi ) This means we can scatter the input representation space on a map Xm ⇒Wmi . We ’ ll show how73 to regress some output space to some input space G ( Xm ) = Yn by scattering this functional space74 topologically on a map ( Xm , Yn ) ⇒ Gmn as we did on SOM.75 3.2 Functional Self Organizing Map76 The main idea of the SOM is competition . By competing over every other node , we get a winning node and update the node on that position . Adopting the same idea , instead of using vectors , we use neural network columns to compose the SOM 2 . Each column is a two layer encoder-decoder unit . For an input vector Xm , each of the columns will give an output vector as follows : Yni = ∑ ml XmEmli D ln i Where Xm is the input vector , Emli is the encoder matrix of column i , D ln i is the decoder matrix and77 Yni is the output vector of dimension n. Note no activation function and weight regularization are78 needed as explained in Fig1 of appendix . Now we can define the loss function of each column to be:79 Li = 1 n ∑ n |Tn −Yni | ( 4 ) BMU index = argmin i ( Li ) where Tn is the target vector . Then the node with minimum loss is selected as the BMU and has the priority to fully back propagate the error . Other error back propagation is penalized by the value of the neighborhood function by limiting the loss value associated with that node , so the overall gradient needs to propagate back is : ∇θ ∑ i hiLi hi = exp ( − ‖ri − rBMU‖2 2σ2 ) where θ is the parameters of all the columns . Note that not all columns can propagate gradient back , only those within some certain radius of the BMU have the opportunity as seen in Fig 2 . At this stage , there ’ s no mutual connection between columns , so different columns will be optimized to different directions although the loss function could be the same . To make columns more coherent with their surrounding , we need to add a coherence update to the parameters of the columns . Now define the optimized parameters of the network to be Êml and D̂ln , we can make coherence update like the following : Eml′i = E ml i + αhi ( Ê ml −Emli ) Dln′i = D ln i + αhi ( D̂ ln −Dlni ) Input Vector Encoder Columns Decoder Columns Output Vectors Loss LosingLosingLosingLosingLosing Wining Gradient Back Propagation Weight Coherence Update Auxilliary Classifier Inference Figure 2 : Functional Self Organizing Map , please see section 3.2 for detail where α is the coherence update rate . This means , we are going to have a topologically functional80 space composed of each column , and also we dynamically routed the input to output by doing a81 winner-take-all competition . Note that during optimization , not all the columns are optimized by82 back propagation , only those surrounding BMU are . So to make an inference on the network we need83 to record the chosen BMU location and train an auxiliary classifier as seen in Fig 2 , then by using84 this classifier we can select the correct column to output . By doing this , the scope of the calculation85 is only limited to the classifier and the selected column during inference.86 3.3 Multi-layered Functional Self Organizing Map87 For multi-layered networks , we need a dynamic routing algorithm between layers . In other works [ 14 ] columns between different layers are fully connected which poses a problem such that we need every loss of the columns to find the BMU and do the optimization . A multi-layered network has a combinatorial number of losses to calculate and the number of losses will grow exponentially large . Here we adopt a simplification strategy by expanding the fully connected layers to a hierarchical cascade network . Fig 3 shows a two layered functional SOM network . The output is calculated by columns in a different layer as following : Yni = ∑ mlk XmEml ĵ Flki D kn i Where ĵ is the selected index of the first layer . The calculations to find the distance function and BMU is the same as in Equation 1 2 . However for there are two layers , we need two neighborhood functions : hi = exp ( − ‖ri − rBMUi ‖ 2 2σ21 ) hj = exp ( − ‖rj − rBMUj ‖ 2 2σ22 ) Where σ1σ2 are the neighborhood radius of each layer , rBMUi , r BMU j are the position coordinates of88 the BMU . To make gradient back propagation work as in the single layer case , we need to apply the89 neighborhood function to the loss . Different layers have different map sizes , so the neighborhood90 function of each layer has a different granular scale . In Fig 3 , each top layer node connects to three91 bottom layer nodes so the neighborhood function needs to triple the size to match the bottom layer92 with the scale-up function upsize3 ( ) , the resulting neighborhood function is the sum of the two:93 h∗i = exp ( − ‖ri − rBMUi ‖ 2 2σ21 ) + βupsize3 ( exp ( − ‖rj − rBMUj ‖ 2 2σ22 ) ) ( 5 ) where β controls the relative ratio of the top layer neighborhood function . And the gradient back propagation is the same : ∇θ ∑ i h∗iLi As in the single layer case , we still need a coherence update to all the column parameters:94 Eml′j = E ml j + αhj ( Ê ml −Emlj ) Flk′i = F lk i + αhi ( F̂ lk − Flki ) Dkn′i = D kn i + αhi ( D̂ kn −Dkni ) After the optimization , we are going to have a coarse functional map for the top layer and a finer map95 belongs to each top node for the bottom layer . We ’ ll show the generated map in the next section.96 4 Experiments97 All the experiments are conducted on one GPU machine of 2080Ti . The code will be published in98 Github online repository1.99 4.1 One Layer Functional Self Organizing Map100 We first test our method on the one-layer case explained in Section 3.2 . The dataset we used is the101 handwritten digit dataset mnist [ 20 ] . The training target Tn in Equation 4 is simply the same as input102 vector Xm which makes the optimization into an auto-encoder regression . All hyperparameters are103 summarized in the following table.104 105 We terminated the optimization after the decoder weight is stabilized and generated the figure of106 decoder weights for visualization . Please see Figure 4 for detail.107 4.2 Two Layer Functional Self Organizing Map108 Secondly , we test our multi-layer method on the same dataset of mnist . The training target is again an109 auto-encoder regression.110 111 We generated the figure of the top encoder and decoder weights , the bottom encoder is not visually112 interpretable , so we just show one sample of it . We also generated the overall neighborhood function113 defined in Equation 5 . We then test the BMU indices of the top layer against the true label to check114 1https : //github.com/threesond/fSOM the unsupervised classification ability . We can see that the decoder map generated is partitioned into115 16 blocks which correspond to the size of the top layer map . Each block is again having 16 columns116 corresponding to each top layer column 5.117 Limitation As we can see in Fig 8 , it ’ s possible that some input handwritten numbers are routed to118 the wrong columns . This problem could be solved by adding more layers , increasing the size of the119 map or casting the network into a CNN structure in future research.120 5 Discussion121 Bus structure and bandwidth saving The location of the BMU of each layer makes the classifi-122 cation while the vector of the BMU column passed to the next layer attains the details within this123 class which means that the information not belonging to the current class has been got rid of . Unlike124 traditional deep neural networks whose feature vector dimension always gets bigger when the network125 get deeper , in our architecture , the feature vector will always gets smaller with the accompanying126 BMU classification.127 The auxiliary classifier of each layer determines which column has the priority to output to the128 next layer which resembles a bus structure in the computer system . This point of view gave us an129 inspiration that the methodology of computer system design could also apply to neural network130 system design at least to some extend.131 Winner take all competition is too strict , we need a finer competition algorithm One limitation132 of our work is that in the method section all the competition strategies we used are winner take all133 ( WTA ) which means only one node is selected as the BMU while all others lose . WTA is a special134 case of the mutually inhibitory neural algorithm . Normally after mutual inhibition , there will be135 multiple BMUs [ 21 ] [ 22 ] which means we can build a more complicated structure in the future.136 Neural columns can latch into each other just like the flip-flop circuits by mutual inhibition and137 lateral excitation [ 23 ] [ 24 ] . With neural columns as the functional unit , mutually latched columns by138 some certain trace can be recalled as a functional group that optimized for some function domain139 while with another trace we can recall another functional group for another function domain . This140 means it ’ s possible for us to develop a universal architecture for the human-like neural network141 machine.142 6 Conclusion143 A novel neural network architecture of functional self organizing map is presented in this paper . By144 introducing competition and coherence update to a columnar network we can route different columns145 to each other dynamically . We conducted experiments on a handwritten digits dataset and verified146 that our method inherited the self organizing ability of the self organizing map , especially on two147 layered case , we can generate a hierarchically topologically related map which have a unsupervised148 classification ability . This result suggest that we successfully mapped the functional space of the149 auto-encoder to a topological space . As for future work , it is interesting to adopt this method to a150 convolutional neural network or other fine grained model for better regression ability . Finally , we151 hope this novel architecture provide a new substrate for neural network research which could bring152 us more advanced artificial intelligence.153 References154 [ 1 ] Stone et al . An Anatomically Constrained Model for Path Integration in the Bee Brain , Current155 Biology ( 2017 ) , http : //dx.doi.org/10.1016/j.cub.2017.08.052156 [ 2 ] Horton JC , Adams DL . The cortical column : a structure without a function . Philos Trans R157 Soc Lond B Biol Sci . 2005 Apr 29 ; 360 ( 1456 ) :837-62. doi : 10.1098/rstb.2005.1623 . PMID:158 15937015 ; PMCID : PMC1569491.159 [ 3 ] Maya-Vetencourt J F , Caleo M . Experience-Dependent Plasticity in the Central Nervous160 System [ M ] . Springer Berlin Heidelberg , 2013.161 [ 4 ] Elliott , K. , Houston , D. & Fritzsch , B. Sensory afferent segregation in three-eared frogs resemble162 the dominance columns observed in three-eyed frogs . Sci Rep 5 , 8338 ( 2015 ) . https : //doi.163 org/10.1038/srep08338164 [ 5 ] Haferlach T , Wessnitzer J , Mangan M , et al . Evolving a Neural Model of Insect Path165 Integration [ J ] . Adaptive Behavior , 2007 , 15 ( 3 ) .166 [ 6 ] Cueva C J , Wang P Y , Chin M , et al . Emergence of functional and structural properties of the167 head direction system by optimization of recurrent neural networks [ J ] . 2019.168 [ 7 ] Kanitscheider I , Fiete I . Training recurrent networks to generate hypotheses about how the brain169 solves hard navigation problems [ J ] . 2016.170 [ 8 ] Hafting T , Fyhn M , Molden S , et al . Microstructure of a spatial map in the entorhinal cortex [ J ] .171 Nature , 2005 , 436 ( 7052 ) :801-806.172 [ 9 ] Cueva C J , Wei X X . Emergence of grid-like representations by training recurrent neural173 networks to perform spatial localization [ J ] . 2018.174 [ 10 ] Banino A , Barry C , Uria B , et al . Vector-based navigation using grid-like representations in175 artificial agents [ J ] . Nature , 2018.176 [ 11 ] Simonyan K , Zisserman A . Very Deep Convolutional Networks for Large-Scale Image177 Recognition [ J ] . Computer Science , 2014.178 [ 12 ] Vaswani A , Shazeer N , Parmar N , et al . Attention Is All You Need [ J ] . arXiv , 2017.179 [ 13 ] D Ulyanov , Vedaldi A , Lempitsky V . Deep Image Prior [ J ] . IEEE , 2017.180 [ 14 ] Sara Sabour , Nicholas Frosst , Geoffrey E Hinton . Dynamic Routing Between Capsules . arXiv:181 1710.09829 , 2017182 [ 15 ] Krogh A . A Simple Weight Decay Can Improve Generalization [ J ] . Advances in Neural Infor-183 mation Processing Systems , 1992 , 4.184 [ 16 ] Hinton G . How to represent part-whole hierarchies in a neural network [ J ] . 2021.185 [ 17 ] Chen C C , Yang C L , Cheng H Y . Efficient and Robust Parallel DNN Training through Model186 Parallelism on Multi-GPU Platform . 2018.187 [ 18 ] Kohonen T . The self-organizing map [ J ] . IEEE Proc Icnn , 1990 , 1 ( 1-3 ) :1-6.188 [ 19 ] Hebb , Donald O . The Organization of Behavior . 1949.189 [ 20 ] Lecun Y , Cortes C . The mnist database of handwritten digits . 2010.190 [ 21 ] Fldik P . Forming sparse representations by local anti-Hebbian learning [ J ] . Biological Cybernet-191 ics , 1990 , 64 ( 2 ) .192 [ 22 ] Olshausen B A , Field D J . Sparse Coding with an Overcomplete Basis Set : A Strategy193 Employed by V1 [ J ] . Vision Research , 1997 , 37 ( 23 ) :607-609.194 [ 23 ] Stern M , Bolding K , Abbott L F , et al . A Transformation From Latency To Ensemble Coding195 In A Model Of Piriform Cortex [ J ] . Elife , 2017 , 7.196 [ 24 ] Haberly L B . Parallel-distributed Processing in Olfactory Cortex : New Insights from Morpho-197 logical and Physiological Analysis of Neuronal Circuitry [ J ] . Chemical Senses , 2001 ( 5 ) :551-576.198 Checklist199 1 . For all authors ... 200 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s201 contributions and scope ? [ Yes ] 202 ( b ) Did you describe the limitations of your work ? [ Yes ] Please see 4.2 5203 ( c ) Did you discuss any potential negative societal impacts of your work ? [ No ] This204 research is general research but not a applied algorithm research that could have any205 societal impacts206 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to207 them ? [ Yes ] 208 2 . If you are including theoretical results ... 209 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 210 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 211 3 . If you ran experiments ... 212 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-213 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] Please214 see 4215 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they216 were chosen ) ? [ Yes ] Please see 4217 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-218 ments multiple times ) ? [ Yes ] Please see 4219 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type220 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] Please see 4221 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 222 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] Please see 4.1223 ( b ) Did you mention the license of the assets ? [ No ] Cant ’ find licenses anywhere , even224 on the homepage of the dataset http : //yann.lecun.com/exdb/mnist/ , should be225 open to use226 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 227 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re228 using/curating ? [ No ] Dataset is open to use229 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable230 information or offensive content ? [ No ] No personally identifiable information in dataset231 5 . If you used crowdsourcing or conducted research with human subjects ... 232 ( a ) Did you include the full text of instructions given to participants and screenshots , if233 applicable ? [ N/A ] 234 ( b ) Did you describe any potential participant risks , with links to Institutional Review235 Board ( IRB ) approvals , if applicable ? [ N/A ] 236 ( c ) Did you include the estimated hourly wage paid to participants and the total amount237 spent on participant compensation ? [ N/A ] 238

Summary: The paper introduces an approach to function-level clustering. Function-level clustering means that a conditional computation model is optimized (in this case, parallel copies of a neural network), and that the optimization step is grouped by the respective computation path. This is achieved by using a  "self organizing map" that can compose  to select minimal loss computation paths. In more detail, it: 1. computes the function through all of its existing computation paths 2. selects the lowest-loss path 2. optimizes that path and its neighbors only