Learning Winner-take-all Competition Between Groups of Neurons in Lateral Inhibitory Networks
Authors: Richard H. R. Hahnloser, H. Sebastian Seung Xiaohui Xie
Published in: Advances in Neural Information Processing Systems 13  (NIPS 2000) (Main Conference Track), 2000

Abstract: It has long been known that lateral inhibition in neural networks can lead  to a winner-take-all competition, so that only a single neuron is active at  a steady  state.  Here we  show how  to  organize lateral inhibition so that  groups  of neurons  compete to  be  active.  Given  a  collection  of poten(cid:173) tially overlapping groups, the inhibitory connectivity is  set by a formula  that can be interpreted as arising from a simple learning rule.  Our analy(cid:173) sis demonstrates that such inhibition generally results in  winner-take-all  competition between  the  given  groups,  with  the exception of some de(cid:173) generate cases.  In a broader context, the network serves  as  a particular  illustration  of the  general  distinction  between  permitted  and  forbidden  sets,  which was introduced recently.  From this  viewpoint, the computa(cid:173) tional function  of our network is  to  store and retrieve memories as  per(cid:173) mitted sets of coactive neurons. 
In  traditional  winner-take-all  networks,  lateral  inhibition  is  used  to  enforce  a  localized,  or "grandmother cell" representation  in  which  only a  single  neuron  is  active  [1,  2,  3,  4].  When used  for  unsupervised learning,  winner-take-all  networks  discover representations  similar to  those learned by  vector quantization  [5].  Recently  many  research efforts have  focused on unsupervised learning algorithms for sparsely distributed representations [6, 7].  These algorithms lead to  networks in which groups of multiple neurons are coactivated to  represent an  object.  Therefore, it is of great interest to find ways of using lateral inhibition  to mediate winner-take-all competition between groups of neurons, as this could be useful  for learning sparsely distributed representations. 
In this paper, we show how winner-take-all competition between groups of neurons can be  learned.  Given a collection of potentially overlapping groups, the inhibitory connectivity  is  set by  a simple formula  that can  be interpreted as  arising from an  online learning rule.  To show that the resulting network functions as advertised, we perform a stability analysis.  If the strength of inhibition is sufficiently great, and the group organization satisfies certain  conditions, we show that the only sets of neurons that can be coactivated at a stable steady  state are  the  given groups and their subsets.  Because of the competition between groups,  only  one group can  be  activated  at  a time.  In  general,  the identity  of the  winning group  depends on the initial conditions of the network dynamics. If the groups are ordered by the  aggregate input that each receives, the possible winners are those above a cutoff that is set  by inequalities to be specified. 
1  Basic definitions 
Let m  groups of neurons be given, where group membership is specified by the matrix 
fl = {I  if the ith neuron is in the ath group  ,  ° otherwise 
(1) 
We will  assume that every neuron belongs to at least one group l, and every group contains  at  least  one  neuron.  A  neuron is  allowed to  belong  to  more than  one group,  so  that the  groups are potentially overlapping.  The inhibitory synaptic connectivity of the network is  defined in terms of the group membership, 
Ji ' = lIm (1  _ ~a ~'!) = {o