Divisive Normalization, Line Attractor Networks and Ideal Observers
Authors: Sophie Denève, Alexandre Pouget, Peter E. Latham
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Gain  control  by  divisive  inhibition,  a.k.a.  divisive  normalization,  has  been  proposed  to  be  a  general  mechanism  throughout  the  vi(cid:173) sual  cortex.  We  explore  in  this  study  the  statistical  properties  of this  normalization in  the  presence  of noise.  Using  simulations,  we  show  that divisive  normalization is  a  close  approximation to  a  maximum likelihood estimator, which, in the context of population  coding, is the same as an ideal observer.  We also demonstrate ana(cid:173) lytically that this is  a general property of a  large class of nonlinear  recurrent  networks  with  line  attractors.  Our  work  suggests  that  divisive  normalization  plays  a  critical  role  in  noise  filtering,  and  that every cortical layer may be an ideal observer of the activity in  the preceding  layer. 
Information  processing  in  the  cortex  is  often  formalized  as  a  sequence  of a  linear  stages followed  by  a  nonlinearity.  In the visual cortex,  the nonlinearity is  best de(cid:173) scribed by squaring combined with a divisive pooling of local activities.  The divisive  part of the nonlinearity has  been  extensively studied by  Heeger  and colleagues  [1],  and several authors have explored the role of this normalization in the computation  of high  order visual features  such  as  orientation of edges or first  and  second order  motion[ 4].  We show in this paper that divisive normalization can also playa role in  noise filtering.  More specifically, we demonstrate through simulations that networks  implementing this normalization come close to performing maximum likelihood es(cid:173) timation.  We  then demonstrate analytically that the ability  to  perform maximum  likelihood estimation, and thus efficiently extract information from  a  population of  noisy neurons,  is  a  property exhibited  by  a  large class of networks. 
Maximum  likelihood  estimation  is  a  framework  commonly  used  in  the  theory  of  ideal observers.  A recent example comes from the work of Itti et al.,  1998, who have  shown  that it  is  possible  to  account  for  the behavior of human  subjects  in simple  discrimination  tasks.  Their  model  comprised  two  distinct  stages:  1)  a  network 
Divisive Normalization.  Line Attractor Networks and Ideal Observers 
105 
which  models  the noisy  response of neurons  with  tuning  curves to  orientation and  spatial frequency  combined with divisive normalization, and 2)  an ideal observer (a  maximum likelihood estimator)  to read out the population activity of the network. 
Our  work  suggests  that  there  is  no  need  to  distinguish  between  these  two  stages,  since,  as  we  will show,  divisive normalization comes close to  providing a  maximum  likelihood estimation.  More generally,  we  propose that there may  not  be any  part  of the cortex that acts as  an  ideal observer for  patterns of activity in sensory areas  but, instead, that each cortical layer acts as an ideal observer of the activity in  the  preceding layer.