A Recurrent Model of Orientation Maps with Simple and Complex Cells
Authors: Paul Merolla, Kwabena A. Boahen
Published in: Advances in Neural Information Processing Systems 16  (NIPS 2003) (Main Conference Track), 2003

Abstract: that utilizes 
We describe a neuromorphic chip  transistor  heterogeneity, introduced by the fabrication process, to generate  orientation maps similar to those imaged in vivo. Our model  consists of a recurrent network of excitatory and inhibitory cells in  parallel with a push-pull stage. Similar to a previous model the  recurrent network displays hotspots of activity that give rise to  visual feature maps. Unlike previous work, however, the map for  orientation does not depend on the sign of contrast. Instead, sign- independent cells driven by both ON and OFF channels anchor the  map, while push-pull interactions give rise to sign-preserving cells.  These two groups of orientation-selective cells are similar to  complex and simple cells observed in V1. 
1 Orientation Maps 
Neurons in visual areas 1 and 2 (V1 and V2) are selectively tuned for a number of  visual features, the most pronounced feature being orientation. Orientation  preference of individual cells varies across the two-dimensional surface of the  cortex in a stereotyped manner, as revealed by electrophysiology [1] and optical  imaging studies [2]. The origin of these preferred orientation (PO) maps is debated,  but experiments demonstrate that they exist in the absence of visual experience [3].  To the dismay of advocates of Hebbian learning, these results suggest that the initial  appearance of PO maps rely on neural mechanisms oblivious to input correlations.  Here, we propose a model that accounts for observed PO maps based on innate noise  in neuron thresholds and synaptic currents. The network is implemented in silicon  where heterogeneity is as ubiquitous as it is in biology. 
2 Patterned Activity Model 
Ernst et al. have previously described a 2D rate model that can account for the  origin of visual maps [4]. Individual units in their network receive isotropic  feedforward input from the geniculate and recurrent connections from neighboring 
units in a Mexican hat profile, described by short-range excitation and long-range  inhibition. If the recurrent connections are sufficiently strong, hotspots of activity  (or ‘bumps’) form periodically across space. In a homogeneous network, these  bumps of activity are equally stable at any position in the network and are free to  wander.  Introducing random jitter to the Mexican hat connectivity profiles breaks the  symmetry and reduces the number of stable states for the bumps. Subsequently, the  bumps are pinned down at the locations that maximize their net local recurrent  feedback. In this regime, moving gratings are able to shift the bumps away from  their stability points such that the responses of the network resemble PO maps.  Therefore, the recurrent network, given an ample amount of noise, can innately  generate its own orientation specificity without the need for specific hardwired  connections or visually driven learning rules. 
2.1 Criticisms of the Bump model 
We might posit that the brain uses a similar opportunistic model to derive and  organize its feature maps – but the parallels between the primary visual cortex and  the Ernst et al. bump model are unconvincing. For instance, the units in their model  represent the collective activity of a column, reducing the network dynamics to a  firing-rate approximation. But this simplification ignores the rich temporal  dynamics of spiking networks, which are known to affect bump stability. More  fundamentally, there is no role for functionally distinct neuron types.  The primary criticism of the Ernst et al.’s bump model is that its input only consists  of a luminance channel, and it is not obvious how to replace this channel with ON  and OFF rectified channels to account for simple and complex cells. One possibility  would be to segregate ON-driven and OFF-driven cells (referred to as simple cells in  this paper) into two distinct recurrent networks. Because each network would have  its own innate noise profile, bumps would form independently. Consequently, there  is no guarantee that ON-driven maps would line up with OFF-driven maps, which  would result in conflicting orientation signals when these simple cells converge onto  sign-independent (complex) cells. 
2.2 Simple Cells Solve a Complex Problem 
To ensure that both ON-driven and OFF-driven simple cells have the same  orientation maps, both ON and OFF bumps must be computed in the same recurrent  network so that they are subjected to the same noise profile. We achieve this by  building our recurrent network out of cells that are sign-independent; that is both  ON and OFF channels drive the network. These cells exhibit complex cell-like  behavior (and are referred to as complex cells in this paper) because they are  modulated at double the spatial frequency of a sinusoidal grating input. The simple  cells subsequently derive their responses from two separate signals: an orientation  selective feedback signal from the complex cells indicating the presence of either an  ON or an OFF bump, and an ON–OFF selection signal that chooses the appropriate  response flavor.  Figure 1 left illustrates the formation of bumps (highlighted cells) by a recurrent  network with a Mexican hat connectivity profile. Extending the Ernst et al. model,  these complex bumps seed simple bumps when driven by a grating. Simple bumps  that match the sign of the input survive, whereas out-of-phase bumps are  extinguished (faded cells) by push-pull inhibition.  Figure 1 right shows the local connections within a microcircuit. An EXC  (excitatory) cell receives excitatory input from both ON and OFF channels, and 
projects to other EXC (not shown) and INH (inhibitory) cells. The INH cell projects  back in a reciprocal configuration to EXC cells. The divergence is indicated in left.  ON-driven and OFF-driven simple cells receive input in a push-pull configuration  (i.e., ON cells are excited by ON inputs and inhibited by OFF inputs, and vise-versa),  while additionally receiving input from the EXC–INH recurrent network. In this  model, we  inhibitory  connections, despite the fact that geniculate input is strictly excitatory. This  simplification, while anatomically incorrect, yields a more efficient implementation  that is functionally equivalent.   
implement our push-pull circuit using monosynaptic