V1Net: A computational model of cortical horizontal connections

The primate visual system builds robust , multi-purpose representations of the external world in order to support several diverse downstream cortical processes . Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting , local texture distortion , etc . A key architectural feature combating such environmental irregularities is long-range horizontal connections that aid the perception of the global form of objects . In this work , we explore the introduction of such horizontal connections into standard deep convolutional networks ; we present V1Net – a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity . We introduce the Texturized Challenge – a new benchmark to evaluate object recognition performance under perceptual noise – which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing . Additionally , we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images . We also present the emergence of several biologically plausible horizontal connectivity patterns , namely centeron surround-off , association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 ( BSDS500 ) . Our findings suggest an increased representational similarity between V1Net and biological visual systems , and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision . 1 INTRODUCTION . Following Hubel and Wiesel ’ s ( Hubel & Wiesel , 1968 ) seminal work on characterizing receptive fields in the cat striate cortex and Fukushima ’ s Neocognitron ( Fukushima , 1980 ) ( a hierarchical extension of this building block ) , two broad families of visual models have been developed by the neuroscience and computer vision communities respectively . The former family of models aims to account for findings from single-cell neurophysiology , either by directly modeling types of neuronal responses ( De Valois et al. , 1982 ; Sullivan & De Sa , 2006 ; Chichilnisky , 2001 ; Pillow , 2007 ) or by proposing computational models that give rise to similar neural phenomena ( Olshausen & Field , 1997 ; Schwartz et al. , 2006 ) . The latter family of models , particularly Deep Convolutional Networks ( DCNs ) ( LeCun et al. , 1998 ) , are also loosely inspired by Hubel & Wiesel ( 1968 ) and Fukushima ( 1980 ) ; they aim to optimize performance on a wide range of computer vision benchmarks including but not limited to image recognition ( Krizhevsky et al. , 2012 ; Hu et al. , 2018 ; Simonyan & Zisserman , 2014 ) , contour detection ( Xie & Tu , 2015 ; Shen et al. , 2015 ) and object segmentation ( He et al. , 2017 ; Long et al. , 2015 ) . Despite their impressive performance on benchmarks in these areas , these models progressively deviate from biological vision and recent work highlights their interesting deficiencies and sensitivities relative to primate vision ( Geirhos et al. , 2019 ; Papernot et al. , 2016 ; Kurakin et al. , 2016 ; Eykholt et al. , 2017 ) . Our motivation is to reverse-engineer the cortical contextual processing principles that have been shown to be mediated by long-range horizontal connections using DCNs . Horizontal connections are capable of systematically filling-in sensory inconsistencies and spatially binding neighbouring features together in order to provide stable perception . In an attempt to complement DCNs with this property and to contribute towards bridging the gap between artificial and biological visual representations , we develop ‘ V1Net ’ , a novel recurrent unit inspired by visual neuroscience and Gestalt psychology literature on cortical horizontal connectivity ( Das & Gilbert , 1995 ; Grossberg & Mingolla , 1985 ) . V1Net can be flexibly incorporated as a module in existing implementations of DCNs . In the following Section 2 , we briefly survey the existing literature on biologically plausible vision models that are related to V1Net . In Section 3 , we introduce the V1Net model along with it ’ s mathematical formulation and an intuitive explanation of its working . In Section 4 , we demonstrate experimental results from : ( 1 ) Texturized Challenge , our proposed benchmark to evaluate object recognition ability under perceptual noise and ( 2 ) an ablation study of V1Net ’ s horizontal connections using the BSDS500 boundary detection benchmark ( Arbelaez et al. , 2011 ) . Subsequently , we demonstrate several biologically plausible horizontal connections emerging in a V1Net while learning the task of boundary detection . We also share qualitative results of V1Net ’ s zero-shot domain transfer on the task of object boundary detection from natural images ( used for training ) to stylized images ( Gatys et al. , 2015 ) . Finally , we discuss our current plans for extending the work . 2 RELATED WORK . Computational models of visual perception have been an interesting and active area of research in the vision science and machine vision communities . Riesenhuber & Poggio ( 1999 ) and Serre et al . ( 2007b ; a ) introduced hierarchical visual models inspired by computations in the visual cortex to perform robust object recognition . Relevant to their models is the recently proposed CORNet by Kubilius et al . ( 2018 ) . CORNet is a deep recurrent-convolutional model of the ventral visual stream primarily motivated by the task of core object recognition . There has been a parallel direction of research related to cortically inspired models , such as Fan et al . ( 2018 ) that compare the representational similarity between the ventral visual stream and hierarchical deep convolutional models trained on image recognition . Concurrently , there have been several efforts such as Cao et al . ( 2018 ) ; Wang et al . ( 2018 ) from the computer vision community on incorporating long-range spatial dependencies and lateral inhibition into deep convolutional networks . There has been a great deal of interest in exploiting the potential of dynamical systems to model early visual perception which is highly relevant to our work . Early computational models of visual routines proposed by Grossberg & Mingolla ( 1985 ) ; Grossberg & Williamson ( 2001 ) ; Li et al . ( 2006 ) were primarily implemented in this dynamical systems framework using partial differential equations . However with the recent hardware and software improvements ( Jouppi et al. , 2017 ; Abadi et al. , 2016 ) to efficiently train Recurrent Neural Networks ( RNNs ) , there has been a surge in the development of biologically plausible models of vision using RNNs from the neuroscience and computer vision communities . Kar et al . ( 2019 ) provide experimental and computational modeling results demonstrating the need for recurrence in core object recognition routines . Through a large-scale neural architecture search over models that vary along the dimensions of their core operations , Nayebi et al . ( 2018 ) demonstrate the emergence of architectures equipped with recurrent and feedback interactions to be ideal for object recognition . Additionally , they provide evidence for the model ’ s neural plausibility by explaining neural data recorded from the visual cortex of awake primates during image-viewing . The horizontal Gated Recurrent Unit ( hGRU ) introduced by Linsley et al . ( 2018 ) is relevant to our proposed V1Net . However , there are several key differences between hGRU and the V1Net in terms of their design and horizontal connection dynamics . Among other differences , V1Net ’ s computations are not deliberately divided into two sequential within-timestep stages . In hGRU the inhibitory horizontal connections are developed in the first stage which are followed by excitatory horizontal connections . However in V1Net , linear excitatory and inhibitory connections are not exercised in any particular order . As a subtle form of ordering , our nonlinear shunting inhibition mechanism is implemented disynaptically as they inhibit the excitatory cells which synapse onto simple cells in a facilitatory fashion ( see Eqn.1 ) . We find this design to be more in line with biological horizontal connections . 3 V1NET : A MODEL OF LONG-RANGE HORIZONTAL CONNECTIONS . We employ recurrent-convolutional neural networks to model V1Net ’ s computation . V1Net consists of a population of neurons akin to simple cells in V1 that learn to detect features from their presynaptic input ( available in the convolutional activation maps from the corresponding bottom layer ) . Each ‘ simple-cell ’ receives recurrent inhibitory and excitatory influences from its neighboring cells which we implement using the following three types of horizontal connections : ( 1 ) Linear additive excitation , ( 2 ) Linear subtractive inhibition , and ( 3 ) Nonlinear divisive shunting inhibition . These three types of horizontal connections create a push-pull integration at each neural site , resulting in a net inhibition/excitation of a neuron ’ s activity as a nonlinear function of its neighbors ’ activity . 3.1 MATHEMATICAL FORMULATION OF V1NET . We derive the V1Net cell by starting from the convolutional variant of a Long Short Term Memory ( LSTM ) cell ( Hochreiter & Schmidhuber , 1997 ) called the ConvLSTM ( Xingjian et al. , 2015 ) . The hidden-state dynamics of a ConvLSTM are governed by that of the original LSTM update equations with the fully-connected operations replaced with convolution operations . At any discrete timestep t of recurrent processing , we utilize the previous convolutional layer ’ s activation map to set the value of our recurrent input tensor at time t , Xt ; i.e , the input tensor Xt is set to the bottom layer ’ s feature representation at all timesteps t = 0 to the final timestep t = T . V1Net ’ s different horizontal connections are implemented using distinct convolutional filter banks with varying spatial dimensions , namely Wshunt , Winh and Wexc . These convolutional filter banks model the horizontal connectivity between all possible pairs of feedforward ‘ simple-cells ’ ( or convolution feature maps ) giving us the ability to visualize the nature ( inhibitory/excitatory ) of the horizontal connections along with their spatial profile and connectivity strength between any pair of feedforward receptive fields . Horizontal connection kernels convolve over the hidden state of V1Net , Ht−1 to produce Hshuntt , Hinht and H exc t respectively . The activity at each position H exc t [ i , j , k ] in layer L denotes the net horizontal excitation received by a feedforward neuron at spatial location ( i , j ) in the kth activation map from its spatial neighbors within a fixed distance from all activation maps in layer L. Similarly , Hinht and H shunt t encode the pointwise net contextual linear and nonlinear inhibition received by the feedforward neurons . Inhibitory and excitatory influences are then integrated with the input at the current timestep to produce c̃t , the candidate memory cell at time t. c̃t is mixed with V1Net ’ s memory cell from the previous timestep to compute ct , the current memory cell of V1Net . The current hidden state Ht of V1Net is computed as a gated nonlinear function of the cell state ct. A key property to note is that receptive fields of neurons in the horizontal kernels grow at every iteration of recurrence , and hence their range of horizontal connectivity scales proportionate to the amount of recurrent processing . The internal working of our proposed V1Net unit is shown in Fig.1 ( left ) alongside that of a standard ConvLSTM ( right ) . The following equations summarize the working of V1Net : ft = σ ( Wf ∗Xt + Uf ∗Ht−1 ) it = σ ( Wi ∗Xt + Ui ∗Ht−1 ) ot = σ ( Wo ∗Xt + Uo ∗Ht−1 ) Hexct , H inh t = σ ( Wexc ∗Ht−1 ) , σ ( Winh ∗Ht−1 ) Hshuntt = σ ( Wshunt ∗Ht−1 ) c̃t = α×Hshuntt × ( Wc ∗Xt +Hexct ) − β ×Hinht ct = ft ct−1 + it σ ( c̃t ) Ht = ot tanh ( ct ) ( 1 ) Each W. and U. is a 2-D convolution kernel , ‘ ∗ ’ and σ ( . ) represent 2D convolution operation and the sigmoid nonlinearity respectively . α and β are learnable parameter vectors that allow V1Net to selectively scale up/down horizontal influences in a channel-wise fashion . Additionally as a straightforward mathematical way to compare a V1Net cell to a ConvLSTM cell , we present below the state-update equations of a standard ConvLSTM cell . ft = σ ( Wf ∗Xt + Uf ∗Ht−1 ) it = σ ( Wi ∗Xt + Ui ∗Ht−1 ) ot = σ ( Wo ∗Xt + Uo ∗Ht−1 ) c̃t = σ ( Wc ∗Xt + Uc ∗Ht−1 + bc ) ct = ft ct−1 + it σ ( c̃t ) Ht = ot tanh ( ct ) ( 2 ) From ( 1 ) and ( 2 ) , it can be observed that the equations governing the representation dynamics of V1Net and ConvLSTM are fairly similar . We also present diagrammatically , the process of converting a ConvLSTM into a V1Net by adding the transparent components in Fig.1 ( right ) .

Summary: I first want to thank the authors for their proposed approach in this paper. Authors made an attempt to bridge the gap between natural (primate) vision and NN models. The paper is easy to read and understand. The authors proposed a Conv-LSTM-inspired model called V1Net. The model shows some merits in detecting the correct labels for noised inputs.