V1Net: A computational model of cortical horizontal connections
Authors: Vijay Veerabadran
Virginia R. de Sa
Venue:  (2020)
Keywords: Biologically plausible deep learning
Recurrent Neural Networks
Perceptual grouping
horizontal connections
visual neuroscience
perceptual robustness
Gestalt psychology

Abstract: The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes. Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc. A key architectural feature combating such environmental irregularities is ‘long-range horizontal connections’ that aid the perception of the global form of objects. In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity. We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing. Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images. We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500). Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

Under review as a conference paper at ICLR 2020
VINET: A COMPUTATIONAL MODEL OF CORTICAL HORIZONTAL CONNECTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The primate visual system builds robust, multi-purpose representations of the ternal world order to support several diverse downstream cortical processes. Such representations caused by dynamically varying lighting, local texture distortion, etc. A key ar- chitectural feature combating such environmental irregularities is long-range hor- izontal connections that aid the perception of the global form of objects. In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V Vet novel convolutional-recurrenn unit that models linear and nonlinear horizontal inhibitory and excitatory connec- tions inspired by primate visual cortical connectivity. We introduce the Texturized Challenge new benchmark to evaluate object recognition performance under perceptual noise which we use to evaluate V Net against an array of carefully se- lected control models with/without recurrent processing. Additionally, we present results from an ablation study of VINet demonstrating the utility of diverse neu- rally inspired horizontal connections for state-of-the-art Al systems on the task object boundary detection from natural images. We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center- surround-off, association fields and border-ownership connectivity patterns a VINet model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500). Our findings suggest an in- creased representational similarity between VINet and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the statc-of-the-an in computer vision.
required be invariant to the sensory inconsistencies
INTRODUCTION
Following Hubel and Wiesel's (Hubel & Wiesel, 1968) seminal work on characterizing receptive fields in the striate cortex and Fukushima's Neocognitron (Fukushima 1980) (a hierarchical extension of this building block), two broad families of visual models have been developed by the neuroscience and computer vision communities respectively. The former family of models aims to account for findings from single-cell neurophysiology, either by directly modeling types of neuronal
responses (De Valois al. 1982 Sullivan & De Sa 2006 Chichilnisky 200T Pillow 2007 or by proposing computational models that give rise to similar neural phenomena (Olshausen & Field
1997 Schwartz al. 2006).
The latter family of models, particularly Deep Convolutional Networks (DCNs) (LeCun et al. 1998) are also loosely inspired by HHueel & Wiesel (1968) and Fukushima (1980); they aim to optimize performance a wide range of computer vision benchmarks including but not limited to image recognition (Krizhevsky et al., 2012 Hu et al. 2018 Simonyan & Zisserman 2014), contour detec- tion (iie & Tu 2015 Shen et al. 2015) and object segmentation (He et al. |
al. 2015).
Despite their impressive performance on benchmarks in these areas, these models progressively de- viate from biological vision and recent work highlights their interesting deficiencies and sensitivities relative to primate vision (Geirhos et al. 2019, Papernot et al. 2016; Kurakin et al.] 2016 Eykholt
al. 2017

---

Under review as a conference paper at ICLR 2020
Our motivation is to reverse-engineer the cortical contextual processing principles that have been shown to be mediated by long-range horizontal connections using DCNs. Horizontal connections capable of systematically filling-in sensory inconsistencies and spatially binding neighbouring features together in order to provide stable perception. In an attempt to complement DCNs with this property and contribute towards bridging the gap between artificial and biological visual representations, we develop 'VINet', a novel recurrent unit inspired by visual neuroscience and Gestalt psychology literature on cortical horizontal connectivity (Das & Gilbert 1995] Grossberg & Mingolla 1985). V1Net can be flexibly incorporated as a module in existing implementations of DCNs.
In the following Section 2. we briefly survey the existing literature on biologically plausible vi- sion models that are related VINet. In Section 3, we introduce the VINet model along with it's mathematical formulation and an intuitive explanation of its working. In Section 4, we demonstrate experimental results from: (1) Texturized Challenge, our proposed benchmark to evaluate object recognition ability under perceptual noise and (2) an ablation study of V horizontal connec- tions using the BSDS500 boundary detection benchmark (Arbelaez et al. 201T). Subsequently, we demonstrate several biologically plausible horizontal connections emerging a VINet while learn- ing the task of boundary detection. We also share qualitative results of VINet's 's zero-shot domain transfer on the task of object boundary detection from natural images (used for training) stylized images (Gatys et al. 2015). Finally, we discuss our current plans for extending the work.
2 RELATED WORK
Computational models of visual perception have been an interesting and active area of research in the vision science and machine vision communities. RRisenhuber & Poggio (1999) and /eeree ct al. (2007b a) introduced hierarchical visual models inspired by computations in the visual cortex to perform robust object recognition. Relevant to their models is the recently proposed CORNet by Kubillius et al. (2018). CORNet is deep recurrent-comvoluttonaal model of the ventral visual stream primarily motivated by the task of core object recognition. There has been a parallel direc- tion of research related cortically inspired models, such as Fan et al. (2018) that compare the representational similarity between the ventral visual stream and hierarchical deep convolutional models trained on image recognition. Coneurrently, there have been several efforts such as [ao et al. (018); Wang et al. (2088) from the computer vision community on incorporating long-range spatial dependencies and lateral inhibition into deep convolutional networks.
There has been great deal of interest in exploiting the potential of dynamical systems to model early visual perception which is highly relevant to our work. Early computational models of vi- sual routines proposed by Grossberg & Mingolla (1985); Grossberg & Williamson (200T); Li et al. (2006) were primarily implemented in this dynamical systems framework using partial differen- tial equations. However with the recent hardware and software improvements (Jouppi et al. 2017 Abadi et al. (0166) to efficiently train Recurrent Neural Networks (RNNs). there has been a surge in the development of biologically plausible models of vision using RNNs from the neuroscience and computer vision communities. KKar et al. (2019) provide experimental and computational mod- cling results demonstrating the need for recurrence in core object recognition routines. Through large-scale neural architecture search over models that vary along the dimensions of their core op- erations. Nayebi et al. (2018) demonstrate the emergence of architectures equipped with recurrent and feedback interactions to be ideal for object recognition. Additionally, they provide evidence for the model's 's neural plausibility by explaining neural data recorded from the visual cortex of awake primates during image-viewing.
The horizontal Gated Recurrent Unit (hGRU) introduced by Linsley et al. 2018 relevant to our proposed VeNet. However, there are several key differences between hGRU and the V1Net in terms of their design and horizontal connection dynamics. Among other differences, VINet's computa- tions are not deliberately divided into two sequential within-timestep stages. In hGRU the inhibitory horizontal connections are developed in the first stage which are followed by excitatory horizontal connections. However in V VINet, linear excitatory and inhibitory connections are not exercised in any particular order. As a subtle form of ordering, our nonlinear shunting inhibition mechanism is implemented disynaptically they inhibit the excitatory cells which synapse onto simple cells in
2

---

Under review as a conference paper at ICLR 2020
facilitatory fashion (see Equ Dp. We find this design to be more in line with biological horizontal
connections.
VINET: MODEL OF LONG-RANGE HORIZONTAL CONNECTIONS
neural networks to model VINet's computation. VINet consists of a population of neurons akin to simple cells in VI that learn to detect features from their presy- naptic input (available in the convolutional activation maps from the corresponding bottom layer). Each "simple-cell" receives recurrent inhibitory and excitatory influences from its neighboring cells which we implement using the following three types of horizontal connections: (I) Linear additive excitation, (2) Linear subtractive inhibition, and (3) Nonlinear divisive shunting inhibition. These three types of horizontal connections create a push-pull integration at each neural site, resulting in net inhubition/excitation of neuron's 's activity as nonlinear function of its neighbors' activity.
We employ
3.1 MATHEMATICAL FORMULATION OF V VINET
We derive the VINet cell by starting from the convolutional variant of Long Short Term Memory (LSTM) cell (Hochreiter & Schmidhuber, 1997) called the ConvLSTM (Xingjian et al.) 2015). The hidden-state dynamics a ConvLSTM are governed by that of the original LSTM update equations with the fully-connected operations replaced with convolution operations. At any discrete timestep t of recurrent processing, we utilize the previous convolutional layer's s activation map to set the value of our recurrent input tensor time t, X₁; i.e, the input tensor X, is set the bottom layer's 's feature representation at all timesteps 0 to the final timestep =
Excitation
Xx
Shurting inhibtion
Subtractive inhibition
Convolution
X
Man
Gating
XF
Wea
E
C
Mrphiny
Hov
Hews
Hy-2
Heas
Op-
Ge-2
Cp-1
C
Figure 1: Comparison of a V1Net cell (left) with ConvLSTM cell (right)
VINer's different horizontal connections are implemented using distinct convolutional filter banks with varying spatial dimensions, namely W Wehunt, Wink and Were. These convolutional filter banks model the horizontal connectivity between all possible pairs of feedforward 'simple-cells' (or con- volution feature maps) giving the ability to visualize the nature (inhibitory/eciiatoryy of the horizontal connections along with their spatial profile and connectivity strength between any pair of feedforward receptive fields.
Horizontal connection kernels convolve over the hidden state of VINet, H,-1 to produce Hehaud H.hh and H, respectively. The activity each position H, j, k] in layer L denotes the horizontal excitation received by a feedforward neuron at spatial location (i.j) in the 1.th activation map from its spatial neighbors within fixed distance from all activation maps layer L. Similarly, Hiah and H. encode the pointwise net contextual linear and nonlinear inhibition received by the feedforward neurons. Inhibitory and excitatory influences are then integrated with the input at the current timestep to produce é, the candidate memory cell at time t. & is mixed with V1Net's memory cell from the previous timestep to compute Gt. the current memory cell VINet. The current hidden state H, of VlNet is computed as a gated nonlinear function of the cell state a-- A key property to note is that receptive fields of neurons in the horizontal kernels grow at every iteration of recurrence, and hence their range of horizontal connectivity scales proportionate to the amount of recurrent processing. The internal working our proposed VeN unit shown in Fig (left) alongside that of standard Convi (right).
3

---

Under review as a conference paper at ICLR 2020
The following equations summarize the working of VINet:
ft o(W, * X, + Uy * H(-1)
4 aWW; X; U; Hę-1)
0 = ( X, U. H{-1)
H Hinhh = o(Wer exe H4-1), (Wanh * H(-1)
Hahunt = =o shunt * -H--))
& H.huun x (We Xt H B H
Q = ft c 4-1 is ((()
H, o2 c tanh(c))
Each W. and U. is 2-D convolution kernel, '*' and a.)) represent 2D convolution operation and the sigmoid nonlinearity respectively. and are learnable parameter vectors that allow VINet to selectively scale up/down horizontal influences in channel-wise fashion. Additionally as straightforward mathematical way compare VINet cell to a ConvLSTM cell, we present below the state-update equations of a standard ConvLSTM cell.
ft oW, X, + +U,+ Hz-1)
i, oW; X₂ + U; Ht-1)
0 a aW, *X, + U, Hı-1)
(2)
â = oW, * X, + Ue H,-1 be)
Ct ft c-1
He = @ tanh(c))
From and 2 it can be observed that the equations governing the representation dynamics of VINet and CoMVLSTM are fairly similar. We also present diagrammatically, the process of convert- ing a ConvLSTM into VINet by adding the transparent components in Fig (right).
3.2 PICTORIAL EXAMPLE DEMONSTRATING V VINET'S WORKING
By design in DCNs, the flow of information between neurons within particular spatial resolution (within layer) is restricted to the dimensions of the convolution kernels that layer. This may not be ideal for tasks such as object recognition in the presence of perceptual noise (such as clutter) or contour tracing wherein long-range neighbouring elements at a particular spatial resolution should explicitly be grouped together/sepparteed in order to form holistic object representation. V1Net's long-range horizontal connections precisely build this functionality of grouping within a DCN layer in order to efficiently link elements and represent objects robustly.
Consider the toy example of multi-digit recognition under perceptual noise (breaking up of object boundary) and clutter shown in Fig. 2 Standard DCNs that leam this task will require explicit supervision for every possible configuration of overlapping digits in order to classify both digits correctly. Moreover, one may suspect that DCN may generalize during test-time overlapping digit pairs that were not presented together during training time. This is because of the lack a DCN's ability group together elements that belong an object and separate out those that belong to different objects.
In a VINet model, the horizontal excitatory connections (highlighted in red in Fig. learn to selectively link elements that are like-oriented in a neighborhood, resulting in a strong Tacilitation and representational grouping of those elements that belong to a digit. On the other hand, inhibitory connections (highlighted in blue) learn to suppress elements that are not like-oriented and (probably) don't belong to the same object, hence perceptually decluttering and separating the different target digits to be recognized. Due to this reason, V1Nett model will require many fewer examples to learn to recognize digits under clutter as it only needs to learn invariant digit templates on top of the horizontal connections, and not require rote-memorization of every possible configuration of overlapping digits. For the same reasons a VINet model will require fewer parameters than a standard DCN at particular performance level, the DCN needs a large number of parameters to memorize all object configurations. Due to this decluttering effect of V ViNet, we hypothesize that will also generalize unseen pairs of overlapping/cluttered objects during test-time.

---

Under review as a conference paper at ICLR 2020
Figure 2: Toy example showing excitatory grouping (in red) and inhibitory decluttering (in blue) during digit recognition under clutter. Output represents the predicted probability distribution over all digits (light shade represents high probability). Transparency of horizontal connections reflect their strength.
4 EXPERIMENTS
We evaluate the proposed VINet model different tasks related to perceptual robustness and early visual representations. First, we describe the Texturized challenge, novel challenge we present along with this paper that aims to assess model's ability robustly recognize objects under an identity-presee preserving image transformation based on Fourier analysis. Second, we show ex- perimental results from a VlNet model trained perform the task of boundary detection on natural images from the Berkeley Segmentation Dataset, BSDS500. Please refer to Appendix A for details about the architectural design for these experiments.
4.1 TEXTURIZED CHALLENGE
Given the ability of horizontal connections to provide robustness against local perceptual noise, we designed challenge that call the 'Texturized challenge'. This challenge aims to assess the perceptual robustness of various models by testing their ability to recognize object shape under per- ceptual noise and uncertainty in their texture. All images used to generate the stimuli for texturized challenge were sourced from the CIFAR 10 object recognition dataset (Krizhevsky et al. 2009).
Phase
FFT
IFFT
Magnitude
a.
Magnilude
FFT
IFFT
Phose
Figure 3: A demonstration of our process used to generate Texturized Challenge images
By applying a Fourier transform on RGB images, we obtain representation of the image in Fourier space separating its phase and magnitude signals. One may incorporate local perceptual noise into an image, particularly in the object texture (Morgan et al. 1991) while retaining the overall object identity (via object form) by reconstructing the image with an intact phase and perturbed magnitude representation. However, corrupting the phase (which is of higher importance for object identity (Oppenheim Lim 1981 Piotrowski & Campbell, 1982) and retaining the magnitude results in reconstructed image with disrupted object identity, recognizing objects from such images is difficult even for humans and is hence not considered for our challenge. By exploiting this conceptual frame- work, generate images for our Texturized Challenge. Texturized images for each sample from CIFAR10 are obtained by reconstructing the sample using its own Fourier phase representation, and a linear combination of its own Fourier magnitude with the Fourier magnitude of different randomly selected image from CIFARIO. This selectively adds local perceptual noise to objects present in the CIFARIO dataset and retains the overall shape of objects. This complete procedure
5

---

Under review as a conference paper at ICLR 2020
demonstrated in Fig. As observable from the inset figures in Fig. 4 humans fairly robust to this challenge. The ratio that governs the amount of randomly selected Fourier magnitude de- notes the difficulty levels in the challenge; present results for increasing difficulty levels in evaluation below.
For this challenge, systematically compare VINet other parameter matched control models that are void of horizontal connections. Each model was developed using the above described layer framework, with the third layer being one of: (1) VINet with horizontal connections, (2) A traditional ConvLSTM model without horizontal connections (ConyLSTMM) and (3) A feedforward baseline with a regular 2D convolution without horizontal connections (Control-FF),
We evaluated the test-time classification accuracy of the above three types of models images from the Texturized challenge, the results for this evaluation are presented in Fig. Each model was tested on range of 4 different versions of the task with increasing difficulty levels plotted on the vertical axis. The horizontal axis represents the test-set accuracy of models different levels of task difficulty, which we measure using the amount of noise present in the Fourier magnitude.
From the below evaluation, we observe two attributes related to the efficacy of recurrence and hor- izontal connections: (1) Feedforward models are the least robust to perceptual noise compared to recurrent models. This validated by the feedforward control's quick deterioration classification accuracy at relatively casier version of the task (compared to recurrent models). This result pro- vides positive evidence to the claim that recurrent processing aids visual recognition, and is capable of adding robustness to visual representations. (2) The results also reflect the superior performance of the V Net model with horizontal connections compared to the control models. VINet's improved robustness relative also the recurrent control (ConvLSTM) highlights the utility of horizontal con- nections in enhancing the robustness of model's 's object representations.
Accunaty Tectutiadl Chelarge
Aemplene
Hasse
Free
Nn VINe
B COnVLSTM
ControlFFF
Figure 4: Performance of (1) V1Net, (2) Recurrent control (ConvI.S*) LTM) and (3) Feedforward con- trol models (Control-FF) on the Texturized challenge. Note that the difficulty level under the green bar is the same difficulty for the adjacent blue and red bars.
4.2 UTILITY OF HORIZONTAL CONNECTIONS TO DETECT OBJECT BOUNDARIES ABLATION STUDY
We evaluate the proposed VINet model on the task of boundary detection on natural images from the Berkeley Segmentation Dataset, BSDS500 (Arbelaez et al.) (011). We chose a boundary detection our model with early layers the primate visual cortex that detect contours from visual input among their other functions. We compare 3 different lesions of VeNet for this ablation study: (1) A standard ConvLSTM without horizontal connections acts comparison baseline with all horizontal connections lesioned. We test 2 variants of this architecture with (a) wider convolution kernels (ConvLSTM-Largehe and (b) greater number of channels (ConvL. STM-MMdRF) to match effective receptive field and/or number of parameters with
task due to the correspondence

---

Under review as a conference paper at ICLR 2020
VINeet respectively. (2) We evaluate the necessity of nonlinear inhibition in the presence of linear horizontal connections by testing lesion of VINet without shunting inhibition (VINet-Linear),
Fig 9 shows sample set of boundary predictions (from VINet) images from the BSDS500 test set. From our experiments with VINet, we have consistently noticed that VINet has relatively higher precision compared to maaceeter-matched control feedforward and recurrent models trained in similar environments.
Input
t=0
t=1
t=2
t=3
t=4.6
1=7
Figure 5: Evolution of V VINet boundary predictions through time on BSDS500 test set
Model
ODS-F
OIS-F
AP
nParams
Human
0.803
0.803
ConvLSTM-Largee
0.690
0.713
0.615
1845921
ConVLSTM-MedRF
0.653
0.671
0.581
527009
ViNet-Linear (t=2)
0.604
0.619
0.526
470241
ViNet-Linear (t=4)
0.669
0.687
0.633
470241
ViNet-Linear (1=6)
0.681
0.699
0.628
470241
VINet (1=6)
0.710
0.727
0.659
520513
VINet (1=6)*
0.738
0.753
0.711
520513
CEDN (Yang
2016)
0.788
0.804
119.6M
BDCN (He et al.
2019)
0.806
0.844
0.890
16.3M
HED
Xie & Tu 2015)
0.788
0.808
0.840
14.7M
Table I: Quantitative evaluation of BSDS; larger batch size
VINet model trained with data augmentation and
As suggested by the Optimal Dataset Scale F-1 scores (ODS-FI) in Table horizontal connec- tions in general are helpful to increase the precision and accuracy of boundary detection. While we have not included quantitative results for parameter-matched feedforward baseline, noticed its performance to be qualitatively worse than ConvLSTM-MedRR our parameter-matched recurrent baseline. These two observations together suggest the importance of recurrent processing for de- tecting object boundaries, and further the advantage of employing horizontal connections to push forward the prediction performance. Comparing our results from full VINet and one with hori- zontal nonlinear connections lesioned (V (ViNet-Lincar) reveals the pivotal role played by nonlinear horizontal connections in the process of boundary detection. In addition, progressively decreasing boundary prediction performance of two variants of VINet-Linear with shorter amounts of recur- rent processing (t=4 & t=2, 4 & 2 timesteps respectively) reveals a aoceddaccuracy tradcoff that is reminiscent of similar previously reported observations in primate behavioral studies.
order assess the impact standard techniques on improving performance on computer sion benchmarks. trained a VINet model (t=6) the BSDS500 training set with online data augmentation and 1.5x larger batch size. This experiment resulted improving VINet's per- formance on all three evaluation metrics, bringing the performance closer to that of state-of-the-a the-art object boundary detection methods (implementation details w/ hyperparameter choices can be found in Appendix. A).
4.3 EMERGENT NEURALLY PLAUSIBLE HORIZONTAL CONNECTIONS
There have been several accounts of "center-on surround-off surround modulation patterns in the visual cortex of primates perceiving natural scenes. We visualized the learned horizontal connection

---

Under review as a conference paper at ICLR 2020
patterns in VINet model trained to perform boundary detection natural images. The connection patterns reveal emergent center-surround and "association- fiiedd like receptive fields, validating VINet's increased biological plausibility. In addition, giving rise to border-sensitive cells our model that we hypothesize to correspond with 'border- ownership" cells (Hesse & Tsao 2016). Such connections in our model horizontally facilitate activ- ity of cells with receptive fields (RFs) on a particular side of an object boundary (inside/outside the object) and inhibit those with RFs the opposite side of the boundary. We believe such cells enabling the model to confidently predict boundaries of objects by providing strong signals of the object's extent and suppress those cells their surround that represent spurious boundaries. A sam- ple of the biologically realistic horizontal connections learned by V Wet while performing boundary detection on natural images is presented in Fig. 6
also observed horizontal connections
Figure 6: Horizontal connectivity patterns learned by VINet. Top and bottom rows show learned convolution kernels in VINet (simple-cells). The middle row shows spatial layout of learned hor- izontal connectivity patterns between the top and bottom cells (synaptic strength increases from blue/green to red).
4.4 ZERO- SHOT GENERALIZATION TO STYLE- TRANSFERRED STIMULI
In separate experiment, we tested how well our Ve model trained on detecting boundaries from natural images generalizes to style-transferred stimuli (top row in Fig. 7 from Sylizee-IImagenn Geirhos et al. 2019). Results from this experiment suggested that indeed, VINet was capable of zero-shot generalizing to robustly detect object boundaries from Stylized-Imagenet. Since there isn't a way to quantitatively analyze this experiment (lack of ground-truth data), we have included sample of VINet's predictions Stylized- Byylized-Imagnet below. We hypothesize that VINet's long- range horizontal connections enable it to learn global features that are spatially distant. Hence, V Net does not fail to detect boundaries, without explicit supervision, despite the texture of objects being stylized.
boundary prediction. Middle row: Bound- ary predictions without recurrent processing showing edges induced by fake-texture (low precision); Bottom row: Predictions from VINet post recurrent processing emphasizing the true object edges over the added-texture edges (high precision).
Figure 7: Zero-shot generalization to
8

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION
Horizontal connections have been a key interest to the field visual neuroscience, owing to their diverse functionality that gives rise to invariant visual representations. To this end, we develop neurally-inspired recurrent neural network model of long-range horizontal connections and demon- strate its importance also to artificial vision systems for giving rise to robust visual representations through a systematic comparison against parameter-matcher control models on our proposed Tex- turized challenge and the BSDS500 object boundary detection benchmark. We subsequently demon- strate the emergence of biologically plausible horizontal connection patterns from VINet suggesting strong representational similarity between VlNet and primate early visual areas. We also present qualitative results of a VINet model generalizing in zero-shot fashion to images with stylized tex- ture. Continuing in this direction of using inspiration from biological vision to advance artificial vision, we are currently working on incorporating learned top-down feedback connections in deep convolutional network along with VINet. We believe that such re-entrant connections along with horizontal connections will allow smooth flow of information within- and across different spatial resolutions of the feature hierarchy contributing to more robust and efficient semantic segmentation models that generalize better than their feedforward counterparts. In parallel, are actively work- ing on quantitatively analyzing the match between VINet's internal representations and single-cell experimental recordings collected from the primate visual cortex.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthiu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. Tensorflow: system for large- scale machine learning. In 12th {USENIX] Symposium on Operating Systems Design and Imple- mentation ( (OSDII} 16), pp. 265-283, 2016.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hier- archical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 33( 916, May 2011. doi: 10.1109/PAMM.2000.166 URL http: dx. dei wwt.oc........1111
ISSN 0162-8828
TPAMT PPAMT.2010.10 2010..61
Chunshui Cao, Yongzhen Huang, Zilei Wang. Liang Wang, Ninglong Xu, and Tieniu Tan. Lateral inhibition-inspired convolutional neural network for visual attention and saliency detection. In Shirty-Second AAAI Conference on Artificial Intelligence, 2018.
EJ Chichilnisky. A simple white noise analysis of neuronal light responses. Network: Computation in Neural Systems, 12(2):199-213, 2001.
Aniruddha Das and Charles D Gilbert. Long-range horizontal connections and their role in cortical reorganization revealed by optical recording of cat primary visual cortex. Nature, 375(6534) 1995.
Russell L De Valois, Duane G Albrecht, and Lisa G Thorell. Spatial frequency selectivity of cells in macaque visual cortex. Vision research, 22(5):545-555 1982.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-worid attacks on deep learning models. arXiv preprint arXiv:1707.0884 08945, 2017.
Judith E Fan, Daniel LK Yamins, and Nicholas B Turk-Browne. Common object representations for visual production and recognition. Cognitive science, 42(8):2670-2698, 2018
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of
pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-202, 1980.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm artistic style. arXiv preprint /508 2015.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traing CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. U htt.:httt////pnnne htw.reeeeee.o.orr////oooyyy

---

Under review as a conference paper at ICLR 2020
Stephen Grossberg and Ennio Mingolla. Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading. Psychological review, 92(2):173, 1985.
Stephen Grossberg and James R. Williamson. A Neural Model of how Horizontal and Interlaminar Connections of Visual Cortex Develop into Adult Circuits that Carry Out Perceptual Grouping and Learning. Cerebral Cortex, 11(1):37-58, 01 2001. ISSN 1047-3211. 1. doi: 10.1093/cercor/1 1.1137.. URL httpss://oo..ozgg///// wtt.://1//////....11111..
Jianzhong He, Shiliang Zhang, Ming Yang, Yanhu Shan, and Tiejun Huang. Bi-directional cascade network for perceptual edge detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3828-38888881 0028-3837,22 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Janis K Hesse and Doris Y Tsao, Consistency of border-ownership cells across artificial stimuli, natural stimuli, and stimuli with ambiguous contours. Journal of Neuroscience, 36(44): 1338- 11349, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7111, 2018.
David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215-243, 1968.
Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Ba- jwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, Richard C. Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexan- der Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Lc, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adri- ana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In- datacenter performance analysis of tensor processing unit. CoRR, abs/l704.04477 704.04760, 2017. URL abs
http:://axiiz http:////////////e/./
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits critical to the ventral streams execution of core object recognition behavior. Nature neuroscience, pp. 1, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James Di- Carlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, pp. 408385, 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(1 11:22782222222 1998.
10

---

Under review as a conference paper at ICLR 2020
Wu Li. Valentin Piëch, and Charles D Gilbert. Contour saliency in primary visual cortex. Neuron, 50(6):951-96) 2006.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in Neural Information Processing Systems, pp. 152-164, 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431 1-4400, 2015.
MJ Morgan, J Ross, and A Hayes. The relative importance of local phase and local amplitude in patchwise image reconstruction. Biological Cybernetics, 65(2):113-19,, 1991.
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel L. Yamins. Task-driven convolutional recurrent models of the visual system. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 5290-5301. Curran Associates, Inc., 2018. URL httt:://papers.nips.ccc///e
pdf
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision research, 37(23):331-33255 1997.
Alan V Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5):529-55) 541, 1981.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Jonathan Pillow. Likelihood-basee approaches to modeling the neural code. Bayesian brain: Prob- abilistic approaches to neural coding, pp. 53-70, 2007.
Leon N Piotrowski and Fergus W Campbell. A demonstration of the visual importance and flexibility of peatial-frequency amplitude and phase. Perception, 113):33773346 1982.
Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex.
Nature neuroscience, 2(1 1019, 1999.
Odelia Schwartz, Jonathan W Pillow. Nicole C Rust, and Eero P Simoncelli. Spike-triggered neural characterization. Journal of vision, 6(4):13-13, 2006.
Thomas Serre, Aude Oliva, and Tomaso Poggio. A feedforward architecture accounts for rapid categorization. Proceedings of the national academy of sciences, 104(15):6424-642) 2007a.
Thomas Serre, Lior Wolf, Stanley Bileschi, Maximilian Riesenhuber, and Tomaso Poggio. Ro- bust object recognition with like mechanisms. IEEE Transactions Pattern Analysis & Machine Intelligence, (3):411-422 2007b.
Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. Deepcontour: A deep con- volutional feature learned by positive-sharing loss for contour detection. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3982-3991, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Thomas Sullivan and Virginia Sa. A model of surround suppression through cortical feed- back. Neural networks, 1955:5664-572. 2006.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794- 7803, 2018
11

---

Under review as a conference paper at ICLR 2020
Saining Xie and Zhuowen Tu. Holistically edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, and Ming- Hsuan Yang. Object contour detec- tion with fully convolutional mncoder-decoder network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 193-202, 2016.
Amir Zamir, Te-Lin Wu, Lin Sun, William Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1308-1317, 2017.
A IMPLEMENTATION DETAILS
A.1 ABLATION STUDY ON BSDS500
The architecture we use to develop models for our experiments fits into a 3 layer hierarchical ar- chitecture that we parallel with the early visual circuit from Retina to V1. While refer to the different layers in our model with early visual areas, the retina and LGN in our experiments are not designed based the working of their biological counterparts.
Soundiry
Boundayy
Bou
Boundday
depac
Datis
Setra
Koboe
ConUL5TM Medra
@d Largsrr
Vihie
Figure 8: Architectural details from the ablation study on BSDS500
All the above four models have x 3 -Dccovoluuio layers in their Retina and 5x5 2-D convolution layers in their LGN. Each model uses a different number of filters in their retina and LGN ranging from 32 to 96 depending on the number of parameters in their VI layer.
Our ConvLSTM models are implemented using TensorFlow's built in ConvLSTM2D layer, ConVLSTM-MedRR (Fig Bl) utilizes 8x8 horizontal connection kernels, while ConvL.STM- LargeRF 8)b utilizes 15x15 convolution kernels. ViNet-Linear (Fig c) and VINet use 15 2-D convolutions for their horizontal excitation and 7x7 2-D convolutions for their hori- zontal inhibition. The spatial dimensions are chosen to be larger for excitatory connections than for inhibitory connections facilitate short-range inhibitory and long-range excitatory connections. In addition, the complete V1Net model (Fig d) employes another set of x 5 2-D convolutions for the shunting horizontal connections.
Similar to Zamir et al. (201)), we set the recurrent input X, to the feedforward convolutional feature representation from the bottom layer. For eg., when a VT block plugged between conv-2 and conv-3, the activation map of conv-2 is used to set X,, V [0,T-11]
1]. 1].
We use the TensorFlow framework to implement all our models in this paper. We found t=6 timesteps to be suitable for our VINet and ConvLSTM models to produce accurate object bound- aries while also fitting in our device's 's memory. We are currently evaluating whether a larger number of timesteps than 6 is useful to improve performance further using higher end training resources. Our hyperparameter choices include: minibatch size (6), learning rate (Ie-4), optimization algo- rithm (rmsprop), learning rate decay (divide by 10 after every 1000 iterations). For the BSDS500
12

---

Under review as a conference paper at ICLR 2020
experiments with data augmentation (last row in T used 5 different data augmentation tech- niques to expand the BSDS training set. We utilized random brightness variation, random contrast variation, random rotation between - /4 to π /4, horizontal flipping and vertical flipping. All these augmentation techniques were also applied to our Texturized challenge experiments. Our models trained single NVIDIA 2080 GPU.
A.2 TEXTURIZED CHALLENGE ARCHITECTURE DETAILS
The Retina in all three models had 2 3x3 convolution layers, and the L.GN utilized 2 3x3 convo- lutions followed by max pooling layer of size 2 with stride Since CIFAR-10 1- images are much smaller in size than BSDS500 (10x smaller), reduced the size of our convolution kernels in the last layer, VINet to 6x6 for Weze and 4x4 for Winh and W Whnmnt respectively. For our control ConvLSTM model, we set the size of the convolution kernels in the last layer to 6x6.
A.3 MORE SAMPLE PREDICTIONS VINET ON BSDS500 TEST SET
Attached below is a set of boundary predictions from VNNt on the BSDS500 test set.
BARL
Figure 9: Sample boundary predictions from a VINet model from the BSDS500 test set
13


Reviews:
[{"review_id": "SylfsS40Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes, inspired by the Primates brain, to add horizontal inhibitory and excitatory connections. In practice, the work proposes a variant of convolutional LSTM cells, that incorporates additional convolutions. \n\nOverall, the paper is hard to read, the experimental setting does not fully convince.  It seems hard to reproduce the results using the information given in the paper. A lot of the method section focuses on intuition, using vague vocabulary which makes it hard to understand concretely what is done in practice. In particular, the contributions are not clear enough. The difference between the proposed approach and existing works needs to be made clearer. \nThe idea is interesting, however, and the paper would benefit greatly from addressing those issues. \n\n\n\nMain points\n\n\nThe claims seem somewhat bigger than the actual contributions. In practice, the contribution is a modification to an LSTM cell.\n\nThe paper is not easy to read, and mixes various terms without introducing them (e.g presynaptic activity is used to introduce the method but never introduced, not even in related work). It would be good to use standard notation and math font (e.g. small bold letter for vectors, etc) and to define notations. In Figure 1, the kronecker symbold represents convolutions but in (1), the hadamard product (*) represents convolutions. These inconsistensies make the paper harder to follow.\n\nThe problem of clarity extends to the experimental section, where the experiments are not clearly explained, \n\nThe method section could be more detailed, in particular, a mathematical comparison of LSTM vs the proposed approachh would be useful. Figure 1 is unclear: what do the red, dark blue and light blue lines correspond to mathematically?\n\nThe related attempts in ML and Deep learning should be reviewed ( e.g. Lateral Inhibition-Inspired Convolutional Neural Networkfor Visual Attention and Saliency Detection, AAAI 2018).\n\nThe experimental setting is not convincing. It seems that a simple state-of-the-art CNN architecture would do better than the proposed approaches. Comparing a ConvLSTM with a single convolution does not seem fair.\n\nVery little implementation details are given. The authors mention a ConvLSTM2D layer is used. However, the inputs are static images: how exactly was the experiment done? Where does the time come from since the dataset considered is static?\n\nAbout emergent neurally plausible horizontal connections: that section is interesting but would benefit from being more detailed and rigorous. There is no actual measure or study of the emergence of said connections.\n\n3.2 is misleading as it corresponds to a setting that is never used in the experimental setting. \n\nMost of the results are qualitative, except for Table 1. It would be useful to have quantitative comparisons on established benchmarks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgoWw8U9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I first want to thank the authors for their proposed approach in this paper. Authors made an attempt to bridge the gap between natural (primate) vision and NN models. The paper is easy to read and understand. The authors proposed a Conv-LSTM-inspired model called V1Net. The model shows some merits in detecting the correct labels for noised inputs. \n\nUnfortunately, the paper lacks some critical analysis, and V1Net usability is limited in real-life. Commonly, the community expects a certain number of experiments to back a claim. Specifically I have the following questions:\n\n1. “V1Net can be flexibly incorporated as a module in existing implementations of DCNs”, can you elaborate how? Current architecture rarely consider using conv-lstm to solve tasks such as object detection. Not saying the current trend is correct or incorrect in doing so, but lack of experiments and details leave this claim unsupported. \n2. The leap between horizontal connectivity and V1Net is rather unmotivated. Specifically, authors should explain why V1Net is the only (or the most suitable) way of implementing horizontal connectivity. \n3. Why is the FFT texturization the correct way to evaluate the robustness? Specifically, could it be that V1Net being a simpler model (such as the case in Fig 4, where the performance is lower than conv-lstm for clean data) simply generalizes better to noisy input. Simpler models sometimes have the tendency to remove noise better. \n4. Necessary comparisons are not made with at least a few state of the art models in CIFAR. As it stands the impact is limited on community who uses conv-lstm for CIFAR classification.\n5. Figure 4 lacks the required standards of a scientific article figure. Borders on only 3 sides, and DPI seems to be low as text is pixelized. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gvZbERKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to modify a convolutional variant of LSTM (ConvLSTM) to include horizontal connections inspired by known interactions in visual cortex: excitation, subtractive inhibition (linear) and shunting multiplicative gating (nonlinear). They evaluate their V1Net model on a new task they call the texturized challenge (spectrally perturbed CIFAR-10 images) and on contour segmentation on BSDS500 and show that their approach outperforms some baselines.\n\nStrengths:\n+ The biological motivation is quite clear\n+ Architecture is simpler than that of previous related work (hGRU)\n\nWeaknesses:\n- Not clear what the objectives/contributions are\n- No advancement of state of the art in computer vision\n- No novel insights about brain function\n- Motivation of the \"texturized challenge\" is unclear\n- Performance on BSDS500 is far from state of the art\n- Value of the qualitative analysis on stylized ImageNet is unclear\n\nOverall, I'm not sure what the goal of this paper is. It neither presents an advance of the state of the art in any computer vision problem nor does it lead to any novel insights about the brain. The lack of a clear statement about the contributions of the paper seems to confirm this impression – the authors don't seem to know either.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a neurally inspired model that is a variant of conv-LSTM called V1net. The reviewers had trouble gleaning the main contributions of the work. Given that it is hard to obtain state of art results in neurally inspired architectures, the bar is much higher to demonstrate that there is value in pursuing these architectures. There are not enough convincing results in the paper to show this. I recommend rejection.
Decision: Reject