Recurrent neural circuits for contour detection
Authors: Drew Linsley*
Junkyung Kim*
Alekh Ashok
Thomas Serre
Venue:  (2020)
Keywords: Contextual illusions
visual cortex
recurrent feedback
neural circuits

Abstract: We introduce a deep recurrent neural network architecture that approximates visual cortical circuits (Mély et al., 2018). We show that this architecture, which we refer to as the 𝜸-net, learns to solve contour detection tasks with better sample efficiency than state-of-the-art feedforward networks, while also exhibiting a classic perceptual illusion, known as the orientation-tilt illusion. Correcting this illusion significantly reduces \gnetw contour detection accuracy by driving it to prefer low-level edges over high-level object boundary contours. Overall, our study suggests that the orientation-tilt illusion is a byproduct of neural circuits that help biological visual systems achieve robust and efficient contour detection, and that incorporating these circuits in artificial neural networks can improve computer vision.

Published as a conference paper at ICLR 2020
RECURRENT NEURAL CIRCUITS FOR CONTOUR DETECTION
Drew Linsleyt, Junkyung Kimif, Alekh Ashok & Thomas Serre Department of Cognitive, Linguistic and Psychological Sciences
Brown University
Providence, RI 02912, USA
(drew_linsley al alekh_ _askk thomas_seeeeeeeeee _eeree] @brown. edu junkyung@google.com
ABSTRACT
We introduce deep recurrent neural network architecture that approximates known visual cortical circuits (Mély et al. 2018). We show that this architecture, which refer the y-Net, learns solve contour detection tasks with better sample efficiency than state-of-the-ar feedforward networks, while also exhibiting a classic perceptual illusion, known the orientation-tilt illusion. Correcting this illusion significantly reduces 7-Net contour detection accuracy by driving it to prefer low- level edges over high-level object boundary contours. Overall, our study suggests that the orientation- tilt illusion is a byproduct of neural circuits that help biological visual systems achieve robust and efficient contour detection, and that incorporating such circuits in artificial neural networks can improve computer vision.
INTRODUCTION
An open debate since the inception of vision science concerns why experience visual illusions. Consider the class of "contextual" illusions, where the perceived qualities of an image region, such its orientation color, biased by the qualities of surrounding image regions. A well-studied contextual illusion is the orientation-tii illusion depicted in Fig. Th, where perception of the central grating's orientation influenced by the orientation of the surrounding grating To'le Wenderoth 1977). When the two orientations are similar, the central grating appears tilted slightly away from the surround (Fig. [.. top). When the two orientations are dissimilar, the central grating appears tilted slightly towards the surround (Fig. a bottom). Is the contextual bias of the orientation-till illusion bug of biology or byproduct of optimized neural computations?
Over the past 50 years, there has been number of neural circuit mechanisms proposed to explain individual contextual illusions (reviewed in Mély et al., 2018). Recently, Mêly et al. (2018) proposed cortical circuit, constrained by physiology of primate visual cortex (VI), that offers unified explanation for contextual illusions across visual domains from the orientation- illusion to color induction. These illusions arise in the circuit from recurrent interactions between neural populations with receptive fields that tile visual space, leading to contextual (center/surround) effects. For the orientation-tilt illusion, neural populations encoding the surrounding grating can either suppress or facilitate the activity of neural populations encoding the central grating, leading to repulsion vs. attraction, respectively. These surround neural populations compete influence encodings of the central grating: suppression predominates when center/surround are similar, and facilitation predominates when center/surround are dissimilar.
The neural circuit of Mély et al. (2018) explains how contextual illusions might emerge, but it does not explain why. One possibility is that contextual illusions like the orientation-till illusion are "bugs": vestiges of evolution or biological constraints on the neural hardware. Another possibility is that contextual illusions are the by-product of efficient neural routines for scene segmentation (Keemink) & van Rossum 2016 Mély et al. 2018). Here, we provide computational evidence for the latter
"These authors contributed equally to this work.
CCrrretly at DeepMind, London, UK.

---

Published as a conference paper at ICLR 2020
possibility and demonstrate that the orientation-tie illusion reflects neural strategies optimized for object contour detection.
Contributions We introduce the y-Net, trainable and hierarchical extension of the neural circuit Mfly et al. (2018), which explains contextual illusions. (i) The i-Net more sample efficient than thee-oof-hhe-rr convolutional architectures on two separate contour detection tasks. (ii) Similar to humans but not state-of-the-art contour detection models, the >-Net exhibits an orientation- illusion after being optimized for contour detection. This illusion emerges from its preference for high-level object-boundary contours over low-level edges, indicating that neural circuits involved in contextual illusions also support sample-efficiett solutions to contour detection tasks.
RELATED WORK
Modeling the visual system Convolutional neural networks (CNNs) are often considered the de facto "standard model" of vision. CNNs and their extensions represent the state of the art for most computer vision applications with performance approaching and sometimes exceeding human
observers on certain visual recognition tasks (He et al. 2016: Lee et al.) 2017 Phillips et al. 2018)
CNNs also provide the best fit to rapid neural responses in the visual cortex (sce Kriegeskorte 2015
Yamins & DiCarlo 2016 for reviews). Nevertheless, multiple lines of evidence suggest that biological
vision is still far more robust and versatile than CNNs (see Seeree 2019, for recent review). CNNs
suffer from occlusions and clutter (Fyall et al.] 2017 Rosenfeld et al. 2018 Tang et al.) 2018). They are also sample inefficient learning visual relations (Kim 2018 and solving simple grouping
tasks (Linsley et al. 2018c). State-of-the-art CNNs require massive datasets to reach their impressive accuracy (Lake al. 20155) and their ability to generalize beyond training data is limited (Geirhos
et al. 2018 Recht et 20118)
Cortical feedback contributes the robustness of biological vision (Hochstein & Ahissar 2002 Wyatte et al., [2114] Kataligonul et al. 2015). Feedforward projections in the visual system are almost
always matched by feedback projections (Felleman & Van Essen 1991). and feedback has been implicated in visual "routines" that cannot be implemented through purely feedforward vision, such as incremental grouping or filling-in (O'Reilly et al.) 2013; Roelfsema (0066.. There is a also a growing body of work demonstrating the potential of recurrent neural networks (RNNs) to account
for neural recordings (Fyall et al. 2017: Klink et al.
2017 Siegel et al., 2015: Tang et al. 2018
Nayebi et al. 2018 Kar al. 2019 Kietzmann et al.
2019
Feedback for computer vision In contrast to CNNs. which build processing depth through a cascade of filtering and pooling stages with unique weights, RNNs process stimuli with filtering stages that reuse weights over "timesteps" of recurrence. On each discrete processing timestep, an RNN updates its hidden state through nonlinear combination of an input and its the hidden state from its previous timestep. RNNs have been extended from their roots in sequence processing (e.g., Mozer 1992) to computer vision by computing the activity of RNN units through convolutional kernels. The common interpretation of these convolutional- (fixed) feedforward drive, which is combined with layer-
is that the input to each layer functions as feedback from an evolving hidden
state to dynamically adjust layer activity (Linsley et al.) 2018c. George et al.] 2017: Lotter et al.
2016 Wen al. 2018: Liao & Poggio 2016 [Spoerer et al. 2017] Nayebi et al. a 2018 Tang et al.
2018 In the current work, we are motivated by a similar convolutional-RNN
the horizontal gated
recurrent unit (hGRU, JinRly et 20012)),)), which approximates the recurrent neural circuit model of
(Mély et al. 2018) for explaining contextual illusions.
3 RECURRENT NEURAL MODELS
We begin by reviewing the dynamical neural circuit of Mély et al. (2018). This model explains contextual illusions by simulating interactions between cortical hypercolumns tiling the visual field (where hypercolumns describe a set of neurons encoding features for multiple visual domains at a single retinotopic position). In the model, hypercolumns are indexed by their 2D coordinate (x, y) and feature channels k. Units in hypercolumns encode idcalized responses for a visual domain (e.g., neural responses from the orientation domain were used simulate orientation-ti illusion;
2

---

Published as a conference paper at ICLR 2020
f+2
To del
From fef
column|
22.5
Comy- resiz/
Hinwys
H
(GRUM (Horizontal)
Surround pathways (f and 7D)
(Top-down))
Hisk
HyS
Conwit
(a)
(b)
Canter pathway (PF)
(c)
Suppressio
Input
Output
Figure The orientation tilt-illusion (O'Toole & Wenderoth, 1977) is a contextual illusion where central grating's perceived orientation is influenced by surround grating's orienta- tion. (a) When a central grating has a similar orientation as its surround, is judged as tilting away from the surround (repulsion). When the two gratings have dissimilar orientations, the central grating is judged as tilting towards the surround (attraction).
(b) We extend the recurrent circuit proposed by MMll al. (2018) to explain this and other contextual illusions into hierarchical model that learns horizontal (within a layer) and top-down (between layer) interactions between units. The
zyle) interactions between units in layer L, which receives feedforward drive from center pathway encoding feature k (e.g., edges oriented at 0 or 22.59) at position (x, y) in an image. Blocks depict different layers, and arrowed connections denote top-down feedback. (c) deep network schematic of the circuit diagram in (b), which forms the basis of the 7-Net introduced here. Horizontal and top-down connections are implemented with feedback gated recurrent units (fGRUs). Image encodings pass through these blocks on every timestep, from bottom-up (left path) to top-down (right path), and predictions are read out from the fGRU closest to image resolution on the final timestep. This motif can be stacked to create a hierarchical model.
circuit simulates dynamical suppressive (H (H)))) and facilitative (H (H)))
Dynamics of single unit zyk obey the following equations (we bold activity tensors to distinguish them from learned kernels and parameters):
Fig. [T).
( zyke zyk
(H)
Zzyk - (aH((F)
oyk + 1)) ((() ayk
Stage Recurrent suppression of Z
7 THI + o² (F)
# Stage 2: Recurrent facilitation of H (()
ayyk
where
((S) (W5 * (H(F)
Dayk
Compute suppression interactions
Tyk
((F) zyk (WF= * H Dayk-
# Compute facilitation interactions
Circuit activities consist of a feedforward drive, recurrent suppression, and recurrent facilitation, respectively denoted as Z, (9) (F)) e XXXXXX (X is width, Y is height of the tensor, and K is its feature channels [ The circuit takes its "feedforrward" input Z from hypercolumns (c.g., orientation encodings from hypercolumn units). and introduces recurrent suppressive and facilitatory interactions between units, (S), ((F) RXXXXX (Fig. [p).. These interactions are implemented with separate kernels for suppression and facilitation, W W E B where E is the spatial extent of connections on single timestep (connectivity this model is constrained by primate physiology).
"Suppression refers to interactions that reduce unit activity, and facilitation refers interactions that increase activity. These computations cause illusory repulsion attraction the level of neural populations.
3

---

Published as a conference paper at ICLR 2020
These interactions are implemented through convolutions, allowing them to serially spread over timesteps of processing to connect units positioned at different spatial locations. The circuit outputs (FF) after reaching steady state.
The circuit model of Mély et al. (2018) depends on competition between H (S) and H (F) to explain
the orientation-bl illusion. Competition is implemented by (i) computing suppression vs. facilitation in separate stages, and (ii) having non-negative activities, which enforces these functionally distinct processing stages. With these constraints in the circuit model, the strength of recurrent suppression but not facilitation multiplicatively increases with the net recurrent output. For the orientation tilt illusion, suppression predominates when center and surround gratings have similar orientations. This causes encodings of the surround grating to "repulse" encodings of the center grating. On the other hand, facilitation predominates (causing "attraction") ") when center and surround gratings have dissimilar orientations because it is additive and not directly scaled by the circuit output.
and patterns of horizontal connections between units tuned by hand. Linear and multiplicative suppression (i.e., shunting inhibition) are controlled by scalars j and a, feedforward drive is modulated by the scalar ç. and linear facilitation controlled by the scalar v. Circuit time constants are scalars denoted by 1). e, and Ø. All activities are non-negative and both stages are linearly rectified (ReLU) [+] max(- 0).
Parameters controlling the circuit's 's integration, suppression/faciitaaaii
Feedback gated recurrent units Linsley et al. (0188a) developed a version of this circuit for computer vision applications, called the hGRU. In their formulation they use gradient descent (rather than hand-tuning like in the original circuit) to fit its connectivity and parameters to image datasets. The hGRU was designed to learn a difficult synthetic incremental grouping task, and single layer of the hGRU learned long-range spatial dependencies that CNNs with orders-of- -farnnitu- more weights could not. The hGRU replaced the circuit's 's time constants with dynamic gates, converted the recurrent state H for suppression into instantaneous activity, and introduced term for quadratic facilitation. The hGRU also relaxed biological constraints from the original circuit, including an assumption of non-negativity, which enforced competition between recurrent suppression vs. facilitation (e.g., guaranteeing that Stage in the circuit model describes suppression of Zzyk).
We extend the hGRU formulation in two important ways. First, like Mély et al. (0188). we introduce non-negativity. This constraint was critical for Mély et al. (2018) to explain contextual illusions, and as we describe below, was also important for our model. Second, we extend the circuit into a hierarchical model which can learn complex contour detection tasks. Recent neurophysiological work indicates that contextual effects emerge from both horizontal and top-down feedback (Chettih) & Harvey 2019). Motivated by this, we develop versions of the circuit to simulate horizontal connections between units within a layer, and top-down connections between units in different layers.
We call our module the feedback gated recurrent unit (fGRU). We describe the evolution of fGRU recurrent units in H E RXXXXX which are influenced by non non-nege non-negative feedforward encodings Z X (c.g., a convolutional layer's response stimulus) over discrete timesteps -[t]:
Stage 1:
S SU = *H(t - 1])
Compute channel-wise selection
C S (H(-- 1 1] Go)
# Compute suppression interactions
S = z - (aHit - 1]
# Suppression of Z
Stage 2:
G = SU +S)
Compute channel-wise recurrent updates
F = F. #
# Compute facilitation interactions
H = (u(Ct +S) w S)
# Facilitation of S
Hit = (1 - GG
② Hlt - 1| + G F
+ H.
Update recurrent state
4

---

Published as a conference paper at ICLR 2020
Like the original circuit, the fGRU has separate stages for suppression (S) and facilitation (H), In the first stage, the feedforward encodings Z are suppressed by non-negative interactions between units in H[t - 1] (an fGRU hidden state from the previous timestep). Suppressive interactions are computed with the kernel W E TR. ER where E describes the spatial extent of horizontal connections on single timestep. This kernel is convolved with a gated version of the persistent hidden state H[(- The gate activity Go computed by applying a sigmoid nonlinearity a convolution of the kernel U E IR fxx1xxxX with H[t - IJ. which transforms its activity into the range [0, 1]. Additive and multiplicative forms of suppression are controlled by the parameters Je, a E R respectively.
In the second stage, additive and multiplicative facilitation is applied to the instantaneous activity S. The kernels W E IR Ex controls facilitation interactions. Additive and multiplicative forms of facilitation are scaled by the parameters V, w E R respectively. A gate activity is also computed during this stage to update the persistent recurrett activity H. The gate activity Gf is computed by applying sigmoid
convolution of the kerne U E R x with S. This gate updates H[t] by interpolating H(t- - 1] with the candidate activity H. After every timestep of processing. H[t] is taken the IGRU output activity, As detailed in the following section, the IGRU output hidden state is either passed to the next convolutional layer (Fig. Dc, FGRU + cony D), or used to compute top-down connections (Fig. D. fGRU(++1) - OG
The fGRU has different configurations for learning horizontal connections between units within layer or top-down connections between layers (Fig. ()) These two configurations stem from changing the activities used for a IGRU's feedforward encodings and recurrent hidden state. "Horizontal connections" between units within a layer are learned by setting the feedforward encodings Z to the activity of a preceding convolutional layer, and setting the hidden state H a persistent activity initialized as zeros (Fig. [c. (f) + (GRU())) "Top-down connections" between layers are learned by setting fGRU feedforward encodings to the persistent hidden state H( of fGRU at layer in a hierarchical model, and the hidden state H to the persistent activity H(f-1) of an fGRU at a layer one level up in the model hierarchy (Fig. D FGRU(++1) -> fGRU((). The functional interpretation of the top-down fGRU is that it first suppresses activity in the lower layer using the higher layer's recurrent horizontal activities, and then applies a kernel to the residue for facilitation, which allows for computations like interpolation, sharpening, or "filling in". Note that an fGRU for top-down connections does not have a persistent state (it mixes high and low-level persistent states), but an fGRU for horizontal connections does.
y-Net Our main objective is test how model with the capacity for contextual illusions performs on natural image analysis. We do this by incorporating fGRUs into leading feedforward architectures for contour detection tasks, augmenting their feedforward processing with modules for learning feedback from horizontal and top-down connections (Fig. D We refer to the resulting hierarchical models as -ettts, because information flows in loop that resembles y: image encodings make a full bottom-up to top-down cycle through the architecture on every timestep, until dense predictions are read-out from the lowest-level recurrent layer of the network (thus, information flows in the top of the hierarchy, loops through the network, and flows out from the top of the hierarchy). In our experiments we convert leading architectures for two contour detection problems into Neeets: A VGGI6 for BSDS500 (He et al.) 2019), and a U-Net for detection of cell membranes in serial electron microscopy images (Lee et al. 2017). See Appendix A for an algorithmic description of neett
4 CONTOUR DETECTION EXPERIMENTS
Overview We evaluated -eett performance on two contour detection tasks: object contour detection in natural images (BSDS500 dataset: Arbeláez et al. 2011) and cell membrane detection in serial electron microscopy (SEM) images of mouse cortex (Kasthuri et al. (2015) and mouse retina (Ding c al. 2016). Different 7-eet configurations were used on each task, with each building on the leading architecture for their respective datasets. All 7-Nets use 8-timesteps of recurrence and instance normalization (normalization controls vanishing gradients in RNN training: Ulyanov et al. 2016 Cooijmans al. 2017| see Appendix aapendix[A]oor for details). The Nete were trained with Tensorflow and NVIDIA Titan RTX GPUs using single-imag batches and the Adam optimizer (Kingma & Ba 2014 dataset-specifie learning rates are detailed below). Models were trained with early stopping, which
5

---

Published as a conference paper at ICLR 2020
Humans
Y-Net
Y-Net
Nert (Durs)
60
BDCN
BDCN
BDCN (ine et al, 2010)
A+100%
5%
10% 100%
(b) Label
5%
10%
100%
Label
5%
10%
100%
Proportion training data used
(c)
Image
Prediction
Label
Timestop
Figure 2: Object contour detection in BSDS500 images. (a) The y-Net par with humans and the statc-of-the-an art for contour detection (BDCN; He et 2019) when trained on the entire training dataset with augmentations. approaches to BSDS500 (LPCB: Deng et al. 2018 RCF: Liu et al. 2019 CED: Wang et al. 2019 DB: Kokkinos 2015] HED: Xie & Tu 2017, and OEF: Hallman Fowlkes 2015). The 7eNet outperforms the BDCN when trained on 5%, 10%, or 100% of the dataset. Performance is reported as F1 ODS (Arbeláez et al. 201 T). (b) BDCN and 7-Net predictions after training on the different proportions of BSDS500 images. (c) The evolution of 7-Nett predictions across timesteps of processing. Predictions from a Nett trained on 100% BSDS are depicted: its initially coarse detections are refined over processing timesteps to select figural object contours.
this regime, it also outperforms the published FI ODS all other
terminated training if the validation loss did not drop for 50 straight epochs. The weights with the best validation-set performance were used for testing.
Model evaluation We evaluated models in two ways. First, we validated them against state-of-the- art models for each contour dataset using standard benchmarks. As discussed below, we verified that implementations of these state-of-the thefthheartt art models matched published performance. Second, we tested sample-efficiency after training on subsets of the contour datasets without augmentations. Sample-efficiene compares the inductive biases of different architectures, and is critical for under- standing how the capacity for exhibiting contextual illusions influences performance. We report model "wall time" in Appendix [A] however, the focus of our work than the hardware/ssffwwrreeeww optimizations that influence wall time. Model performance is evalu- ated as the F-measure the Optimal Dataset Scale across images after non-maximum suppression post-pa post-processing (F1-ODS: Arbeláez et al. 2011), as is standard for contour detection tasks.
sample efficiency rather
4.1 ОВJЕCТ CONTOUR DETECTION NATURAL IMAGES
Dataset We trained models for object contour detection on the BSDS500 dataset (Arbeláez et al.) 2011).
The dataset contains object-contour annotations for 500 natural images, which are split into train (200), validation (100), and test (200) sets.
Architecture details The leading approach to BSDS500 the Bi-Directional Cascade Network (BDCN, He et al. 2019), which places multi-layer readout modules every processing block in

---

Published as a conference paper at ICLR 2020
1.00
SNEMI3D
Ding et al,, 2016
SNEMI3D (Mouse cortexi
Ding al. 2016 (Mouso retina)
0.30
Test F1 ODS
0.80
0.70
U-Net
0.80
(a)
100 A+100
of training data used
100 A-100
(b) Label
100%
10%
100%
retiria
(c)
Image
3
Prediction
Label
Timestop
Figure 3: Membrane prediction serial electron mieroscopy (SEM) images of neural tissue.
(a) The NeNet outperforms state-of-the-att U-Net (Lee al. 2017) for membrane detection when trained on SEM image datasets mouse visual cortex (SNEMI3D) and retina (Ding et al. 2016). Performance is F1 ODS. (b) Network predictions after training different proportions of each dataset. (c) The evolution of y-Net prediction across timesteps of processing after training on 100% of the datasets. 7-Net learns iteratively suppress contours belonging internal cell features, such as organelles, which should not be annotated as contours for the purpose of neural tissue reconstruction.
I.SVRC12-pretrainee VGG16, and optimizes loss that balances contributions from each of these readouts to achieve better scale tolerance in final prediction. All leading deep learning approaches to BSDS500 begin with a VGG16 pretrained on ILSVRCI2 object recognition (He et al. 2019).
Our N-Net for BSDS500 begins with the same ILSVRCI2-pretraaine were introduced for learning horizontal (conv2_2, conv3_3, conv4_3, _3, conv5_3, 3) and top-down connections (conv5_3 conv4_3 conv4_ 3 conv3 3, and conv3 conv2 2). To pass top- down activities between layers, higher-level activities were resized to match lower-level ones, then passed through two layers of convolutions with linear rectification, which registered feature representations from nigher-to-lowee layers. The 7-Net was trained with learning rates of 3 on its randomly initialized (GRU weights and le on its VGG-initialized weights. Training time and parameter counts for this >-Net and the BDCN is in Appendix Table SI.
VGG16 as the BDCN. fGRUs
In contrast the BDCN (and other recent approaches to BSDS) with multiple read-outs and engi- neered loss functions, we take 7-Net predictions as linear transformation of the lowest fGRU layer in its feature hierarchy, and optimize the model with binary cross entropy between per-pixel predictions and labels (following the method of Xie & Tu 2017). This approach works because the 7-Net uses feedback merge high- and low-level image feature representations at the bottom of its feature hierarchy, resembling classic "V1 scratchpad" hypotheses for computation in visual cortex (Gilbert & Sigman, [007] Lee Mumford] (2003). We compared --Nets with a BDCN implementation released by the authors, which was trained using the routine described in He et al. 2019
'e replicated published results by following this routine and training the BDCN with the same regular- ization, batch size, and optimizer as He et al. (2019). For sample efficiency experiments, we searched through multiple learning rates and found that had affect performance these small BSDS500 datasets (Fig. S3
7

---

Published as a conference paper at ICLR 2020
We validated the againns the BDCN after training full and augmented BSDS training set (Xie & Tu, 2017). The y-Net performed similarly in F1 ODS (0.802) as the BDCN (0.806) and humans (0.803), and outperformed all other approaches to BSDS (Fig. 2h Deng et al.
Results
2018 Xie & Tu 2017 Hallman & Fowlkes 2015
Kokkinos 2015
Wang et al. 2019 Liu et al. 2019).
Our hypothesis is that contextual illusions reflect routines for efficient scene analysis, and that the capacity for exhibiting such illusions improves model sample efficiency. Consistent with this hypothesis, the 7-Net was up to an order-of-magnitde more efficient than the BDCN. A 7-Net trained on 5% of BSDS performs on par with a BDCN trained on 10% of the BSDS, and 7-Net trained on 10% of the BSDS performs on par with BDCN trained on 100% of BSDS. Unlike the BDCN, the Y-Net trained on 100% of BSDS outperformed the state of the art for non-deep learning based models (Hallman & Fowlkes 2015), and nearly matched the performance of the popular HED trained with augmentations (Xie & Tu 2017). We also evaluated lesioned versions of y-Net measure connections, recurrence, non-negativity, and different specifications of its IGRU recurrent modules for detecting contours in BSDS500 (Fig. S4
the importance of its horizontal/top-dowe
We examined recurrent feedback strategies learned by 7-Net for object contour by visualizing its performance every timestep of processing. This was done by passing its activity at timestep through the final linear readout layer. The --Ne refines its initially coarse contour predictions. For example, the top of Fig. 2 shows that the 7-Net selectively enhances the boundaries around the runner's bodies while suppressing the feature activities created by the crowd. In the next row of predictions, salient zebra stripes are gradually suppressed in favor of body contours (see Fig. S5 for 7-Net prediction dynamics and its tendency towards steady state solutions).
4.2 CELL MEMBRANE DETECTION
Datasets "Connectomics" involves extracting the wiring diagrams of neurons from serial electron microscope (SEM) imaging data, and is an important step towards understanding the algorithms of brains (Briggman & Bock 2012). CNNs can automate this procedure by segmenting neuron membranes in SEM images. Large-scale challenges like SNEMI3D (Kasthuri et al., 2015). which contains annotated images of mouse cortex, have helped drive progress towards automation. Here, we test models on membrane detection in SNEMI3D and separate SEM dataset of mouse retina ("Ding" from Ding et al. 2016). We split both datasets into training (80 images for SNEMI3D and 307 images for Ding) and test sets (20 images for SNEMI3D and 77 images for Ding). Next, generated versions of each training dataset with 100%, 10%, or 5% of the images, as well as versions of the full datasets augmented with random left-right and up-down flips (A+100%)
Architecture details The state-of-the-art on SNEMI3D is a U-Net variant (Ronneberger et al., 2013), which uses different depth, different number of feature maps at every layer, and introduces new features like residual connections (Lee et al. 2017). We developed a 7-Net for cell membrane segmentation, which resembled this U-Net variant Lee et al.] (2017) (see Appendix [or details). These NeNet were trained from random initialization with learning rate of 1e to minimize class-balanced per-pixel binary cross- cross-entropy, and compared to the U-Net from Lee et al. (2017). Training time and parameter counts for these models is in Appendix Table S2.
The 7-Net and U-Net of Lee et al. (011)) performed similarly when trained on full aug- mented versions of both the SNEMI3D and Ding datasets (Fig. 3 A+100%). However, 7-Nets were consistently more sample efficient than U-Nets on every reduced dataset condition (Fig. 3p).
Results
We visualized the recurrent membrane detection strategies of y-Nets trained on 100% of both datasets. Membrane predictions were obtained by passing neural activity at every timestep through the final linear readout. The -eet prediction timecourse indicates that it learns a complex visual strategy for membrane detection: it gathers a coarse "gist" of membranes in the first timestep of processing, and iteratively refines these predictions by enhancing cell boundaries and clearing out spurious contours of elements like cell organelles (Fig. 33).
5 BUGS OR BYPRODUCTS OF OPTIMIZED NEURAL COMPUTATIONS!
Orientation-till illusion Like the neural circuit of Mély et al, (2018), fGRU modules are designed with asymmetry in their ability to suppress and facilitate feedforward input (see Section 3. This
8

---

Published as a conference paper at ICLR 2020
Human Response
Y-Net (Ours)
BDCN (Ho et al., 2019)
(OTToole and Wenderoth, 1977)
40
Percelved shift in orientation
Models' percelved shift in orientation
Repulsion
30
20-
10-
Attraction
-10-
-20
20
40
60
80
20
60
80
Center-Surround orientation difference
Center-Surround orientation difference
Figure 4: Optimizing for contour detection produces an orientation-tilt illusion in the 7-Net. The orientation- illusion (O'Toole & Wenderoth 1977) describes how perception of the center grating's s orientation is repulsed from the surround when the two are in similar orientations (e.g., s 30%), and attracted to the surround when the two are dissimilar (but not orthogonal) orientations (e.g., x 60%). We test for the orientation-till illusion in models trained BSDS500 contour detection. Model weights were fixed and new layers were trained to decode the orientation of grating stimuli of single orientation. These models were tested on grating stimuli in which surround orientations were systematically varied w.r.t. the center (exemplars depicted in the left panel). The y-Net but not the BDCN had an orientation-tin -iltt illusion. Gray curves depict fourth-order polynomial fit.
potentially gives 7-Nets (which contain fGRU modules) the capacity to exhibit similar contextual illusions as humans. Here, we tested whether a 7-Net trained on contour detection natural images exhibits an orientation-tilt illusion.
We tested for this illusion by training orientation decoders on the outputs of models trained on the full BSDS500 dataset. These decoders were trained on 100K grating images, in which the center and surround orientations were the same (Fig. S2a). These images were sampled from all orientations and spatial frequencies. The decoders had two I 1x1 convolution layers and intervening linear rectification to map model outputs into the sine and cosine of grating orientation. Both the NeNet and BDCN achieved nearly perfect performance
held-out validation set of gratings.
We tested these models on 1K grating stimuli generated with different center-surround grating orienta- tions (following the method of Toole & Wenderoth 1977 (Sp), and recorded model predictions Surprisingly, 7-Net encodings of these
for the center pixel in these images (detailed in Appendix test images exhibited similar tilt illusion as found in human perceptual data (Fig. 4p) There was repulsion when the central and surround gratings had similar orientations, and attraction when these gratings were dissimilar. This illusory phenomenon cannot be explained by accidental factors such as aliasing between the center and the surround, which would predict the opposite pattern, suggesting that the illusion emerges from the model's strategy for contour detection. which only relies on feedforward processing, did not exhibit the effect (Fig. 4p). We also tested lesioned 7-Nets for the orientation-til illusion, but only the full 7-Net and a version lesioned have only (spatially broad) horizontal connections replicated the entire illusion (thought this lesioned model performed worse on contour detection than the full model; Fig. (S4). These findings are consistent with the work of Mély al. (2018), who used spatially broad horizontal connections to model contextual illusions, as well as the more recent neurophysiological work Chetthh & Harvey (2019), who explained contextual effects through both horizontal and top-down interactions.
contrast, the BDCN,
Correcting the orientation-tin illusion What visual strategies does the orientation-till illusion reflect? We tested this question by taking 7-Net that was trained to decode central grating orientation in tilt-illusion, and then training further to decode the central grating orientation tilt- iiltiilusion stimuli (Fig. 5a. "illesion-correctee"" in red). Importantly, 7-Net weights training, but the orientation decoder was not. Thus, improving performance for decoding the orientation of these illusory stimuli comes at the expense of changing -ett weights that were
optimized during

---

Published as a conference paper at ICLR 2020
4= =.0.043 0.003 1=13.500,px0.00
Image
0.9
0.0-
Label Sarne
Busion-correctenn
0.7
0.6-
0.5
Finetuned on:
0.4
||
0.3
Domain-Iransfer control y-Net
Illuiio-corracted
Domain-transfer control F1 ODS
Feature preferred by:
y-Net
orientation-tilt illusion. (a) F1 ODS scores on BSDS500 test images (200 total) from y-Nets after correcting an orientation tilt illusion "'llusion-corrretedd" or not Aeoomain-transtee control"). The domain-transfer control
Figure 5: Contour detection performance of the y-Net depends
trained to decode the orientation of single-grating stimuli (blue), and the illusion-correcte y-Net was trained to decode the orientation of the central grating in illusory grating stimuli (red). Readouts for decoding orientation were fixed, and 7-Net weights were allowed to change during training. Per-image Fl ODS was significantly greater for The domain-transfer control 7-Net than the N-Net. (b) The maasiinnncorrected NeNet was biased towards low-level contours, whereas the comain-transien control Neee was biased towards contours on object boundaries.
7-Net was was
responsible for its orientation-tilt illusion. As a control, another 7-Net was trained with the same routine to decode the orientation of full-image gratings, for which there is no illusion (Fig. 5 "domain-transfer control" in blue; see Fig. 56 for training performance of both models). Both models were tested the BSDS500 test set.
Correcting the orientation-tilt illusion of 7NNe significantly hurts its object contour detection performance (Fig. 5]; I-sample T-test of the per-image ODS FI difference between models, T' (1999) = 13.570, < 0.001). The illusion reflects 7-Net strategies for selecting object-boundarie rather than low-level contours (Fig. 5b: Fig, S7 for examples).
CONCLUSION
Why do we experience visual illusions? Our experiments indicate that one representative contextual illusion, the orientation-tilt illusion, is consequence of neural strategies for efficient scene seg- mentation. We directly tested whether this contextual illusion is a bug or a byproduct of optimized neural computations using the y-net: dense prediction model with recurrent dynamics inspired by neural circuits in visual cortex. On separate contour detection tasks, the 7-Net performed on par with state-of-the-ar models when trained in typical regimes with full augmented datasets, but was far more efficient than these models when trained on sample-limited versions of the same datasets. At the same time, the "-Net exhibited an orientation- illusion which biased it towards high-level boject-boundary contours over low-level edges, and its performance was reduced when it was trained to correct its illusion.
While Nets are more sample efficient than leading feedforward models for contour detection, they also take much more "wall-time" train than these feedforward models. Learning algorithms and GPU optimizations for RNNs are lagging behind their feedforward counterparts in computational efficiency, raising the need for more efficient approaches for training hierarchical RNNs like 7eNett to unlock their full potential.
More generally, our work demonstrates novel synergy between artificial vision and vision neuro- science: we demonstrated that circuit-level insights from biology can improve the sample efficiency of deep learning models. The neural circuit that inspired the fGRU module explained biological illusions in color, motion, and depth processing (Mély et al., 2018), and suspect that Nett will have similar success learning sample-efficient strategies and exhibiting contextual illusions in these domains.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmen- tation. IEEE Trans. Pattern Anal. Mach. Intell., 33(5):898 916, May 2011.
K. L. Briggman and D. D. Bock. Volume electron microscopy for neuronal circuit reconstruction. Curr. Opin. Neurobiol., 22(1):154-161, February 2012.
S. N. Chettih and C. D. Harvey. Single-neuron perturbations reveal feature-specific competition in
V1. Nature, 567(7748):334-34) March 2019.
T. Cooijmans, N. Ballas, C. Laurent, Ç. Gülgehre, and A. Courville. Recurrent batch normalization.
In International Conference on Learning Representations, 2017.
R. Deng, C. Shen, S. Liu, H. Wang, and X. Liu. Learning to predict crisp boundaries. In Computer Vision ECCV 2018, pp. 570-586. Springer International Publishing. 2018.
H. Ding, R. G. Smith, A. Poleg-Pose Polsg,, J. Diamond, and K. L. Briggman. Species-speciffe wiring for direction selectivity in the mammalian retina. Nature, 535(7610):005-110, July 2016.
D. Felleman and D. C. Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex, 1(1):1-47, 1991.
A. M. Fyall, El-Shamayleh, H. Choi, Shea-Brown, and A. Pasupathy. Dynamic representation of partially occluded objects in primate prefrontal and visual cortex. Elife, 6, September 2017.
R. Geirhos, C. M. Temme, Rauber, H. Schüitt, M. Bethge, and F. A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7549-7561. Curran Associates, Inc., 2018.
D. George, W. Lehrach, K. Kansky, M. ázzaro-Gredilla, Laan, B. Marthi, X. Lou, Z. Meng, Liu, H. Wang. A. Lavin, and D. S. Phoenix. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science, 358(6368), December 2017.
C. D. Gilbert and M. Sigman. Brain states: top-down influences in sensory processing. Neuron, 54 (5)667766 June 2007.
R. H. Hahnloser, R. Sarpeshkar, M. a. Mahowald, R. J. Douglas, and H. S. Seung. Digital selection and analogue amplification coexist a cortex-inspired silicon circuit. Nature, 40566789):44-9511 June 2000.
S. Hallman and C. Fowlkes. Oriented edge forests for boundary detection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1732-1740, June 2015.
He, Zhang, M. Yang, Y. Shan, and T. Huang. Bi-Directional cascade network for perceptual edge detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
pp.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
J. C. Heck and F. M. Salem. Simplified minimal gated unit variations for recurrent neural networks. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. August 2017.
S. Hochstein and M. Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36(5):791-804 2002.
M. Januszewski and V. Jain. Eigmeentation-inhaneee CycleGAN. February 2019.
H. Kafaligonul, B. G. Breitmeyer, and H. Ögmen. Feedforward and feedback processes in vision. Front. Psychol., 6:279, March 2015.
11

---

Published as a conference paper at ICLR 2020
K. Kar, J. Kubilius, K. Schmidt, E. B. Issa, and J. J. DiCarlo. Evidence that recurrent circuits critical to the ventral stream's execution of core object recognition behavior. Nat. Neurosci. April 2019.
N. Kasthuri, K. J. Hayworth, D. Berger, R. L. Schalek, J. A. Conchello, S. Knowles-Barley D. Lee, A. Vázquez-Reinee Reina, V. Kaynig, T. R. Jones, M. Roberts, J. L. Morgan, J. C. Tapia, H. S. Seung, W. G. Roneal, J. T. Vogelstein, R. Burns, D. Sussman, C. Priebe, H. Pfister, and W. Lichtman. Saturated reconstruction of volume of neocortex. Cell, 162(():648-66)) (,,44,666,JJ0 July 2015.
S. W. Keemink and M. C. W. van Rossum. A unified account of tilt illusions, association fields, and contour detection based on elastica. Vision Res., 126:164-1111 16:-174177 173, September 2016.
T. C. Kietzmann, C. J. Spoerer, L. Sörensen, and others. Recurrence required to capture the dynamic computations of the human ventral visual stream. arXiv preprint arXiv, 2019.
Kim, M. Ricci, and T. Serre. NoL-SO-CLEVR: learning different relations strains feedforward neural networks. Interface Focus theme issue on "Understanding images in biological and computer vision". 2018.
D. P. Kingma J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
C. Klink, B. Dagnino, M.-A. Gariel-Mathis, and P. R. Roelfsema. Distinct feedforward and feedback effects of microstimulation in visual cortex reveal neural mechanisms of texture segregation. Neuron, June 2017.
Kokkinos. Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386. 2015.
N. Kriegeskorte. Deep neural networks: A new framework for modeling biological vision and brain information processing. Annu Rev Vis Sci, 1:417-446, November 2015.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 330(6266): 1338, December 2015.
K. Lee, J. Zung, P. Li, V. Jain, and H. Sebastian Scung. Superhuman accuracy on the SNEMI3D connectomics challenge. In Neural Information Processing Systems, 2017.
T. S. Lee and D. Mumford. Hierarchical bayesian inference in the visual cortex. J. Opt. Soc. Am. A Opt. Image Sci. Vis., 20(7):1434-1444, July 2003.
Q Liao and T. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. April 2016.
D. Linsley, K. Kim, Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Neural Information Processing Systems (NIPS), 2018a.
D. Linsley, J. Kim, D. Berson, and T. Serre. Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks. November 2018b.
D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependen- cies with horizontal gated recurrent units. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Gareett (eds.), Advances in Neural Information Processing Systems 31, pp. 152- 164. Curran Associates, Inc., 2018c.
N.
D. Linsley, D. Shiebler, Eberhardt, and Serre. Learning what and where to attend with humans in the loop. In International Conference on Learning Representations, 2019.
Y. Liu, M.-M. Cheng, X. Hu, J.-W. Bian, I.. Zhang, X. Bai, Jnd Tang. Richer convolutional features for edge detection. IEEE Trans, Pattern Anal. Mach. Intell., 41(8); 1939-196,,, August 2019.
W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. May 2016.
12

---

Published as a conference paper at ICLR 2020
D. A. Mély, D. Linsley, and T. Serre. Complementary surrounds explain diverse contextual phenomena across visual modalities. Psychol. Rev., 2018.
M. C. Mozer. Induction multiscale temporal structure. In J. Moody, S. J. Hanson, and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems 4, pp. 275-282- Morgan- Kaufmann, 1992.
A. Nayebi, D. Bear, J. Kubilius, K. Kar, S. Ganguli, D. Sussillo, J. DiCarlo, and D. L. Yamins. Task-Driven convolutional recurrent models of the visual system. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Informa- tion Processing Systems 31, pp. 5295-5306. Curran Associates, Inc., 2018.
Nunez-Iglesias, R. Kennedy, T. Parag, J. Shi, and B. Chklovskii. Machine learning of hierarchical clustering to segment 2D and 3D images. PLOS One, 8(8):771715, August 2013.
R. C. O' O'Reilly, D. Wyatte, S. Herd, B. Mingus, and D. J. Jilk. Recurrent processing during object recognition. Front. Psychol. 4(April):1-14, 2013.
B. O'Toole and Wenderoth. The tilt illusion: repulsion and attraction effects in the oblique meridian. Vision Res., 17(3):367-374, 1977.
P. J. Phillips, A. N. Yates, Y. Hu, C. Hahn, E. Noyes, K. Jackson, J. G. Cavazos, G. Jeckeln, R. Ranjan, S. Sankaranarayanan, J.-C. Chen, C. D. Castillo, Chellappa, D. White, and A. O'Toole. Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proc. Natl. Acad. Sci. U. S. A., 115(24):6111-6176, June 2018.
B. Recht, R. Roelofs, Schmidt, and Shankar. Do CIFAR-I0 classifiers generalize to CIFAR-10? June 2018.
P. R. Roelfsema. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci,, 29:203-2271 January 2006.
O. Ronneberger, Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asiitted Intervention MICCAI 2015, pp. 234-241. Springer International Publishing, 2015.
A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant in the room. August 2018.
T. Serre. Deep learning: The good, the bad, and the ugly. Annu Rev Vis Sei, 5:399-426, September 2019.
M. Siegel, T. Buschman, and Miller. Cortical information flow during flexible sensorimotor decisions. Science. 3486 June 2015.
C. J. Spoerer, P. McClure, and N. Kriegeskorte. Recurrent convolutional neural networks: A better model of biological object recognition. Front. Psychol., 8:1551, September 2017.
C. Tallee and Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018.
H. Tang, M. Schrimpf, W. Lotter, C. Moerman, A. Paredes, J. Ortega Caro, W. Hardesty, D. Cox, and G. Kreiman. Recurrent computations for visual pattern completion. Proc. Natl. Acad. Sei. U. S. A., 115(35):883-88400 August 2018.
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. July 2016.
E. Vorontsov, C. Trabelsi, S. Kadoury, and C. Pal. On orthogonality and learning recurrent networks with long term dependencies. January 2017.
Y. Wang, X. Zhao, Y. Li, and K. Huang. Deep crisp boundaries: From boundaries to Higher-Level tasks. IEEE Trans. Image Process., 28(3): 1285-1298, March 2019.
H. Wen, K. Han, J. Shi. Y. Zhang, E. Culurciello, and Z. Liu. Deep predictive coding network for object recognition. February 2018.
13

---

Published as a conference paper at ICLR 2020
D. Wyatte, D. J. Jilk, and R. C. O'Reilly. Early recurrent feedback facilitates visual object recognition under challenging conditions. Front. Psychol., 5:674, July 2014.
S. Xie and Z. Tu. Holistically-Nested edge detection. Int. J. Comput. Vis., 125(1-3):3-18, December 2017.
D. L.. Yamins and J. J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci., 1963)3356-365, February 2016.
14

---

Published as a conference paper at ICLR 2020
A y-NET
Algorithm A generic 7-Netarchitectture See Section 3 in the main text for the treatment.
Require: Image batch I
1: Z 6-1
2: H' <
3: for tort=To
> Recurrent loops over static images
4:
for f= = to do
Bottom-up pass
5:
Z' ReLU(Convv('))
D Feedforward activity
6:
11
(Z H- 1))
Update horizontal feedback state
7:
if L < I. then
8:
- Maxpool(H*())
9:
for = L to do
Top-down feedback pass
10:
if f < L then
11:
[4 IGRUTD(Z-H CCnneetCCoonCUsaaaan (4)))
12:
return H
Hidden state of layer l
The current standard for computer vision applications in connectomics is to train and test separate partitions of the same tissue volume (Linsley et al. 20186 Januszewski & Jain 2019). This makes it difficult to develop new model architectures without overtitting to any particular dataset. For this reason, we first tuned our connectomics 7-Net and its hyperparameters synthetic dataset of cell images (data not shown).
Connectomics
In our experiments on synthetic data, we noted monotonically improved performance with increasing timesteps, which motivated our choice of building these models with as many timesteps as could fit into GPU memory without sacrificing (we carried this concept over the design of our 7-Nets for BSDS). Thus, we settled on 8 timesteps for the y-Nets. We also compared our use of fGRU modules learn recurrent connection vs. the classic LSTM and GRU recurrent modules, and found that the 7-Netwas far more effective on small datasets, which we take as evidence that its recurrent application of suppression separately from facilitation a better inductive bias for learning contour tasks (see Hahnloser et al. 22000 for theoretical discussion on how these operations amount digital selection task-relevant features through inhibition, followed by an analog amplification of the residuals through excitation).
We found that 7-Net cell membrane detection was improved when every bottom-up unit (from a typical convolution) was given hidden state. Like with gated recurrent architectures, these gates enable gradients to effectively skip timesteps of processing where they pathologically decay. We do this by converting every convolutional layer (except the first and last) into "minimal gated unit" (Heck & Salem 2017). This conversion introduced two additional kernels to each convolutional layer, UF .. RIX BIXIXXX where the former was responsible for selecting channels from persistent activity H E XX-XXXX for processing on a given timestep and updating the persistent activity. The latter kernel transformed a modulated version of the hidden state H. This transformed hidden state was combined with a vanilla convolutional feedforward encoding, Z E IR Xxx x (see Eq. 1 for the treatment). Weights in these layers were initialized with orthogonal random initializations, which help training recurrent networks (Vorontsov et al. 2017).
F = o (Z + F Hit 1] bF)
H F + H 1]+ F)) ELU(Z+ H (FO H(t 1)) + bz)
IGRU Here we describe additional details the IGRU. IGRU kernels for computing suppressive and facilitative interactions have symmetric weights between channels, similar to the original circuit is equal to the weight
of Mély et al. (2018). This means that the weight Az. W Az. ky where To and yo denote kernel center. This constraint means that there nearly half as many learnable connections as a normal convolutional kernel. our experiments, this constraint improved performance.
15

---

Published as a conference paper at ICLR 2020
BSDS (20M parameters) 7-Net
Layer cony-1-down
Operation
Output shape
conv 3 3
320 x 480 x 64
conv x
320 x 480 x
maxpool 2 x 2 conv 3 x 3/
160 x 240 x 64
conv-2-down
160 x 240 x 128
conv 3
160 240 x 128
FGRU-horizontal x 3/1
160 x 240 x 128
maxpool x 2 2
80 120 128
conv-3-down
conv 3
80 120 x 256
conv x /1
80 x 120 x 256
conv 3/1
80 120 256
FGRU-horizontal 3/1
80 x 120 x 256
maxpool 2 2 2
40 x 60 256
conv-4-down
conv 3 x 37 I
40 x 60 512
conv
60 512
conv 3 3
40 x 60 512
fGRU-horizontal 3/1
40 x0 512
maxpool 2 x 2
20 30 512
conv-5-down
conv 3 x 3
20 x 30 512
conv 3
20 x 30 512
conv 3 3/1
20 x 30 512
fGRU-horizontal 3/1
x 30 512
conv-4-up
instance-norm
20 x: 512
bilinear-resize
x0 x 512
conv 1/1
40 60 x
conv x 1/1
x0 512
fGRUU-topdown x 1/1
40 512
conv-3-up
instance-norm bilinear-resize
40 x 60 512
80 120 512
conv 1/1
80 x 120 x 16
conv 1/1
80 120 256
fGRU-top-down x 1/1
80 120 x 256
conv-2-up
instance-norm
80 x 120 x 256
bilinear-resize
160 240 256
conv x 1/1
160 x 240 x 64
conv 1/1
160 240 x 128
fGRU-top-down x 1/1
160 x 240 x 128
Readout
nostance-norm bilinear-resize
160 x 240 x 128
320 x 480 x 128
conv 1/1
320 x 480 x
Table S1: 7-Net architecture for contour detection in BSDS natural images. For comparison, the BDCN, which is the state of the art on BSDS, contains x x1.. parameters. When training on NVIDIA GeForce RTX, this -Nettakes 1.8 seconds per image, whereas the BDCN takes 0.1 seconds image. "Down" refers down-sampling layers; "up" refers up-sampling layers, and "readout" maps model activities into per-pixel decisions. Kernels are described as kernel-height x kernel-width stride size. All convolutional layers except for the Readout non non-linearities All non- non-lineen -ineearrssss in this network are linear rectifications. Model predictions come from the fGRU hidden state for conv-2-down, which are resized match the input image resolution and passed the linear per-pixel readout.
16

---

Published as a conference paper at ICLR 2020
Connectomics y-Net (450K parameters)
Layer conv-I-down
Operation
Output shape
conv conv x 3 1
384 384 24
384 x 384 x 24
fGRU-horizontal 9
384 x 384 x 24
maxpool 2 x 2 2
192 x 192 x 24
conv-2-down
conv x 3 1
192 x 192 x 28
fGRU-hhrizontall 7 x 7 1
192 x 192 x 28
maxpool 2 x 2/2
96 x 96 28
conv-3-down
conv 3/1
96 36
FGRU-horizontal 5 x 5
96 x 96 36
maxpool 2 x 2 2
x 48 36
conv conv-4-down
conv X 3/1
48 x 48
48
IGRU-horizontal x 3 maxpool 2 x 2 2
48 x 48
48
24 24
48
conv-5-down
conv x 3 1
24 x 24
64
FGRU-horizontal
1/1
x 24 64
conv-4-up
transpose-cony 4 x 4/2
48 x 48
48
conv x / instance-norm fGRU-top-down 1 x 1/1
48 x 48
48
48 x 48
48
48 x 48 48
conv-3-up
transpose-cony 4 x 4 conv x /
96 36
96 x 96 36
instance-norm fGRU-top-down I x 1/1
x 96 36
96 x 96 36
conv-2-up
transpose-cony 4 4/2
192 x 192 x 28
conv / instance-norm
192 x 192 x 28
192 x 192 x 28
IGRU-top-down I 1/1
192 192 28
conv-l-up
transpose-conv 4 x 472
384 x 384 x 24
conv 3 1
384 x 384 x
instance-norm fGRU-top-down 1 X 1/1
384 x 384 x 24
384 x 384 x 24
Readout
instance-norm conv x 5
384 x 384 x 24
384 X 384 x 24
Table S2: >-Netarchitecttree for cell membrane detection in SEM images. 2D version of the U-Net Lee et al. (2017), which is the state of the art SNEMI3D, contains R600K parameters. When training on an NVIDIA GeForce RTX, this N-Nettakes 0.7 seconds per image, whereas the U-Net takes 0.06 seconds per image. "Down" refers to down-sampling layers; "up" refers to up- sampling layers, and "readout" maps model activities into per-pixel decisions. Kernels are described kernel-height kemel-width / stride size. All fGRU non-linearities all convolutional non-linearities are exponential linear units (ELU), as in (Lee et al. (017). All convolutional layers except for the Readout use non-linearities. Model predictions come from the fGRU hidden state for conv- conv-1-down 1-www,, which are passed to the linear readout.
linear rectifications, and
17

---

Published as a conference paper at ICLR 2020
While optimizing y-Nets synthetic cell image datasets, found that small modification of the fGRU input gate offered a modest improvement in performance. We realized that the input gate in the fGRU is conceptually similar to recently developed models for feedforward self-attention in deep neural networks. Specifically, the global-and-locst attention modules of (Linsley et al. 2019), in which non-linear transformation of layer's activity is used modulate the original activity. Here, we took inspiration from global-and-low attention, and introduced an additional gate into the IGRU, resulting in the following modification of the main equations.
Stage
A S U H(( 1]
# Compute channel-wise selection
M UM Hlt 1]
Compute spatial selection
G = sigmoid (IN (A @ (M
Compute suppression gate
C = IN(W" -H G
# Compute suppression interactions
S = z - [aH - (aH
C
# Additive and multiplicative suppression of Z
Stage? 2:
G = sig * +S))
Compute channel-wise recurrent updates
Cº = IN +S)
# Compute facilitation interactions
H = |r(C + w(C * =S]
+
# Additive and multiplicative facilitation of S
H[t = (1 - G²) Hit 1| G @
Update recurrent state
B(r]
where IN (N = =
Var r| +
This yields the global input gate activity A E RXXXX and the local input gate activity M RXXEx/, which are computed as filter responses between the previous hidden state H(: H[(-1] and the global gate kernel U E R x xxxx and the local gate kernel U e 33x3xxx1, Note that the latter filter is learning mapping into dimension and is therefore first tiled into K dimensions, yielding S before elementwise multiplication with A All results in the main text use this implementation.
Following the lead of (Linsley et al., 2018a), we incorporated normalizations into fGRU. Let E Rd denote the vector of layer activations that will be normalized. We chose instance normal- ization (Ulyanov et al. 2016) since it is independent of batch size, which was for 7-Nets in our experiments. Instance normalization introduces two k-dimensional learned parameters, 6, E Rd which control the scale and bias of normalized activities, and are are shared across timesteps of processing, In contrast, means and variances are computed on every timestep, since IGRU activities are not i.i.d. across timesteps. Elementwise multiplication is denoted by O and is a regularization hyperparameter.
Learnable gates, such as those in the fGRU, are helpful for training RNNs. But there are other heuristics that are also important for optimizing performance. We use several of these with y-Nets. such as Chronos initialization of fGRU gate biases (Tallec & Ollivier 2018) and random orthogonal initialization of kernels (Vorontsov et al., 2017). We initialized the learnable scale parameter 5 of fGRU normalizations to 0.1, since values near 0 optimize the dynamic range of gradients passing through its sigmoidal gates (Cooijmans et al.) 2017). Similarly, fGRU parameters for learning additive suppression/facilitttioo (n, v) were initialized to 0, and parameters for learning multiplicative inhibition/excitation (a, w) were initialized to 0.1. Finally, when implementing top-down connections, incorporated extra skip connection. The activity of layer l was added to the fGRU EeRU-computed top-down interactions between layer and layer l + 1. This additional skip connection improved the stability of training.
18

---

Published as a conference paper at ICLR 2020
Volume
Neurites
Membrane
Segmentations
Figure S1: We trained the reference 3D U-Net from (Lee et al. 2017 on the SNEMI3D dataset to validate the implementation. Segmentations here are derived by watershedding and agglomeration with GALA (Nunez-Iglesiaa et al. 2013), resulting in "superhuman" ARAND (cvaluated according the SNEMI3D standard; lower better) of 0.04, which is below the reported human- meman-performance threshold of 0.06 and on par with the published result (see Table in Lee et al. 2017 mean affinity
agglomeration).
B MEMBRANE PREDICTION MODELS
Our reference model for membrane prediction is the 3D U-Net of (Lee et al., 2017). This archi- tecture consists of four encoder blocks (multiple convolutions and skip connections, pooling and subsampling), followed by four decoder blocks (transpose convolution and convolution). This U-Net uses spatial pooling between each of its encoder blocks to downsample the input, and transpose convolutions between each of its decoder blocks upsample intermediate activities. We validated our implementation of this U-Net following the author's 's training routine, and were able to replicate their reported "superhuman" performance in cell segmentation SNEMI3D (Fig, (SI).
The y-Net for connectomics resembles the U-Net architecture oce et al. (2017). This retracces the blocks of convolutions and skip connections of that model with single layer of convolution followed by (GRU (as in the high level diagram of Fig. D.. (onv(0) (GRU(()) ) In the encoder pathway, fGRUs store horizontal interactions between spatially neighboring units of the preceding convolutional layer in their hidden states. In the decoder pathway, NeNet introduces fGRUs that learn top-down connections between layers, and connect recurrent units from higher-feature processing layers to lower-feature processing ones.
19

---

Published as a conference paper at ICLR 2020
Name
Tissue Mouse cortex Mouse retina
Imaging mbSEM
Resolution
Voxels (X/Y/2/Volumes)
SNEMI3D Ding
6 x x 29nm
1024 1024 100 384 X 384 x 384 x
SBEM
13.2 x 13.2 X 26nm
Table S3: SEM image volumes used membrane prediction. SNEMI3D images and annotations are publicly available (Kasthuri et al. 2013), whereas the Ding dataset is volume from (Ding et al. 2016) that we annotated.
(a)
(b)
Figure S2: Examples of tilt-illusion stimuli. (a) For training images, we sample over range of size and wavelength generate single oriented grating patches. (b) Test images obtained by sampling a full range of surround orientation, while fixing all other parameters such as size and frequency of gratings as well as the orientation of the center gratings (at 45 degrees).
Key to the approach of Lee et al. (2017) is their use of large of random data augmentations applied to SEM image volumes, which simulate common noise and errors SEM imaging. These are (i) misalignment between consecutive z-locations in each input image volume. (ii) Partial- or fully-missing sections of the imput image volumes. (iii) Blurring of portions of the image volume. Augmentations that simulated these types of noise, well as random flips over the zyz-plane, rotations by 90", brightness and contrast perturbations, were applied to volumes following the settings of Lee et al. (2017). The model was trained using Adam (Kingma & Ba 2014) and the learning rate schedule of Lee al. (2017), in which the optimizer step-size was halved when validation loss stopped decreasing (up to four times). Training involved single-SEM volume batches of 160 x 160 x 18 (X/Y/Z), normalized [0. 1]. As in Lee al. (2017). models were trained to predict nearest-neighbor voxel affinities, as well as 3 other mid- to long-range voxel distances. Only nearest neighbor affinities were used test time.
C ORIENTATION-TLLY ILLUSION IMAGE DATASET
Models were tested for a tilt illusion by first training on grating images of a single orientation, then testing on images in which center grating had the same/different orientation as surround grating. Each image in the training dataset consisted of circular patch of oriented grating on gray canvas of size 500 x 500 pixels. ensure that the decoder successfully decoded orientation information from model activities, the training dataset incorporated wide variety of grating stimuli with 4 randomly sampled image parameters: r, A, 0, and Φb denotes the radius of the circle in which oriented grating has been rendered, and was sampled from uniform distribution with interval between 80 and 240 pixels; specifies the wavelength of the grating pattern and was sampled from a uniform distribution with interval between 30 and 90 pixels; specifies the orientation of the gratings and is uniformly sampled from all possible orientations; ó denotes the phase offset of the oriented gratings and is also uniformly sampled from all possible values. The models' BSIDS-trained weights were fixed and
20

---

Published as a conference paper at ICLR 2020
10a
Val
10
10-1.
Loss
Loss
10a
Train
10
14%
10%
100%
0.1x
1x
10x
25000 Training iterations
1%
10% 100%
Training dotaset size
Learningg rate
Training dataset size
(a) BDCN (BSDS)
(b) U-Net (Connectomics)
Figure S3: Searching over learning rates did not rescue BSDS performance on small BSDS datasets had no affect on performance. over learning rates did not rescue from overfitting. The left panel depicts training and validation losses for BSDS on different sized subsets of BSDS500. (b) Performance after training with three different learning rates on the 5% split. There is little difference in best validation performance between the three learning rates. (c) The full training and validation loss curves for the BDCN trained on 5% of BSDS. The model overfits immediately. The model also overfit on the other dataset sizes, but because there was more data, this happened later training.
readout layers were trained to decode orientation the center of each image (procedure described in main text).
This setup allowed us to tease apart the effects of the surround on the representation of orientation in the center by introducing separate surround regions each test image filled with gratings with same/different orientations as the center (Fig S2b). Each test image was generated with one additional parameter, A which specified orientation difference of the surround gratings with respect to the center orientation, 6, and sampled from uniform distribution with interval between 90 and +90 degrees. The radius of the surround grating is denoted by r and was sampled from the same uniform distribution we used in training dataset. Center gratings then rendered in circle of
radius that is one half of the surround gratings.
21

---

Published as a conference paper at ICLR 2020
Horizontal-only (3x3 kernel)
Horizontal-only (16x15 kernel)
Untied weights
Top-down only
30
35
Models' perceived shift in orientation
20
20
10
10
BSDS500 Test F1 ODS
89
United wrights
Hontsontal only (186
6%
10%
100%
6%
10%S
100%
No additive feedback
multiplicative feedback
No recurrence
No non-negativity
40
30
30
20
10.
y-Nit
10
incamencc
sobilnee
80
icn-negshiity
9%
10%
100%
5%
10%
100%
25
00
25
60
75
26
50
75
25
50
75
Proportion training data used
Center-surround orientation difference
Figure $4: Performance of lesioned 7-Nets. We evaluate the F1 ODS of these models on BSDS500 test images (left) and for the presence of an orientation- tilt illusion (right). Top row: A com- parison of 7-Nets with "untied weights" (i.e., unrolled to 8-timesteps with unique weights on ev- ery timestep), only top-down connections, only horizontal connections (using the standard 3x3 horizontal kernels), and only horizontal connections but with larger kernels (15x15) The full y Net outperformed each of these models on all subsets of BSDS500 data. Both horizontal-only models captured the repulsive regime of the orientation-til illusion when center and surround gratings were similar, but only the version with larger kernels also showed an attractive regime, when the center and surround orientations were dissimilar. Bottom one timestep of processing), no constraint for non-negativity (see fGRU formulation in main text), no parameters for additive feedback (μ and x), and parameters for multiplicative feedback (7 and w). Once again, the full y-Net outperformed each of these models. While none of these models showed the full orientation -ttltt illusion. the 7-Net without negatiiaty constraints and the 7-Net without additive feedback showed the repulsive regime of the orientation-tilt illusion.
A comparison of 7yNNts with recurrence (i.e.,
8
Image
Label
Prediction
Prediction processing timestep
23496TH
Figure S5: 7-Nets trained for contour detection learn to approach steady-state solution. The processing timecourse of 7--Net predictions on representative images from the BSDS500 test set. The L2 norm of per-pixel differences between predictions on consecutive timesteps (i.e., timestep 2 timestep 1) approaches 0, indicating that the model converges towards steady state by the end of its processing timecourse.
22

---

Published as a conference paper at ICLR 2020
(A)
Unidwing omargent uik allsson worsens y Net
contour dotuction
0.8
1008
0.8
Recument connections Thstaned or:
0.4
Graknngs
Domain-transter Control
Nusion-correcter
100
1:0
200
250
300
300
Steps of training
80
160
2400
Steps training
Figure S6: Performance of y-Nets during experiments correct orientation-til illusion. The clousion model was trained to have veridical representations of the central grating in tilt-illusion stimuli. To control for potential detrimental effects of the training procedure per se, a control model ("domain-transfer control") was trained to decode orientations of single-grating stimuli. (a) Training causes contour-detection performance of both models to drop. However, the illusion-correctdd model performance drops significantly more than the biased model (see main text for hypothesis testing). The losses for both models converge towards across training, indicating that both learned to decode central-orient orientttoisn of their stimuli. (b) Contour detection examples for biased and birr-corretted models across steps of this training procedure.
Label
Festure preferred by:
Figure S7: Differences in contour predictions for the illusion-correetedd and domain-transffr control NeNets on BSDS500.
23


Reviews:
[{"review_id": "r1ee0w3k9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new artificial neural network architecture that is derived from a human visual model (Mély et al., 2018). The original (human vision) model can explain some of the human visual illusions, specifically contextual ones. While the adaptation from this human visual model to artificial neural networks was previously done by (Linsley et. al., 2018a), in this paper the authors extend (Linsley et. al., 2018a) to better capture some of the constraints in the human visual model, and also to add a formulation that can also model top-down connections (across layers).\nThe goal of replicating the structure of the visual human model in artificial neural networks is to improve the machine vision by mimicking the human vision. The results in this paper on contour detection show an improvement regarding sample efficiency with respect to other state of the art methods. Also, the authors show that the artificial model also suffers from visual illusions, and when these are explicitly corrected, its performance drops. This shows that these illusions are a byproduct of the system improving its visual abilities.\n\nStrengths:\n1 - Well motivated by cognitive science theory and modeling of the human visual system. The artificial modeling is not just inspired by the human visual system, but derived from an actual human system, replicating it. The paper presents an extension of (Linsley et. al., 2018a), and both the base model and the extension are biologically motivated. The paper also shows that this extension is important to get good results.\n2 - Clear structure and ideas. Clearly explained, well written, reasonable ablations and transparent presentation of results, assumptions, limitations, contributions and experiments.\n3 - Very good results in contour detection for a very-low data regime, which proves the strength of the model inductive bias. The results show both better performance and good mimicking of the human vision behavior. \n\nWeaknesses\n1 – Limited experimental results. \n- Is contour detection the only task in which surround suppression helps?\n- Lack of experiments in larger contour detection datasets (like Semantic Border Dataset or Cityscapes). Does the model improve accuracy in those, or it just improves sample efficiency in (extremely) small (subsets of) datasets? \n- The only results are regarding sample efficiency. Not accuracy, or number of parameters, or running time. \n2 – Weak results supporting the claim that the computer vision model mimics the human visual illusions. The authors show an elegant experiment where they explicitly correct the visual illusion and obtain worse results, backing the presented hypothesis. However, the ablation experiments show that the same model without the top-down connections does not present the same results. One would expect that removing a part that is not in the initial formulation from (Mély et al., 2018) should not affect in the visual illusion experiments. Both the explanation at the end of Section 2 and at the sixth paragraph of Section 3 seem to indicate that the _non-negativity_ is the important factor to explain contextual illusions, not the top-down formulation. Can the authors explain whether or not the top-down formulation is a necessary part to model (Mély et al., 2018)? If this is not the case (and top-down is not necessary), the current results would not support the idea that implementing (Mély et al., 2018) in artificial neural networks produces visual illusions in the computer. Also, did the authors perform the visual illusion experiment without the non-negativity? Is it a necessary requisite for the visual illusions to appear? As a positive remark regarding weakness #2, the results still show that the model that performs best is the one having visual illusions. \n3 – Related to weaknesses #1 and #2, did the authors perform any experiment on other contextual illusions like the ones explained in (Mély et al., 2018), namely “color induction” or “enhanced color shifts”? Consistent findings across different visual illusions would reinforce the presented hypothesis.\n\nAdditional comments:\n- While it may not be a \"bug\", arguing that the visual illusions are a \"feature\" (both in machine and human visual systems) is probably too much of a claim. At some point the authors refer to it as a \"byproduct\" of the of the system improving its visual abilities, which I consider a more suitable word. \n- For an audience outside of neuroscience, a brief explanation of the concepts “suppression” and “facilitation”, which are very important in the paper, would be convenient.\n- Are 8 time-steps sufficient for reaching a steady state? Is the steady state checked in any way?\n\n-------- Updated --------\nRating:\n- Weak accept\nThe authors addressed the initial concerns I raised, providing detailed explanations and additional experiments. While some of these concerns still remain to some extent, I believe this paper explores an interesting direction connecting human visual models with computer vision, obtaining good experimental results. The authors softened some of their initial claims, and the current ones are more supported by the experiments.\nOverall, I believe this paper can be a valuable contribution to the conference, and I recommend acceptance.\n\n-------- Previously --------\nRating:\n- Weak reject \nOverall it is a clear and well-structured paper, with interesting biologically derived architectures (strength #1), but as it stands the experimental results either do not completely support the claims of the paper (weakness #2) or are limited in scope (weakness #1). If the authors can address my concerns, mostly motivating the importance of the experiments for weakness #1, and providing an explanation (possibly correcting me) for weakness #2, I will be happy to increase my rating. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgVXkl0FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The presented paper introduces a novel neural network architecture to explore the question whether visual illusions are corner cases of the human visual system, or whether they represent limitations of perception. The developed recurrent network architecture aims at being more sample efficient than existing methods. The findings discussed in the paper suggest that visual illusions are a byproduct of neural circuits that help to increase the robustness of the human visual system, which in turn suggests that neural networks for processing visual data could benefit from integrating circuits in similar ways. While existing work predominantly aims at explaining whether visual illusions are features or artifacts of the visual system, this work focuses on finding a computational solution to support the hypothesis that visual illusions are features. In particular, the contributions of this work are: (1) novel neural network architecture, called \\gamma-networks, which is derived from the work of [Meley et al. 2018] and (2) that the proposed architecture is more sample efficient than SOTA convolutional architectures on contour detection tasks. \n\nOverall, I think this is an interesting paper that can be accepted for publication. However, I am not an expert in this field and am excited to see what the other reviewers think. While the overall contribution of this work appears rather narrow, exploring traits of the human visual system to leverage them in the context of convolutional neural networks for segmentation tasks appears interesting and has the potential to simulate further research in this field. Furthermore, the results are promising and show that the introduced approach is on par with SOTA work in the field. On the downside, many sections of the paper appear unnecessarily cryptic and lack important details. Other sections rely on explanations in the appendix or just refer to related work instead of providing context. This gives me the impression that this work might be better suited for a journal instead of trying to discuss all necessary details into the page limits of ICLR (the paper is also two pages over the common page count). Moreover the topic this paper addresses might be a better fit for a venue that specifically focuses on neuroscience (similar as Mely et al. 2018).   \n\n\nSpecific comments: \n\n- Abstract and introduction can be improved by more carefully introducing visual illusions and how the effects of specific traits of the human visual system lead to visual illusions. In its current form abstract and introduction are quite difficult to follow for readers that are less well-versed in neurscience. \n- While the main contributions are listed at the end of the introduction, they are not clearly described. It is not clear why it is beneficial that the network generates \"an orientation-tilt illusion after it is optimized for contour detection\". More details need to be provided here. \n- The term \"hyper columns\" is not properly introduced and needs better motivation, similarly for terms like \"suppression\" and \"facilitation\", \"circuit integration\". Overall it seems the methods section relies too much on the related work for explaining concepts and terminology, which makes this section quite difficult to follow. This needs to be improved. \n- It is not clear why it is beneficial to enforce non-negativity. The provided explanation is unclear: \"The non-negativity constraints we introduce into the fGRU are necessary to guarantee separate stages of suppression followed by facilitation that can implement asymmetric contextual interactions\", which is again because terms were not properly introduced. The term \"fGRU\" is introduced after it is used.\n- Not providing training time with the argument that it is outside the scope of this work seems like a flawed argument and is uncommon practice.  \n- The datasets listed in Section 4.1. are introduced without references. \n- The term \"F1 ODS\" is not properly introduced. \n- Figure 3 (a): the term \"Ding\" is not introduced. Does this refer to [Ding et al. 2016]?\n- The references show many inconsistencies (abbreviated vs. long conference names, etc.). This should be fixed for the final version of this work. \n- No limitations are discussed. What are the edge cases in which the method does not perform well?\n\n\nTypos: \nSec 6: hihg-level", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeQ23xaKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a complex hierarchical recurrent model for contour detection loosely inspired by the organization of cortical circuits. Their model performs state-of-the-art on sample-limited versions of popular contour detection (BSDS500) and cell segmentation (SNEMI3D) datasets, and it reproduces the well-known tilt illusion when transfer-learning orientation estimation. Interestingly, \"untraining\" the tilt illusion degrades performance on contour detection.\n\nStrengths:\n+ State-of-the-art performance on data-limited contour detection tasks\n+ Nice illustration of how the network refines its predictions over time\n+ Demonstrates a contextual visual illusion in a task-trained neural network\n+ Shows that tilt illusion is actually necessary for optimal performance in their model\n\nWeaknesses:\n- Architecture seems very complicated (unnecessarily so?)\n- No ablation studies showing the usefulness of various model components\n- Title seems a bit overly general given the quite specific result\n- Not clear whether their results support their interpretation of the function of illusions\n\nIt's a relatively straightforward paper that is easy to follow and has a clear result that is both interesting and novel. Thus, I'm generally very supportive of the paper.\n\nThere are a couple of weaknesses summarised above and detailed below that I would love to see addressed, but none of them is overly critical:\n\n1. Unfortunately the paper suffers from the same issue as the original work on fGRU, which it's based on: The fGRU architecture seems overly complicated and its numerous details and design choices not well motivated. Ablation studies showing which components are really necessary are missing. While this was understandable for the original paper, which introduced a novel approach, one would hope that follow-up work would subsequently get rid of some of the slack and simplify the architecture to the minimum that's really required.\n\n2. The title suggests that the paper explains the function of contextual illusions in general, but the paper actually \"just\" shows that one contextual illusion emerges when one trains a biologically inspired model on one particular task. I suggest aligning the title better with the actual contribution.\n\n3. (somewhat philosophical) The paper does not really answer the question posed in the abstract, does it? Do visual illusions reflect basic limitations of the visual system or do they correspond to corner cases of neural computations that are efficient in everyday settings? The authors seem argue for the second possibility. But if that was the case, wouldn't one expect other systems trained on the same tasks to also exhibit these illusions? It seems to me as if their results might suggest quite the opposite: Because only brain-like architectures exhibit this illusion, and because only they are hurt by \"unlearning\" the illusion, this visual illusion may reflect a basic limitation of how the visual system solves the task. I think it would be great if the author could comment on this point and clarify their reasoning in the paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers recommend accept, and the found the paper interesting and novel. 
Decision: Accept (Poster)