Disentangling neural mechanisms for perceptual grouping
Authors: Junkyung Kim*
Drew Linsley*
Kalpit Thakkar
Thomas Serre
Venue:  (2020)
Keywords: Perceptual grouping
visual cortex
recurrent feedback
horizontal connections
top-down connections

Abstract: Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence. This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons. However, the relative contributions of these connections to perceptual grouping are poorly understood. We address this question by systematically evaluating neural network architectures featuring combinations bottom-up, horizontal, and top-down connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping. We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up connections. Horizontal connections resolve straining on tasks with Gestalt cues by supporting incremental grouping, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object. Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.

Published as a conference paper at ICLR 2020
DISENTANGLING NEURAL MECHANISMS FOR PERCEPTUAL GROUPING
Junkyung Kimit, Drew Linsleyt, Kalpit Thakkar & Thomas Serre Department of Cognitive, Linguistic and Psychological Sciences
Brown University
Providence, RI 02912, USA
junkyung@google com
thomas_ @brown. edu [drew_linsley: kaap h
ABSTRACT
Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence. This ability is thought arise in the brain from computations implemented by bottom-up. horizontal, and top-down connections between neurons. However, the relative contributions of these connections perceptual grouping are poorly understood. We address this question by systemati- cally evaluating neural network architectures featuring combinations bottom-up, horizontal, and top-down connections two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping. We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up connections. Horizontal connections resolve straining on tasks with Gestalt cues by supporting incremental grouping, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object. Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups
INTRODUCTION
The ability form perceptual groups and segment scenes into discrete set of objects constitutes fundamental component of visual intelligence. Decades of research in biological vision have suggested that biological visual systems extract these objects through visual routines for perceptual grouping, which rely on feedforward (or bottom-up) and potentially feedback (or recurrent) processes (Roelfsema) 20066] Roelfsema & Houtkamp 2011 Wyatte et al. [2044). Feedforward processes group scenes by encoding increasingly more complex feature conjunctions through cascade of filtering, rectification and normalization operations. As shown in Fig. Th. these computations can be sufficient to detect and localize objects in scenes with little no background clutter, when object "pops out" because of color, contrast, etc. (Nothdurfi 1991). However, as illustrated in Fig. Пp. visual scenes are usually complex and contain objects interposed in background clutter. When this the case, feedforward processes alone are often insufficient for perceptual grouping (Herzog & Clarke) 2014 Pelli al. 2004 Freeman et 2012) Freeman & Pelli 22000), and has been suggested that our visual system leverages feedback mechanisms to refine an imitially coarse scene segmentation
(Ullman 1984 Roelfsema Houtkamp, 2011 Treisman & Gelade 1980 Lamme & Roelfsemal 2000).
To rough approximation, there are two distinct types of perceptual grouping routines that are associ- ated with feedback computations. One routine involves grouping low-level visual features with their neighbors according to Gestalt laws like similarity, good continuation, etc. (Fig. [Tp. top: Jolicoeur et al. [986; [99] Pringle & Egeth| 1988; RocIfsema et al. 1999 Houtkamp & Roelfsema, 2010 Houtkamp et al. 2003 Roclisema et al. 2002). Another routine is object-based and mediated by
"Theee authors contributed equally to this work.
Meed London, UK.

---

Published as a conference paper at ICLR 2020
Easy
Hard Low-level/ / "Gestalt"
Biological vs, artificial vision
Houtkamp & Rocltsema (2010) Linsley, Kim, Serre (2018)
Object-level / Semantics
Vecera Farah (1997)
Kim", Linsley", & Serre (2019)
fP
(a)
(b)
(c)
Figure 1: "Are both dots the same object?" (a) Bottom-up grouping, Our visual system can segregate an object from its background with rapid, bottom-up mechanisms, when the object is dissimilar to its background. In this case, the object is said "pop-out" from its background, making it trivial to judge whether the two dots are on the same object surface or not. (b) Non-bottom-up processes. Perceptual grouping mechanisms help segment a visual scene. (b, Top) The elements of behaviorally relevant perceptual object, such as a path formed by tracks in the snow, are transitively grouped together according to low-level "Gestalt" principles. This allows the observer trace path from one end to the other. (b, Bottom) Alternatively, observers may rely on prior knowledge or semantic cues to segment an object from a cluttered background. (c) Synthetic grouping tasks. (c, Top) The Pathfinder challenge (reproduced with permission from Linsley et al. (201 8b) and inspired byHokkmammp & Roelisema (010)) involves answering a simple question: "Are the two dots connected by path?" This task can be easily solved with Gestalt grouping strategies. (c, Bottom) Here, we introduce a novel "cluttered ABC" (cABC) challenge, inspired by Vecera & Farah (1997). The cABC challenge asks the same question visual stimuli for which Gestalt strategies are ineffective. Instead, cABC taps into object-based strategies for perceptual grouping.
high-level expectations about the shape and structure of perceptual objects. In this routine, feedback refines a coarse initial feedforward analysis of a scene with high-level hypotheses about the objects it contains (Fig. Dp, bottom; Vecera & Farah 1997, Vecera & O' O'Reilly] 1998 Vecera 1993 Zemel
ct al. 2002). Both of these feedback routines are iterative and rely on recurrent computations.
What are the neural circuits that implement Gestalt vs. object-based routines for perceptual grouping? Visual neuroscience studies have suggested that these routines emerge from specific types of neural interactions: (i) horizontal connections between neurons within an area, spanning spatial locations and potentially feature selectivity (Stettler et al. [2022] Gilbert & Wiesel 1989 McManus et al.
2011: Bosking et al.] 1997 Schmidt et al.)) 1997] Wannig et al.) (011)). and (ii) descending top-down connections from neurons in howwerrto-lower areas (Ko & von der Heydi, 2018 Gilbert & 1.i 2013
Tang et al.) 2018 Lamme et al. 1998 Murray et al. 2004]] 2002). The anatomical and functional properties of these feedback connections have been well-documented (see Gilbert & Li 2013 for review), but the relative contributions of horizontal vs. top-down connections for perceptual grouping remains an open question.
Here we investigate the function of horizontal vs. top-down connections during perceptual grouping. We evaluate each network on two perceptual grouping "challenges" that are designed to be solved by either Gestalt bbject-based visual routines. Our contributions are:
Contributions
We introduce mhologcaallyinnsiiee deep recurrent neural network (RNN) architecture featuring feedforward and feedback connections (the TD+H-CNN). We develop a range of lesioned variants of this network for measuring the relative contributions of feedforward
2

---

Published as a conference paper at ICLR 2020
and feedback processing by selectively disabling either horizontal (TD-CNN), top-down (H-CNN), or both types of connections (BU-CNN).
We introduce the cluttered ABC challenge (cABC), a synthetic visual reasoning challenge that requires object-based grouping strategies (Fig. Tc. bottom-right). We pair this dataset with the Pathfinder challenge (Linsley et al. 2018b, Fig. [e. top-right) which features low- level Gestalt cues. Both challenges allow for parametric control over image variability. With these challenges, we investigate whether each of our models can learn to leverage either object-level of Gestalt cues for grouping.
We find that top-down connections are key for grouping objects when semantic cues present, whereas horizontal connections support grouping when they are not. Of the network models tested (which included ResNets and U-Nets), only our recurrent model with the full array of connections (the TD+H-CNN) solves both tasks across levels of difficulty.
We compare model predictions to human psychophysics data on the same visual tasks and show that human judgements are significantly more consistent with image predictions from our TD+H-CNN than ResNet and U-Net models. This indicates that the strategies used by the human visual system on difficult segmentation tasks are best matched by highly-recurrent model with bottom-up, horizontal, and top-down connections.
2 RELATED WORK
Recurrent neural networks Recurrent neural networks (RNNs) are a class of models that can be trained with gradient descent approximate discrete-time dynamical systems. RNNs are classically featured in sequence learning, but have also begun to show promise in computer vision tasks. One example of this includes autoregressive RNNs that learn the statistical relationship between neighboring pixels for image generation (Van Den Oord et al. 2016). Another approach, successfully applied to object recognition and super-resolution tasks, is to incorporate convolutional kernels into
RNNs O'Reilly al., 2013; Liang & Hu 2015 Liao & Poggio 2016 Kim et al., 2016 Zamir cct al. 2017). These convolutional RNNs share kernels across processing timesteps, allowing them to achieve processing depth with fraction of the parameters needed for CNN of equivalent depth. Such convolutional RNN models have also been used by multiple groups as a foundation for vision models with biologically-inspree feedback mechanisms (Lotter et al. [2066] Spoerer al. 2017
Kietzmann et al.) 2019 Wen et al. 2018 Linsley et al. 2018b Nayebi et al. 2018a Kar et al. 2019).
Here, we construct convolutional-RNN architectures using the feedback gated recurrent unit (fGRU) proposed by Linsley et al.] Sa). The fGRU extends the horizontal gated recurrent unit (hGRU, Linsley et al. (0188)) implement (i) horizontal connections between units processing layer separated by spatial location and/or feature channel, and (ii) top-down connections from units in togherrto-lowwr network layers. The fGRU was applied to contour detection in natural and electron microscopy images, where it was found to be more sample efficient than state- titt--of-hh-arr models (Linsley al., 2020). We leverage a modified version of this architecture to study the relative contributions of horizontal and top-down interactions for perceptual grouping.
Synthetic visual tasks There is a long history of using synthetic visual recognition challenges for evaluating computer vision algorithms [Ullman] 1996 Fleuret al. 2011). There is a popular dataset of simple geometric shapes used to evaluate instance segmentation algorithms (featured in Reichert & Serre 2014 Greff al. 2016), and variants of the cluttered MNIST dataset and CAPTCHA datasets
have been helpful for testing model tolerance to visual clutter (e.g., Gregor et al. 2015 Sabbah et al. 2017; Spoerer et al. 2017, George et 2017 7a Jaderberg et al. 2015 Sabour al. 2017). Recent synthetic challenges have sought to significantly ramp up task difficulty (Michaelis et al. (018). and/or to parameterically control intra-class image variability (Kim et al. 2018]]
[ Linsley et al. 2018b).
The current work follows this trend of developing novel synthetic stimuli that are tightly controlled for low-level biases (unlike e.g., MNIST), and to design parametric controls (c.g., intra-class variability) for task difficulty to try strain network models.
In the current study, we begin with the "Pathfinder" challenge (Linsley et al. 2018b), which consists of images with two white markers that may or may not be connected by a long path (Fig. (2). The challenge illustrated how a shallow network with horizontal connections could recognize a connected path by learning efficient incremental grouping routine, while feedforward architectures (CNNs)
3

---

Published as a conference paper at ICLR 2020
Positive
Pathfinder
Negative
Shortor curves
Longer curves
Positive
LX
CABC
Negative
WA
Correlated
Independent
Easy (low variability)
Intermediate
Hard (high variability)
Figure 2: An overview of the two synthetic visual reasoning challenges used: the "Pathfinder" (top two rows) and the "cluttered ABC" (cABC, bottom two rows). On both tasks, a model must judge whether the two white markers fall on the same or different paths/letters. For both, parametrically controlled task difficulty by adjusting intra-class image variability in the image dataset. In the Pathfinder challenge, correlation between geometric transformations applied to the two letters while also increasing the relative positional variability between letters.
increased the length of the target curve; in cABC, we decreased the
with orders of magnitude more free parameters struggled to learn the same task. Our novel "cluttered ABC" (cABC) challenge complements the Pathfinder challenge by posing the same task images that feature semantic objects rather than curves. Thus, cABC may be difficult to solve using the same incremental grouping routine as Pathfinder, which could capture both objects in a single group (Fig.
SYNTHETIC PERCEPTUAL GROUPING CHALLENGES
Pathfinder and cABC challenges share a key similarity in their stimulus design: white shapes are placed on black background along with two circular and white "markers" (Fig. 2 These markers are either placed on two different objects the same object, and the task posed both datasets is to discriminate between these two alternatives. The two challenges depict different types of objects. Images in the Pathfinder challenge contain two flexible curves, whereas the cABC challenge uses overlapping English-alphabe characters that transformed in appearance and position.
Local vs. object-level cues for perceptual grouping Differences in the types of shapes used in the two challenges make them ideally suited for different feedback grouping routines. The Pathfinder challenge features smooth and flexible curves, making it well suited for incremental Gestalt- based grouping (Linsley et al., 20185). Conversely, the English-alphan alphabet characters in the cABC challenge make it good match for grouping routine that relies semantic-level object cues.
4

---

Published as a conference paper at ICLR 2020
The two synthetic challenges are designed to cause high computational burden in models that rely on sub-optimal grouping routines. For instance, cluttered, flexible curves in the Pathfinder challenge are well-characterized by local continuity but have set "global shape" or any semantic-level cues. This makes the task difficult for feedforward models like ResNets because the number of fixed templates that they must learn to solve it increases with the number of possible shapes the flexible paths may take. In contrast, letters in cABC images are globally stable but locally degenerate (due to warping and pixelation; see Appendix A for details of its design.), and letters may overlap with each other. Because the strokes of overlapping letters form arbitrary conjunctions, a feedforward network needs to learn a large number of templates to discriminate real from spurious stroke conjunctions.
Overall, we consider these challenges two extremes on a continuum of perceptual grouping cues found in nature. On one hand, Pathfinder curves provide reliable Gestalt cues without any semantic information. On the other hand, cABC letters are semantic but do not have local grouping cues. As we describe in the following sections, human observers easily solve both challenges with minimal training, indicating that biological vision is capable of relying a single routine for perceptual grouping.
Parameterization of image variability It is difficult to disentangle visual strategies of neural network architecture choices using standard computer vision datasets. For instance, the relative advantages of visual routines implemented by horizontal vs. top-down connections can be washed out by neural networks latching onto trivial dataset biases. More importantly, the typical training regime for deep learning involves big datasets, and in this regime, the ability of neural networks to act as universal function approximators can make inductive biases less important.
We overcome these limitations with our synthetic challenges. Each challenge consists of three limited-size datasets of different task difficulties (easy, intermediate and hard; characterized by increasing levels of intra-class variability). We train and test network architectures on each of the three datasets and measure how accuracy changes when the need for stronger generalization capabilities arises as difficulty increases. This method, called "straining" (Kim et al., 2018 Linsley et al., 201 2018b), dissociates an architecture's 's expressiveness from other incidental factors tied to particular image domain. Architectures with appropriate inductive biases are more likely to generalize as difficulty/intra-llass variability increases compared to architectures that rely on low-bias routines that can resemble "rote memorization" (Zhang et al. 2016).
Difficulty of Pathfinder challenge is parameterized by the length of target curves in the images (6-, 9-, or 14-length; Linsley et al. 2018b). cABC challenge difficulty is controlled in two ways: first, by increasing the number of possible relative positional arrangements between the two letters. Second, by decreasing the correlation between random transformation parameters (rotation, scale and shear) applied to each letter in an image. For instance, on the "Easy" difficulty dataset, letter transformations are correlated: the centers of two letters are displaced by a distance uniformly sampled in an interval between 25 and 30 pixels, and the same affine transformation is applied to both letters. In contrast, letter transformations are independent: letters are randomly separated by 15 to 40 pixels, and random affine transformations are independently sampled and applied to each. We balance the variation of these random transformation parameters across the three difficulty levels so that the total variability of individual letter shape remains constant (i.e., the number of transformations applied to each letter in dataset is the same across difficulty levels). Harder cABC datasets lead to increasingly varied conjunctions between the strokes of overlapping letters, making them well suited for models that can iteratively process semantic groups instead of learning fixed feature templates. See Appendix A for more details about cABC generation.
4 NETWORK ARCHITECTURES
We test the role of horizontal vs. top-down connections for perceptual grouping using different configurations of the same recurrent CNN architecture (Fig. [p). This model is equipped with recurrent modules called fGRUs (Linsley et al.) 2018a), which can implement top-down and/or horizontal feedback (Fig. 3p). By selectively disabling top-down connections, horizontal connections, or both types of feedback. we directly compared the relative contributions of these connections for solving Pathfinder and CABC tasks over a basic deep convolutional network "backbone" capable of only feedforward computations. As an additional reference, we test representative feedforward
5

---

Published as a conference paper at ICLR 2020
architectures for computer vision: residual networks (He et al. 2016) and U-Net (Rnnebergeer iet al. 2013).
The (GRU module The IGRU takes input from two unit populations: an external drive, X, and an internal state, H. The fGRU integrates these two inputs to get an updated state for given processing timestep H(t), which is passed the next layer. Each layer's recurrent state is sequentially updated during single timestep, letting the model perform bottom-up to top-down sweeps of processing. When applied to static images, timesteps of processing allow feedback connections to spread and evolve, allowing units in different spatial positions and layers to interact with each other. This iterative spread of activity is critical for implementing dynamic grouping strategies. On the final timestep of processing, the fGRU hidden state H closest to image resolution is sent to readout module to compute a binary category prediction.
In our recurrent architecture (the TD+H-CNN), an fGRU module implements either horizontal or top-down feedback. When implementing horizontal feedback, an fGRU takes as external drive (X) the activities from a preceding convolutional layer and iteratively updates its evolving hidden state H using spatial (horizontal) kernels (Fig. 3 green fGRU).
In alternative configuration, fGRU can implement top-down feedback by taking input hidden state activities from two other fGRUs, one from a lower layer and the other from a high layer (Fig. 3b.
red fGRU). This top-down fGRU uses a local (1 X 1) kernel to first suppress the low-level H with the activity in H ((++)[(t [t] then separate kernel to "facilitate" the residual activity, supporting operations like sharpening or "filling-in". Additional details regarding the fGRU module are in Appendix
IGAUX
Top-down
Com-ppool
Transpoon Cony
Conn-pool pool
Transpoen Cony
Cons-pool
Transpose
Horizontal
IGRU
(GRUR
IORUT)
IGRUX
1GRUN
TGRURI
Curn
Comi
Cons
Resdout
Feedforward
X
X
(a)
(b)
t 0
t= 2,
t 8
Figure 3: We define convolutional RNN model that learn feedforward (gray), horizontal (green), and/or top-down (red) connections to disentangle the contributions of each type of connections for perceptual grouping. (a) A conceptual diagram of the TD+H-CNN architecture, depicting three layers of activity and feedforward, horizontal, and top-down connections. (b) A deep learning diagram of the TD+H-CNN, unraveled over processing timesteps. Bottom-up to top-down passes of the imput image allow recurrent horizontal (green fGRUs) and top-down (red (GRU) interactions between units evolve. This allows the model implement complex grouping strategies, like transitively linking target features together or selecting whole objects from clutter.
Recurrent networks with feedback Our full model architecture, which we call TD+H-CNN (Fig. introduces three fGRU modules into a CNN to implement top-down (TD) and horizontal (H) feedback. The architecture consists of downsampling (Conv-pool) pathway and an upsampling (Transpose Conv) pathway as in the U-Net (Ronneberger et al. 2015), and processes images in bottom-up to top-down loop through its entire architecture at every timestep. The downsampling pathway lets the model implement feedforward and horizontal connections, and the upsampling pathway lets it implement top-down interactions (conceptualized Fig. 3b. red (GRU). The first two fGRUs in the model learn horizontal connections the lowest- and highest-leva feature processing layers, whereas the third fGRU learns top-down interactions between these two fGRUs.
We define three variants of the TD+H-CNN by lesioning fGRU modules for horizontal and/or top- down connections: a top-down-only architecture (TD-CNN); horizontal architecture (H-CNN); and feedforward-only architecture (" ("hottom-up" BU-CNN; Appendix C for details).

---

Published as a conference paper at ICLR 2020
H-OW
Max Valceat Accuracy
(Pathitider)
0.8
0.6
0.5
TDi
CM
TO-ONA
Max Valcalion Accuracy
R
P
R
(Gluttered ABC) ABC)
6.7
0.6
0.5
(a)
(b)
Input
Fr-2
L2 Difference per timestep
C-4
EnG
En0
Figure 4: Pathfinder and cABC challenges disentangle the relative contributions of horizontal vs. top-down connections for perceptual grouping. (a) Accuracies of human and neural network models on three levels of the Pathfinder and cABC challenges. While human participants and the TD+H-CNN model are able to solve all difficulties of each challenge, Pathfinder strains the TD-CNN and cABC strains the H-CNN. (b) We visualized the computations learned by our networks using images containing single markers (See Appendix B for details.) We found that H-CNN and TD-CNN learn distinct strategies to solve Pathfinder and cABC challenges, respectively. The H-CNN solves Pathfinder by iteratively grouping segments of target path while suppressing background clutter. The TD-CNN solves cABC by globally facilitating activity of a target letter while suppressing the other letter. Model strategies are visualized by plotting the per-timestep L2 difference between hidden states of the fGRU responsible for model predictions.
Reference networks We also measure the performance of popular neural network architectures on Pathfinder and cABC. We tested three "Residual Network" (ResNets, He et al. (016) with 18, 50, and 152 layers and a "U-Net" (Ronneberger et al. 2015) which uses a VGG16 encoder. Both model types utilize skip connections to mix information between processing layers. However, the U-Net uses skip connections to pass activities between layers of downsampling encoder and an upsampling decoder. This pattern residual connections effectively makes the U-Net equivalent network with tow- feedback that is simulated for one timestep.
5 EXPERIMENTS
Training In initial pilot experiments, we calibrated the difficulty of Pathfinder and cABC challenges by identifying the smallest dataset size of the "easy" version of the challenges for which all models exceeded 90% validation accuracy. For Pathfinder this was 60K images, and for cABC this was 45K, and we used these image budgets for all difficulty levels of the respective challenges (with 10% held out for evaluation).
Our objective was understand whether models could learn these challenges on their image budgets.
We were interested in evaluating model sample efficiency. not absolute accuracy on single dataset nor speed of training ("wall-time"), Thus, we adopted early stopping criteria during training: training continued until 10 straight epochs had passed without reduction in validation loss or if 48 epochs of training had elapsed. Models were trained with batches of images and the Adam optimizer. We trained each model with five random weight initializations and searched over learning rates [le-3 le le le "J. We report the best performance achieved by each model (according to validation accuracy) out of the 20 training runs.
Screening feedback computations Our study revealed three main findings. First, we found that human participants performed well both Pathfinder and cABC challenges with no significant
7

---

Published as a conference paper at ICLR 2020
Marr Valdertca Accurecy (Pathfinder)
H-CWA
H1TD CNN
Explained vartance
bertiett
TD-CHI
Resteet E0d
Hoo
Reestet
Ner Selicator Accurecy
1.0
Difficllly
Explained vartance
[Glutteeed ABC)
0.9
TD Chi
0.8
0.6
0.7
0.4
Rastiets
0.6
02
0.5
DU-CNH
Reutee 13
8.
TD-CHM
10
100
1000
(a) AU-CWN Bo--et-113
(b)
parameter multiplier
(c)
(reative H-CNN)
Figure Pathfinder and BACC challenges are efficiently solved by visual strategies that depend on feedback not feedforward processing. (a) Performance of feedforward models on each difficulty of the two challenges. Deeper ResNet models solved Pathfinder, whereas the ResNet-18 and U-Net solved cABC, indicating trade-off between capacity vs. parameter efficiency for feedforward strategies to Pathfinder vs. cABC, respectively (failures to generalize reflect overfitting). (b) The number of parameters of each model (plotted as multiple of the number of parameters in the H-CNN) plotted against model performance on the hard dataset of each challenge. The H+TD-CNN solves both challenges with prder-of-magnitude fewer parameters than reference ResNet and U-Net models. (c) Correlations between human and leading model decisions on the hard dataset of each challenge. The TD+H-CNN and its successful lesioned variants were significantly more correlated with humans than successful feedforward reference models. Significance comparisons with the worst feedforward reference model omitted due to space constraints. P-values are derived from bootstrapped comparisons of model performance (Edgington, 1964). Error bars depict SEM; p <0.05; **: <0.01.
drop in accuracy as difficulty increased (Fig- 4a.. The fact that each challenge has been designed to feature a single cue for particular grouping (c.g.. local Gestalt for Pathfinder and global semantics for cABC) suggests that human participants were able to utilize different grouping routine for each challenge. Second, the pattern of accuracies from our recurrent architectures reveal complementary contributions of horizontal vs. top-down connections (Fig. 4D) The H-CNN solved the Pathfinder challenge with minimal straining as difficulty increased, but it struggled on the easy cABC dataset. On the other hand, the TD-CNN performed well the entire cABC challenge but struggled on the Pathfinder challenge as difficulty increased. Together, these results suggest that top-down interactions (from higher-to-lowe
layers) help process object-level grouping cues, whereas horizontal feedback (between units in different feature columns/spatial locations in layer) help process local Gestalt grouping cues. Our recurrent architecture which possessed both types of feedback, the TD+H-CNN, was able to solve both challenges at all levels of difficulties, roughly matching human performance.
All three of these recurrent models relied on recurrence achieve their performance on these datasets. In particular, an H-CNN with one processing timestep could not solve Pathfinder, and TD-CNN with one processing timestep could not solve cABC (Fig. S7p. Separately, performance of the TD+H- CNN was relatively robust to different model configurations, and deeper versions of the model still performed well on both challenges (Fig. (S8).
We also tested models two controlled variants of the cABC challenge: one where the two letters never touch or overlap ("position control") and another where the two letters are rendered different pixel intensities ("luminance control"). No significant straining was found these tasks for any of our networks, confirming our initial hypothesis that local ambiguities and occlusion between the letters makes top-down feedback important for grouping cABC images (Fig. S5).
8

---

Published as a conference paper at ICLR 2020
In contrast our recurrent architectures, its frrffly-feedforwward variant, the BU-CNN, struggled on both challenges, demonstrating the importance of feedback for perceptual grouping (Fig. (5)). Of the deep feedforward networks tested, only the ResNet-50 and ResNet-152 were able to solve the entire Pathfinder challenge, but these same architectures failed to solve cABC. In contrast, the ResNet-18 and U-Net solved the cABC challenge, but strained on difficult Pathfinder datasets. Overall, no feedforward architecture was general enough to learn both types of grouping strategies efficiently, suggesting that these networks face a trade-off between Pathfinder vs, cABC performance.
We explored this tradeoff' faced by ResNets in two ways, First, we tested whether the big ResNets which failed to learn the hard cABC challenge could be rescued with larger training datasets. We found that the ResNet-50 but not ResNet-1522 was able to solve this task when trained on a dataset that was 4x as large, demonstrating that ResNets are very sensitive to overfitting on cABC (and that the 152 needs more than 4x as much data as a ResNet- 18 to learn cABC). Second, we tested whether the ResNet-18, which failed to learn the hard Pathfinder challenge, could be rescued by increasing its capacity. Indeed, we found that a wide ResNet-18, with similar number of parameters as a ResNet-50, was able to perform better (although still far worse than the ResNet-50) on the hard Pathfinder challenge (Fig. S8
Modeling biological feedback Only the H+TD-CNN and human observers solved all levels of the Pathfinder and cABC challenges. To what extent do the visual strategies learned by the H+TD-CNN and other reference models in our experiments match those of humans for solving these tasks? We investigated this with a large-scale psychophysics study, in which human participants categorized exemplars from Pathfinder or cABC that were also viewed by the models (see Appendix D for experiment details).
We recruited 648 participants on Amazon Mechanical Turk to complete a
psychophysics experiments. Participants were given between 800ms- 1300ms to categorize each image in Pathfinder, and 800ms-1600m to categorize each image in cABC (324 per challenge). Each participant viewed a subset of the images challenge, which spanned all three levels of difficulty. The experiment took approximately 2 minutes, and no participant viewed images from more than one challenge.
We gathered 20 human decisions for every image in the Pathfinder and cABC challenges, and measured human performance each image logit of the average accuracy across participants (using average accuracy as a stand-in for "human confidence"). Focusing on the hard-difficulty dataset, we used two-step procedure to measure how much human decision variance each model explained. First, we estimated a ceiling for human inter-rater reliability. We randomly (without replacement) split participants for every image into half, recorded per-image accuracy for each group, then recorded the correlation between groups. This correlation was then corrected for the split-half procedure using a eeeurman-browee correction (Spearman [1910). By repeating this procedure 1,000 times we built a distribution of inter-rater correlations. We took the 95th percentile score of this distribution to represent the ceiling for inter-rater reliability. Second, we recorded the correlation between each model's logits and human logits, and calculated explained variance of human decisions model
as Toammnooooosinn
Both the TD+H-CNN and the H-CNN explained a significantly greater fraction of human decision variance on Pathfinder images than the 50- and 152-layer ResNets, which were also able solve this challenge (Fig. 5c: H-CNN > ResNet 152: p < 0.001; H-CNN > ResNet 50: p < 0.001; TD+H-CNN > ResNet 152: p < 0.01; TD+H-CNN ResNet 50: p< 0.001; p-values derived from bootstrap test, Edgington 1964). We also computed partial correlations between human and model scores on every image, which controlled for model accuracy in the measured association. These partial correlations also indicated that human decisions were significantly more correlated with the TD+H-CNN and H-CNN than they were with either ResNet model (Explained variance/Partial correlation; H-CNN: 0.721/0.487; TD+H-CNN: 0.618/0.425. ResNet-152. 0.426/0.290; ResNet-50: 0.234/0.159).
We repeated this analysis for the cABC challenge. Once again, the recurrent models explained a significantly greater fraction of human decisions than either the feedforward U-Net ResNet 18 that also solved the challenge (Fig. 5: TD-CNN > U-Net: p < 0.01: TD-CNN > ResNet- p 0.001; TD+H-CNN > U-Net: p 0.001; TD+H-CNNY ResNet- p < 0.001). Partial correlations also reflected a similar pattern of associations (Explained variance/Partial correlation; TD+H-CNN: 0.633/0.437, TD-CNN:
0,538/0.376; U-Net: 0. 0.324/0.1900 ResNet- 18: 0.249/0.176).

---

Published as a conference paper at ICLR 2020
6 DISCUSSION
Perceptual grouping is essential for reasoning about the visual world. Although it is known that bottom-up, horizontal and top-down connections in the visual system are important for perceptual grouping, their relative contributions are not well understood. We directly tested long-held theory related to the role of horizontal vs. top-down connections for perceptual grouping by screening neural network architectures connections a priori, we found a dissociation between horizontal vs. top-down feedback connections which emerged from training network architectures for classification. Our study provides direct computational evidence for the distinet roles played by these cortical mechanisms.
controlled synthetic visual tasks. Without specifying any role for feedback
Our study also demonstrates clear limitation of network models that rely solely on feedforward processing, including ResNets of arbitrary depths, which are strained by perceptual grouping tasks that involve cluttered visual stimuli. Deep ResNets performed better the Pathfinder challenge, whereas the shallower ResNet- 18 performed better on the cABC challenge. With more data, deep ResNets performed better cABC; with more width, the ResNet-18 performed better on Pathfinder. We take this pattern of results as evidence that the architectures of feedforward models like ResNets must be carefully tuned on complex visual tasks. Because Pathfinder does not have semantic cues, presents greater computational burden for feedforward models than cABC, making it a better fit for very deep feedforward architectures; because cABC has relatively small datasets, deeper ResNets were quicker to overfit. The highly-recurrent TD+H-CNN was far more flexible and efficient than the reference feedforward models, and made decisions on Pathfinder and cABC images that were significantly more similar to those of human observers. Our study thus adds to a growing body of literature (George et al. 20176 Nayebi al. 2018b Linsley et al. 20T8b a Kar et al.) 2019) which suggests that recurrent circuits necessary to explain complex visual recognition processes.
REFERENCES
W. H. Bosking, Y. Zhang, B. Schofield, and D. Fitzpatrick. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex. J. Neurosci., (166:22122-2127, March 1997.
K. Cho, B. Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN Encoder-Decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 1724-1734, Stroudsburg, PA, USA, 2014a. Association for Computational Linguistics.
K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using RNN Encoder-Decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 1724 1734, Stroudsburg, PA, USA, 2014b. Association for Computational Linguistics.
T. Cooijmans, N. Ballas, C. Laurent, Ç. Gülçehre, and A. Courville. Recurrent batch normalization. In International Conference on Learning Representations, 2017.
S. Eberhardt, J. G. Cader, and T. Serre. How deep is the feature analysis underlying rapid visual categorization? In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1100-1108. Curran Associates, Inc., 2016.
E. S. Edgington. RANDOMIZATION TESTS. J. Psychol., 57:445 April 1964.
F. Fleuret, T. Li, C. Dubout, E. K. Wampler, S. Yantis, and D. Geman. Comparing machines and humans on a visual categorization test. Proc. Natl. Acad. Sci. U. S. A., 108(43):17621--7725, 2011.
Freeman and D. Pelli. Attention can relieve crowding. J. Vis., 70):330-330, March 2010.
J. Freeman, R. Chakravarthi, and D. G. Pelli. Substitution and pooling in crowding. Atten. Percept. Psychophys., 74(2):379-396 February 2012.
10

---

Published as a conference paper at ICLR 2020
D. George, W. Lehrach, K. Kansky, M. Lázaro-Gredilla, C. Laan, B. Marthi, X. Lou, Z. Meng, Y. Liu, H. Wang, A. Lavin, and D. Phoenix. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science, 358(6368), December 2017b.
D. George, W. Lehrach, K. Kansky, M. Lázaro-Gredilla, C. Laan, B. Marthi, X. Lou, Z. Meng, Liu, H. Wang, A. Lavin, and D. S. Phoenix. A generative vision model that trains with high data efficiency and breaks bext-based CAPTCHAS. Science, 358(6368), December 2017a.
C. D. Gilbert and T. N. Wiesel. Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex. J. Neurosci, 977:2432-2442, July 1989.
C. D. Gilbert and W. Li. Top-down influences visual processing. Nat. Rev. Neurosci., 14(5): 350-363, May 2013.
K. Greff, A. Rasmus, M. Berglund, T. Hao, H. Valpola, and J. Schmidhuber. Tagger: Deep unsuper- vised perceptual grouping. D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4484 4492. Curran Associates, Inc., 2016.
K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra. DRAW: A recurrent neural network for image generation. February 2015.
K. He, Zhang. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, 2016.
M. H. Herzog and A. M. Clarke. Why vision is not both hierarchical and feedforward. Front. Comput. Neurosci., 8:135, October 2014.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735-178) November 1997.
R. Houtkamp, Spekreijse, and P. Roelfsema. gradual spread of attention during mental curve tracing. Percept. Psychophys. (67):1136-1144 October 2003.
R. Houtkamp and P. R. Roelfsema. Parallel and serial grouping of image elements in visual perception. Exp. Psychol. Hum. Percept. Perform. 1443-1444,,,59, December 2010.
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. In
C. Cortes, N. D. Lawrence, D). D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, 2017-2025. Curran Associates, Inc., 2015.
P. Jolicoeur, S. Ullman, and M. Mackay. Curve tracing: possible basic operation in the perception of spatial relations. Mem. Cognit., 14(2):129-140, March 1986.
Jolicoeur, S. Ullman, and M. Mackay. Visual curve tracing properties. Exp. Psychol. Hum. Percept. Perform., 17(4):977-1022, November 1991.
K. Kar, J. Kubilius, Schmidt, E. B. Issa, and DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nat. Neurosci. April 2019.
C. Kietzmann, Spoerer, L. Sörensen, and others. Recurrence required to capture the dynamic computations of the human ventral visual stream. arXiv preprint arXiv, 2019.
J. K. Kim, M. Ricci, and T. Serre. Not-So-CLEVR: learning same-different relations strains feedforward neural networks. Interface Focus theme issue on "Understanding images in biological and computer vision". ", 2018.
Kim, Lee, and K. M. Lee. Deeply-Recursiv convolutional network for image Super- Resolution. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1637-1645. IEEE, June 2016.
H.-K. Ko and R. von der Heydt. Figure-ground organization the visual cortex: does meaning matter? J. Neurophysiol, (19(1):160-15) January 2018.
11

---

Published as a conference paper at ICLR 2020
V. a. Lamme and P. R. Roelfsema. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci., 23(1 1):57--579, November 2000.
V. A. Lamme, H. Supër, and H. Spekreijse. Feedforward, horizontal, and feedback processing in the visual cortex. Curr. Opin. Neurobiol., 8(4):529-55) 535, August 1998.
M. Liang and X. Hu. Recurrent convolutional neural network for object recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3367-3375. IEEE Computer Society, June 2015.
Q. Liao and T. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. April 2016.
D. Linsley, J. Kim, and Serre. Sample-efficient image segmentation through recurrence. November 2018a.
D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependen- cies with horizontal gated recurrent units. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018b.
pp. 152-164
D. Linsley, J. Kim, A. Ashok, and T. Serre. Recurrent neural circuits for contour detection. Interna- fional Conference on Learning Representations, 2020.
W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. May 2016.
N. McManus, W. Li, and C. D. Gilbert. Adaptive shape processing in primary visual cortex. Proc. Natl. Acad. Sei. U. S. A., May 2011.
D. A. Mély, D. Linsley, and T. Serre. Complementary surrounds explain diverse contextual phenomena across visual modalities. Psychol. Rev., 125(5):769-784 October 2018.
C. Michaelis, M. Bethge, and A. Ecker. One-Shot segmentation in clutter. J.
J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3549-3558, Stockholmsmässsnn Stockholm Sweden, 2018. PMLR.
S. O. Murray, D. Kersten, B. A. Olshausen, P. Schrater, and D. L. Woods. Shape perception reduces activity in human primary visual cortex. Proc. Natl. Acad, Sci. U. S. A., 15123):15164-11161 November 2002.
S. O. Murray, P. Schrater, and D. Kersten. Perceptual grouping and the interactions between visual cortical areas. Neural Netw,, 17(5-6):695-705, 2004.
A. Nayebi, D. Bear, Kubilius, K. Kar, S. Ganguli, D. Sussillo, J. J. DiCarlo, and D. Yamins. Task- Driven convolutional recurrent models of the visual system. In Neural Information Processing Systems, 2018a
A. Nayebi, D. Bear, J. Kubilius, K. Kar, Ganguli, D. Sussillo, J. DiCarlo, and D. L. Yamins. Task-Driven convolutional recurrent models of the visual system. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett (eds.), Advances in Neural Informa- tion Processing Systems 31, pp. 5295-5306. Curran Associates, Inc., 2018b.
H. C. Nothdurft. Texture segmentation and pop-out from orientation contrast. Vision Res., 31(6): 1073-1078, 1991.
R. C. O'Reilly, D. Wyatte, S. Herd, B. Mingus, and J. Jilk. Recurrent processing during object recognition. Front. Psychol., (AArii))1-14,, 2013.
D. G. Pelli, M. Palomares, and N. J. Majaj. Crowding is unlike ordinary masking: distinguishing feature integration from detection. J. Vis., 4(12):1136-1166 December 2004.
R. Pringle and H. E. Egeth. Mental curve tracing with elementary stimuli. J. Exp. Psychol. Hum. Percept. Perform., 14(4):716-728, November 1988.
12

---

Published as a conference paper at ICLR 2020
D. P. Reichert and T. Serre. Neuronal synchrony in Complex- Valued deep networks. In Int. Conf. Learning Visual Representations, December 2014.
P. R. Roelfsema. Cortical algorithms for perceptual grouping Annu. Rev. Neurosci., 29:203-227, January 2006.
P. R. Roelfsema and R. Houtkamp. Incremental grouping of image elements in vision. Alten, Percept. Psychophys. 73(8)22542-257, November 2011.
P. R. Roelfsema, H. Steven Scholte, and H. Spekreijse. Temporal constraints on the grouping of contour segments into spatially extended objects, 1999.
P. R. Roelfsema, V. A. F. Lamme, H. Spekreijse, and H. Bosch. Figure-Groume Ground segregation in recurrent network architecture. Cogn. Neurosci,, 14(4):525-535 -577, May 2002.
O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. May 2015.
S. Sabbah, J. A. Gemmer, A. Bhatia-Lin, G. Manoff, G. Castro, J. K. Siegel, N. Jeffery, and D. M. Berson. A retinal code for motion along the gravitational and body axes. Nature, 546(7659): 492-497, 497, June 2017.
Sabour, N. Frosst, and G. Hinton. Dynamic routing between capsules. October 2017.
K. E. Schmidt, R. Goebel, S. Löwel, and W. Singer. The perceptual grouping criterion of colinearity is reflected by anisotropies of connections in the primary visual cortex. Eur. J. Neurosci. 9(5): 1083-1089, May 1997.
C. Spearman. Correlation calculated from faulty data. Br. J. Psychol., 3(3):711-2955 October 1910.
C. J. Spocrer, P. McClure, and N. Kriegeskorte. Recurrent convolutional neural networks: A better model of biological object recognition. Front. Psychol., 8:1551, September 2017.
D. D. Stettler, A. Das, J. Bennett, and C. D. Gilbert. Lateral connectivity and contextual interactions in macaque primary visual cortex. Neuron, 36(4):739-750, November 2002.
C. Tallee and Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018.
H. Tang, M. Schrimpf, Lotter, C. Moerman, Paredes, J. Ortega Caro, W. Hardesty, D. Cox, and G. Kreiman. Recurrent computations for visual pattern completion. Proc. Natl. Acad. Sei. U. S. A., 115(35):883-88400 August 2018.
A. M. Treisman and G. Gelade. A feature-integration theory of attention. Cogn. Psychol., 12(1): 97-136, January 1980.
S. Ullman. Visual routines. Cognition, 18(1-3):97-159, December 1984.
Ullman. High-level vision: Object recognition and visual cognition. The MIT Press, Cambridge. MA, 1996.
A. Van Den Oord, N. Kalchbrenner, and K. Kavukeuoglu. Pixel recurrent neural networks. In Pro- ceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1747-1756. New York, NY. USA, 2016. JMLR.org.
S. P. Vecera. Object knowledge influences visual image segmentation. In Proceedings of the 15th annual conference of the Cognitive Science Society, pp. 1040-1045, 1993.
S. P. Vecera and M. J. Farah. visual image segmentation a bottom-up or an interactive process? Percept. Psychophys, 5988):1280-1223 1296, November 1997.
S. P. Vecera and R. C. O'Reilly. Figure-grount organization and object recognition processes: an interactive account. J. Exp. Psychol. Hum. Percept. Perform., (121)441-46 April 1998.
13

---

Published as a conference paper at ICLR 2020
A. Wannig, L. Stanisor, and P. R. Roelfsema. Automatic spread of attentional response modulation along gestalt criteria in primary visual cortex. Nat. Neurosci,, 14(10))1243-1244) October 2011.
H. Wen, K. Han, J. Shi, Y. Zhang, E. Culurciello, and Z. Liu. Deep predictive coding network for object recognition. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5266-5275, Stockholmsmässsan, Stockholm Sweden, 2018. PMLR.
D. Wyatte, D. Jilk, and R. C. O'Reilly Early recurrent feedback facilitates visual object recognition under challenging conditions. Front. Psychol., 5:674, July 2014.
A. R. Zamir, T. Wu, L.. Sun, W. B. Shen, B. E. Shi, J. Malik, and S. Savarese. Feedback networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1808-187,July 2017.
R. S. Zemel, M. Behrmann, M. C. Mozer, and D. Bavelier. Experience-dependent perceptual grouping and object-based attention. J. Exp. Psychol. Hum. Percept. Perform., 28(1):202, 2002.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. November 2016.
14

---

Published as a conference paper at ICLR 2020
A CLUTTERED ABC
The goal of the cluttered ABC (cABC) challenge is to test the ability of models and humans make perceptual grouping judgements based solely on object-level semantic cues. Images in the challenge depict pair of letters positioned sufficiently close each other to ensure frequent overlap. To further minimize local image cues which might permit trivial solutions to the problem, we use fonts as letter images that contain uniform and regular strokes with no distinct decorations. Two publicly available fonts are chosen from the website https: //ww:///www... www. dafont. com: "Futurist fixed-width" and "IIntruction"
Random transformations including affine transformations, geometric distortions, and pixelation are applied before placing each letter on the image to further eliminate local category cues. Positioning and transformations of letters are applied randomly, and their variances are used to control image variability each difficulty level of the challenge.
Letter transformations Although each letter in the cABC challenge is sampled from just two different fonts, we ensure that each letter appear sufficiently varied forms by applying random linear transformations to each letter. We use three different linear transformations in our challenge: rotation, scaling, and shearing. Each type of transformation applied to an image uses three different sets of parameters: two sets that are sampled and applied separately to each letter, and an additional one one that is sampled once commonly applied to both letters. We call the first type of transformation parameters "letter-wise and the second type "common" transformation parameters. For example, rotations applied to the first and the second letter are each described by the sum of letter-wise and common rotational parameters, Φ1 &e and Φ22 the respectively. Scaling is described by the product of letter-wiss wise and common scale parameters, S,S. and SyS., Shearing is described by the sum of letter-wise and common shear parameters, E₁ + Eg and E2 E.. Each shear parameter specifies the value of the off-diagonal shear matrix applied to each letter image. Either vertical or horizontal shear applied to both letters in each image with shear axis randomly chosen with equal probability. The random transformation procedure is visually detailed in Fig. [S
We decompose each transformation to letter-wise and common transformation independently increase variability at the level of an image while keeping letter-wise variability constant. This achieved by gradually increasing variance of the letter-wise transformation parameters while decreasing variance of the common parameters. See for summary of the distributions of random transformation parameters used in the three levels of difficulty.
We apply additional, nonlinear transformations to letter images. First, geometric warping is applied independently to each transforrdnf letter image by computing a "warp template" for each image. A warp template is an image in same dimensions placed Gaussians with sigma (width) of 20 pixels. Each pixel in a letter image is translated by a displacement proportional to the gradient of the warp template in the corresponding location. This process is depicted in Fig. [S2 Second, the resulting letter image is "pixelated" by first partitioning the letter image into grid of 5x5 pixels. Each grid is filled with 255s (white) if and only if more than 30% of the pixels are white, resulting in binary letter image in _ of the original resolution. The squares then undergo random translation with displacement independently sampled for each axis. We use normal distribution with standard deviation of 2 pixels truncated at two standard deviations.
each letter image, consisting of 10 randomly
Letter positioning We sample the positions of letters by first defining an invisible circle of radius placed at the center of the image. Two additional random variables, 0 and A are sampled to specify two angular positions of the letters on the circle. and 0 + A0. r. 0 and AO are sampled from uniform distributions whose ranges increases with difficulty level (Table[ST]. By sampling these parameters from increasingly larger intervals, we increase relative positional variability of letters in harder datasets. In easy difficulty, choose R ~ U(40, 70) and R. U(30, 80), respectively. For and A0, we chose 0₃ ~ U( U(-30, 30) and AO ~ U(170, 190) for the easy difficulty, 01 U(-60, 60) and AA ~ U(110, 250) for the intermediate difficulty, and 0₁ ~ U(-90, 90) and A ~ U(70, 290) for the hard difficulty.
choose U(50, 60). In intermediate and hard difficulties,
Letter images placed on the sampled positions by aligning the positions of each letter's "center of mass", which is defined by the average (x, y) coordinate of all the foreground pixels in each letter image, with each of the sampled coordinates in image.

---

Published as a conference paper at ICLR 2020
ABC
Font, Letter
G
Rotate & Shear
Scale
Warp
Pixelate
A BC
F
jS, Sa. Sa!
(ir.. tha Pal (E, Es. Es!
Position 1st letter
Position 2nd letter
Place markers
Radius
Render
A0
Visual depiction of the cABC image generation algorithm. (Top row) Generation procedure of individual letter images. Two independently sampled letter categories are rendered in randomly sampled (but same) font. Each letter image undergoes three linear transformations scale, rotation and shear well two steps of nonlinear transformations warp and pixelation. Transformation magnitudes are determined by random variables whose distributions differ between levels of difficulties. (Bottom row) Positional sampling procedure. Two letters are placed on two randomly chosen points on an invisible circle of radius т. Two angular positions on the circle are sampled by first sampling the position of the first letter, 0, and then sampling the angular displacement of the second letter relative to the first letter, A6. Each letter is positioned by aligning its "center of mass", which is the average coordinate white pixels letter image, with the coordinate of the sampled position on the circle.
Figure S1:
Letter image
Warp mask
Warped
Figure S2: Two example letter images undergoing geometric warping. "warp template" for each image is sampled by randomly overlaying 10 Gaussian images. Each pixel in letter image is translated by displacement proportional the gradient of its warp template in the corresponding location.
2

---

Published as a conference paper at ICLR 2020
Parameters
Easy
Intermediate
Hard
0 (degrees)
U[ U(-30.30 30)
U(-60, 60, 60)
U[-90, 90)
Ad (degrees)
U(170, 180)
U(110, 180)
U(70, 180)
(pixels)
U 30)
U120, 35)
U(15,40)
11, @2 (degrees)
(o,
(0)
No X6
de (degrees)
(0,)30)
S1, S2
logA 3y2
(1.5,,
log.(1.5, 0,
logA((1.5, 2/22)
E1,
0
A(, moy
N(0, :
E.
(O., )
A(00, 1007
Table S1: Distributions of transformation parameters used in cluttered ABC challenge. U(a, b) denotes uniform distribution with range (a, ((,, or) denotes normal distribution with mean and standard deviation o; logN(z, o) denotes log-normal distribution with base z, mean and standard deviation o.
PE
Luminance control
Position control
Figure S3: Two control cABC challenges. In luminance control, two letters rendered in different, randomly sampled pixel intensity values. In positional control, two letters are always rendered without touching overlapping each other.
Input
Output
Input
Output
Pathfinder
Cluttered ABC
Figure $4: Segmentation version the two challenges. Here, a model is tasked with producing as output an image which contains only the object which is marked in the input image.
B ADDITIONAL EXPERIMENTS
Control cABC challenges To further validate our implementation of the cABC, we added two control cABC challenges our study (Fig (S3). In the "luminance control" challenge, two letters are rendered with a pair of uniformly sampled random pixel intensity values that are greater than 128 but always differ by least 40 between the letters. In the "positional control" challenge, the letters are disallowed from touching overlapping. Relative to the original challenge, the two controls provide additional local grouping cues with which determine the extent of each letter. The luminance control challenge permits additional pixel-level cue with which a model can infer image category by comparing the values of letter pixels surrounding two markers. Positional control challenge provides unambiguous local Gestalt cue solve the task. Ensuring that the two letters spatially separated has an additional effect of minimizing the amount clutter to interfere the feature extraction process in each letter.
In both control challenges, we found no straining effect in any the models we tested. This strengthens initial assumption behind the design of cABC that the only way to efficiently solve the default cABC challenge is to employ object- based grouping strategy. Because all networks successfully solved the control challenges, the relative difficulty suffered by the H-CNN reference
3

---

Published as a conference paper at ICLR 2020
Max Validation Accuracy
0.9
(cABC,
Difficulty
Positive
Negative
0.5
Hard
Position Control
(a)
(b)
HCM Cn
BU-CM
Rasker
Berten
U-it
Max Validation Accuracy
0.9
0.8
Difficulty
Positive
Negative
0.5
Har
(c)
Luminance Control
(d)
Figure S5: Performance on two cABC-control challenges. Top-down feedback is critical when local grouping cues are unavailable. We verify this by constructing two controlled versions of the cABC challenge, where local Gestalt grouping cues are provided in the form of spatial segregation between two letters (a), or in the form of luminance segregation (b). In both cases, the architectures that lack top-down feedback mechanism can solve the challenge at all levels of difficulties.
Max Validation Accuracy
0.9
IGRU -TD+H--ON
0.8-
LSTH
0.7
0.6-
Difficulty
Easy
Pathfinder
cABC
Figure S6: fGRUs outperform GRUs and LSTMs on Pathfinder and cABC. Our recurrent hier- archical architectures performed better on Pathfinder and cABC challenges when they used fGRUs instead of either LSTMs (Hochreiter & Schmidhuber 1997) or GRUs (Cho et al. 2014a).
feedforward networks on the original cABC couldn't have been caused by idiosyncracies of image features we used such the average size of letters. These networks lack the mechanisms to separate the mutually occluding letter strokes according to high-level hypothesis about the categories letters present in each image.
Segmentation challenges To further understand and visualize the nature of computations employed by our recurrent architectures when solving the Pathfinder and the cABC challenge, we construct variant of each dataset where we pose segmentation problem. Here, we modify each dataset by placing only one marker per image instead of two. The task is to output a per-pixel prediction of the object tagged by the marker (Figure (S4). We train each model using 10 thousand images of the cABC challenge and 40 thousand images of the Pathfinder challenge and validate using 400 images both challenges. We only use hard difficulty each challenge. We use only the TD- and H-CNN in this experiment. We replace the readout module in each of our recurrent architecture by two 1x 1 convolution layers with bias and rectification after each layer. We also include the "reference" recurrent architectures in which the fGRU modules have been replaced by either LSTM Hochreiter & Schmidhuber (1997) or GRU et al. (2014a). Because the U-Net is dense prediction architecture by design, simply remove the readout block that used in the main categorization challenge.
4

---

Published as a conference paper at ICLR 2020
Max Validation Accuracy
1.0
Eaoy
tnt.
Hard
0.9
0.8
TD Cn-
0.7
0.6
Pathfinder
cABC
Figure S7: Recurrence improves fGRU model performance on Pathfinder and cABC. Each of our recurrent models performed better when given 8-timesteps of recurrent processing instead of timestep.
Max Validation Accuracy
1.0
TD+H-CNN
(Pathfinder)
0.9
0.8
X
0.7
ResNet-18
0.6
0.5
Normal
Max Validation Accuracy
1.0
(Cluttered ABC)
0.9
x
X Deeper
0.8
0.7
0.6
Fasy lh selt
Wider
0.5
10
100
1000
Free parameter multiplier (relative H-CNN)
Figure S8: A comparison of different parameterizations of H+TD-CNN and 18 models on Pathfinder and cABC. (a) We compared the original H+TD-CN to a deeper version (5 instead of convolutional blocks between its low- and high-level fGRU modules). This deeper H+TD-CNI performed similarly to the original version. We also tested whether widening the ResNet- 18 could rescue its performance Pathfinder. Indeed, the wider ResNet-18 18 (matched in number of parameters to ResNet-50) performed far better on Pathfinder than the original version, but was still far worse on the task than normal ResNet-50. (b) Performance of each of these models plotted as a function of their total free parameters. Variants of the TD+H-CNN are far more parameter efficient at solving Pathfinder and cABC than variants ResNet-18 18. Ellipses group variants of each model class.
We found that the target curves in the Pathfinder challenge best segmented by H-CNN (0.79 fl score), followed by H-CNN with GRU (0.72) and H-CNN with LSTM (0.69). No top-down-only architecture was able to successfully solve this challenge (we set a cutoff of greater than 0.6 fl). The CABC challenge is best segmented by the TD-CNN (0.83), followed by TD-CNN with LSTM (0.64) and TD-CNN with GRU (0.62). Consistent with the main challenge, no purely horizontal architecture able successfully solve this challenge. In short, were able reproduce the dissociation in performance we found between horizontal and top-down connections other recurrent architectures (LSTMM and GRU) as well, although these recurrent modules were less successful than the fGRU.

---

Published as a conference paper at ICLR 2020
We visualize the internal update performed by recurrent networks by plotting the difference of norm of each per-pixel feature vector at every timestep in the low-level recurrent state, H (closest to image resolution), vs. its immediately preceding timestep. Of course, this method does not reveal the full extent of the computations taking place the IGRU, but least serves as proxy for inferring the spatial layout of how activities evolve over timesteps 4D)) More examples of both sucessful and mistaken segmentations and the temporal evolution of internal activities of H-CNN (on the Pathfinder challenge) ID-CNN (on the cABC challenge) are shown in Fig SIO and Fig S11
Using conventional recurrent layers We tested the performance of modifications of TD+H-CNN by replacing the IGRU modules with conventional recurrent modules LSTMs (Hochreiter & Schmidhuber 1997) and GRUs (Cho et al. 2014a). We found that neither of these architectures matched the performance of our fGRU-based architecture either grouping challenges [S6] The degradation of performance was most pronounced in Pathfinder. Similar to the findings in Linsley believe that the use of both multiplicative and additive interactions well as the
(et al. (2088)), separation of the suppressive and facilitatory stages in fGRU are critical for learning stable transitive grouping in Pathfinder.
C NETWORK ARCHITECTURES
C.1 THE FGRU MODULE
At timestep t, the fGRU module takes two inputs, an instantancous external drive X E |xxx and persistent recurrent state H[t E =xl and produces as oulput an updated recurrent state H Detailed description of its internal operation over single iteration from [t - 1] to [t] is defined by the following formulae. For clarity, tensors are bolded but kernels and parameters are not:
G' sigmoid(BN(U' Hit 1)))
C BN(W' * [
Z = X [
G sigmoid(BN(U** * Z))
C BN(W + zZ)
(I)
H = [K(C. + Z) + w Z)
H = (1 - G H -1] G @ H
Er
where (N(r;6,v)
8
Var(r) +
The evolution of the recurrent state can be broadly described by two discrete stages. During the first stage, H is combined with an extrinsic drive X. Interactions between units in the hidden state are computed in C² by convolving Hit - 1] with kernel W These interactions are next linearly and multiplicatively combined with H - 1] via the k-dimensional learnable coefficients μ and a to compute the intermediate "suppressed" activity, Z.
The second stage involves updating with a transformation of the intermediate activity Z. Here, convolved with the kernel W to compute E A candidate output H[{ is calculated via the k-dimensional learnable coefficients x and which control the linear and multiplicative terms of self-interactions between - and Z.
Two gates modulate the dynamics of both of these stages: the "gain" G [t), which modulates channels in Hit 1] during the first stage, and the "mix" G G which mixes a candidate H[t) with the persistent Hit during the second stage. Both the gain and mix transformed into the range [0, by a sigmoid nonlinearity. This two-stage computational structure in the fGRU captures complex nonlinear interactions between units in X and H. Brackets [-] denote linear rectification. Separate

---

Published as a conference paper at ICLR 2020
applications of batch-norm used every timestep, where E R is the vector of activations that will be normalized. The parameters 6, V E IR'' control the scale and bias of normalized activities, 17 is regularization hyperparameter, and @ is elementwise multiplication.
Learnable gates, like those in the IGRU, support RNN training. But there are multiple other heuristics that also help optimize performance. We use several heuristics to train models, including Chronos initialization of fGRU gate biases (Tallec & Ollivier (018). We also initialized the learnable scale parameter of IGRU normalizations to 0.1, since values near O optimize the dynamic range of gradients passing through its sigmoidal gates (Cooijmans et al. 2017). Similarly, fGRU parameters for learning additive suppresioonffaciitttionn (μ, K) were initialized to 0, and parameters for learning multiplicative suppression/faciltaaii
(a, w) were initialized to 0.1. Finally, when implementing top- down connections, we incorporated an extra learnable gate, which we found improved the stability of training. Consider the horizontal activities in two layers, H _ and the function fGRU, which implements top down connections. The introduction of this extra gate means that top-down connections are learned as: H (1 sigmoid((3) @ (H))(H()) H((+)) + sigmoid( (3) H()
where 3 E R is initialized to O and learns how to incorporate/ignon top-down connections.
The broad structure of the IGRU is related the Gated Recurrent Unit (GRU, (Cho et al. 2014b)
Nevertheless, one of the main aspects in which the fGRU diverges from the GRU is that its state update is carried out via two discrete steps of processing instead of one. This is inspired by computational neuroscience model of contextual effects in visual cortex (see Mély et al. 2018 for details). Unlike that model, the FGRU allows for both additive and multiplicative combinations following each step of convolution, which potentially encourages a more diverse range of nonlinear computations to be learned (Linsley et al. 2018b
C.2 ARCHITECTURE DETAILS
Using the fGRU module, we constreut three recurrent convolutional architectures as well as one feedforward variant. First, the H+TD-CNN (Fig. (9h) is equipped with both top-down and horizontal connections. The TD-CNN (Fig. S9p) constructed by selectively lesioning the horizontal connec- tions in the +++T-CNNY The H- H-CNN (Fig, (99) is constructed by selectively lesioning the top-down connections in the H+TD-CNN, essentially making recurrence only take place within the low-level FGRU. Lastly the BU-CNN (Fig. (Fig is constructed by disabling both types of recurrence, turning our recurrent archecture into a feedforward encoder-decoder network. All experiments were run with NVIDIA Titan X GPUs.
TD+H-CNN Our main recurrent network architecture, TD+H-CNN, built on convolutional and munspose-convolutione "backbone". An input image X is first processed by "Conv" block which consists of two convolutional layers of 7x7 kernels with 20 output channels and ReLU activation applied between. The resulting feature activity is sent to the first fGRU module (FGRU which is equipped with 15x15 kernels with 20 output channels carry out horizontal interactions. (GR
iteratively updates its recurrent state, (!)) The output undergoes a batch normalization and a pooling layer with a 2xx pool kernel with 2x2 strides which then passes through the downsampling block consisting of two convolutional "stacks" (Fig. S9p, The gray box named "DS"). Each stack consists of three convolutional layers with 3x3 kernels, each followed by a ReL.U activations and batch normalization layer. The convolutional stacks increase the number of output channels from 20 to 32 to 128, respectively, while they progressively downsample the activity via 2x2 pool-stride layer after each stack. The resulting activity is fed to another fGRU module in the top layer, (GRU(22, where it is integrated with the higher-level recurrent state, (2). The kernels in (2 unlike (1 is strictly local with 1x1 filter size. This essentially makes f2RU(2) persistent memory module without performing any spatial integration over timesteps. The output of this fGRU module then passes through the "upsampling" block which consists of two transpose convolutional stacks (Fig. S99, The gray box named "US").. Each stack consists of one transpose convolutional layer of 4 kernels. Each stack also reduces the number of output channels from 128 to 32 to 12, respectively, while it progressively upsamples the activity via a 2x2 stride. Each transpose convolutional layer is followed by ReLU activations and batch normalization layer. The resulting activity, which now has the same shape the output of the initial FGRU, is sent to the third fGRU module (FG where it is integrated with the recurrent state of (GRU(1) (1)) This module plays the role of integrating
7

---

Published as a conference paper at ICLR 2020
top-down activity (originating form "US") and bottom-up activity (originating from fGRU via 1x1 1 kernels. To summarize, FGRU(1) in our architecture implements horizontal feedback in the first layer while the FGRU(3) implements top-down feedback.
The above steps repeat over 8 timesteps, and in the 8th timestep, the final recurrent activity from FGRU(1), H (2), is extracted from the network and passed through batch normalization. The resulting activity is processed by the "Readout" block which consists of two convolutional layers of x kernels with a global max-pooling layer between. Each convolutional layer has one output channel and uses ReLU activations.
Lesioned architectures Our top- dop-down-only architecture, the TD-CNN, is identical the TD+H- CNN, but its horizontal connections have been disabled by replacing the spatial kernel of (GRU(1) with 1 kernel, thereby preventing spatial propagation of activities within the fGRU altogether. Thus, its recurrent updates in the low-level layer are only driven by top-down feedback (in fGRU()). The H-CNN is implemented by disabling FGRU(3), This effectively lesions both the downsampling and upsampling pathways entirely from contributing to the final output. As a result, its final category decision is determined entirely by low-level horizontal propagation of activities in the netwrok.
Lastly, we construct the purely feedforward architecture, called the BU-CNN, by lesioning both horizontal and top-down feedback. It replaces the spatial kernel fGRU(") by a 1 x 1 kernel and disables (GRU(3), By running for only one timestep, the BU-CNN is equivalent to a deep convolutional encoder-decoder network.
Feedforward reference networks Because U-Nets (Ronneberger et al. 2015) are designed for dense image predictions, we attach the same readout block that we use to decode fGRU activities (H to the output layer of a U-Net, which pairs a VGG16 as its downsampling architecture, with upsampling convolutional layers transform its encodings into the same resolution the input image.
D HUMAN EXPERIMENTS
We devised psychophysics categorization experiments understand the visual processes underlying the ability to solve tasks that encourage perceptual grouping.
These visual categorization experiments adopted paradigm similar to Eberhardt et al. (2016), where stimuli were flashed quickly and participants had to respond within a fixed period of time. We conducted separate experiments for the "Pathfinder" and the "cluttered ABC" challenge datasets. Participants had several possible response time (RT) widows for responding to stimuli in each dataset. For "Pathfinder", we considered (800, 1050, 1300} ms RT windows, while for the "cluttered ABC". we consider {800, 1200, 1600} ms. We recruited 324 participants for each experiment (648 overall) from Amazon Mechanical Turk (www. mturk com).
The "Pathfinder" and the "cluttered ABC" are binary categorization challenges with "Same" and "Different" the possible responses. For both experiments, we present 90 stimuli each participant and assign a single RT to that participant. These 90 stimuli consist of three sets of 30, corresponding the three level of difficulties present in the challenge datasets. We randomize the order of the three difficulties (sets of 30 stimuli) across participants as well as the assignment of keys to class "Same" and "Different". Participant accuracy is shown in Fig.
Stimulus generation We generated a short video for each stimuli image from the challenge dataset based on the response time allowed from the onset of stimuli. We generated three stimulus videos for each challenge image, correponding to the three different response times chosen. In each stimulus video, we included image of cross (black plus sign overlaid on a white background) for 1000 ms before the onset of stimulus that shown for RT ms. The stimuli were stored as webm videos and presented for 1000+RT ms during the experiments. We created video that the cross image and the stimulus image are seamlessly combined and shown without delays, which would otherwise be caused by loading their separate images in the browser, which introduces noise in stimulus timing.
8

---

Published as a conference paper at ICLR 2020
TGRUR
IGRU
TGRU
Comv-pool
Transpose Conv
Conv-pool
Transpose Conv
Conv-pool
Transpose Conv
TGRUm
IGRUP
IGRU
IGRU
1GRU
fGRU
Conv
Conv
Conv
Readout
X
X
X
Y
t=
t 2,
t 8
(a) TD+H-CNN
1GRU
Y
(GRUP
(GRU
Readout
Conv
Readout
US
X
1GRU
(b) TD-CNN
DS
1GRU
Conv
Conv
Readout
X
(d) BU-CNN
X
X
Y
(c) H-CNN
Figure S9: Our recurrent convolutional architectures. Feedforward or bottom-up connections (gray) pass information from lower-order to higher-order layers; horizontal connections (green) pass information between units in the same layer separated by spatial location and/or channels; top-down connections (red) pass information from higher-order to lower-order layers. We use "feedback" umbrella term for both top-down and horizontal connections as these can only be implemented in model with recurrent activity that supports incremental updates model units.
We selected set of 200 images per difficulty both challenges (600/challenge). Each participant responded to 30 images from each difficulty. We collected responses to the 600 images in each challenge from 18 different participants.
Psychophysics experiment We conducted separate experiments for: (1) Pathfinder and (2) cABC images. The image dataset used for each experiment consisted of three difficulties casy, intermediate and hard. Participants were assigned three blocks trials, each having stimuli from a single difficulty level. The order of these three blocks is randomized and balanced across participants. The keys for responses to the stimuli are set to be + and Their assignment to "Same" and "Different" class label randomized across participants well.
Each experiment began with a training phase, where participants responded to 6 trials without a time limit. This phase ensured that participants understood the task and the type of image stimuli. Next,

---

Published as a conference paper at ICLR 2020
participants went through 15 trials, 5 from each difficulty level, where they responded by pressing + or within a limited response period. For this set of images, participants were given feedback on correct or incorrec.. After this block of "training" trials, each participant
whether their response completed 90 "test" trials and responded with either "Same" or "Different" in the stipulated amount of response time, without feedback their performance. Participants were given time to and the difficulty level of the viewed images changed after every 30 trials. Before starting new block of trials, representative image from that difficulty level was shown and instructions were given to respond as quickly as possible.
Experiments were implemented using jsPsych and custom javascript functions. We used the webm format to ensure fast loading times. Videos were rendered at 256 x 256 pixels.
Processing Responses To generate the final results from human responses on the two categorization tasks, we filtered responses rendered in less than 450 ms, which we deem unreliable responses following the criteria of Eberhardt et al. (2016).
10

---

Published as a conference paper at ICLR 2020
Recurrent activities
Image
Label
Predicted
1=2
t=7
yand
Successful
Mistakes
Figure S10: Activity time-courses for the Pathfinder challenge (Hard).
11

---

Published as a conference paper at ICLR 2020
Recurrent activities
Image
Label
Predicted
t=1
t=2
t=3
t=4
f=5
f=6
t=7
Successful
D
F
c
Mistakes
Figure S11: Activity time-courses for the cABC challenge (Hard).
12


Reviews:
[{"review_id": "ryesUNO59H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper. It seeks to disentangle the need for top-down and horizontal connections for grouping tasks using (a) a new synthetic dataset that seeks to evaluate one over the other, and (b) a new recurrent neural network model.\n\n\nI think this is an interesting scientific question that is worth answering. It is especially useful given the context of what we know about the visual cortex in the brain, which is the presence of a large number of horizontal and top-down connections.\nI think this paper takes a good first step in understanding this. I liked the problem setup and the approach of looking at accuracy at a given sample complexity rather than accuracy alone. I also liked the fact that the authors used both deeper networks and state-of-the-art baselines and corrected for the parameter count. In these respects the paper is novel and thought-provoking.\n\nI would like the evaluation to be stronger, however. I would like to see the following experiments: \n(1) How do variants of the architecture (deeper/shallower) perform under the same settings? Do the conclusions change with network depth?\n(2) How do these results generalize to real-world segmentation datasets? Are both top-down and horizontal connections needed for e.g., PASCAL/COCO?\n\nSince this is not a vision conference, I am giving this a weak accept, but I would really like to see at least an evaluation of (2).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkeA5ZK6FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article tries to examine existing hypotheses from the neuroscience and perception literature by using neural networks as a computational model of the brain. Namely, the authors assess the efficiency of different strategies for solving two visual challenges, one of which is novel. The authors also evaluate the level of consistency between the performance of humans and different types of neural architectures.\n\nI believe that the quality of this work is above the acceptance threshold. The results seem to clearly support the claims. The experiments are well-designed and an adequate number of baselines are provided. However, it should be mentioned that the conclusions are by no means surprising.\n\nThe following sentence should probably be fixed:\n> \".. models that not learn overfit the training set.\"\n\nSome questions (answering is optional):\n -  In the second paragraph of the introduction, the authors state that the two feedback mechanisms should be iterative. Can the authors provide elaborate as to why these strategies should be inherently iterative and simply applying the same model a small/finite amount of times is not enough?\n - The authors claim that the relatively low performance of ResNet-18 and U-Net on Pathfinder is due to a higher computational burden, yet the reason for the poor performance of ResNet-50 and ResNet-152  on cABC is the result of overfitting. Is there any evidence to support this distinction or are the authors simply arguing this because it is the most plausible explanation?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1xpB_j6KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a dataset and designed a relevant network structure to analyze the function of horizontal and top-down connections for perceptual grouping. The used two datasets smartly isolate the requirements for exploiting Gestalt cues and object-based strategies. Appendix A detailed describes the cABC dataset, and the control experiments in Appendix B further validate the designing of the cABC. The proposed network flexibly integrates three types of connections and successfully solves both two challenges. The visualization results in Figure 4, S8 and S9 are insightful and also validate the intuitions. Overall the paper is clearly written and easy to follow.\n\n\n\nAlthough the use of different types of connections in the proposed network is clear, I think the author should conduct more analysis for the standard networks and datasets.\n\nIn section 3, the author claims that the two challenges will cause a high computational burden for feedforward models like ResNets. However, in Figure 5, the results show that ResNet-152 and ResNet-50 could easily solve the Pathfinder. For cABC, ResNet-18 could solve and ResNet-50 and ResNet-152 will fail while the author illustrates this is due to ResNet-50 and ResNet-152 overfit to cABC. Those phenomena are quite misaligning with the arguments. \n\n- Can we find a suitable ResNet structure that could solve both two challenges? \n- Can we increase the depth of U-Net and make it solve both two challenges? \n- Can we add more data or increase the difficulty for cABC and prevent the overfitting for ResNet-50 and ResNet-152? \n- Can we keep increasing the difficulties of cABC and straining ResNet-18 and UNet? Does that difficulty level would also strain TD+H-CNN?  \n- We could also perform similar experiments on Pathfinder that increasing its difficulty for ResNet-50 and ResNet-152. \n\nAlso, if we train the TD+H-CNN on standard image datasets such as cifar-10 or ImageNet, what about the recurrent activities and the final acc?\n\n=========================================================\nAfter Rebuttal:\n\nI thank the author for the response. \n\nI hope the comments are useful for preparing a future version of this work when you have enough time.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers recommend acceptance. The reviews found the paper to be interesting with substantial insights. 
Decision: Accept (Spotlight)