Disentangling neural mechanisms for perceptual grouping

1 INTRODUCTION . The ability to form perceptual groups and segment scenes into a discrete set of objects constitutes a fundamental component of visual intelligence . Decades of research in biological vision have suggested that biological visual systems extract these objects through visual routines for perceptual grouping , which rely on feedforward ( or bottom-up ) and potentially feedback ( or recurrent ) processes ( Roelfsema , 2006 ; Roelfsema & Houtkamp , 2011 ; Wyatte et al. , 2014 ) . Feedforward processes group scenes by encoding increasingly more complex feature conjunctions through a cascade of filtering , rectification and normalization operations . As shown in Fig . 1a , these computations can be sufficient to detect and localize objects in scenes with little or no background clutter , or when an object “ pops out ” because of color , contrast , etc . ( Nothdurft , 1991 ) . However , as illustrated in Fig . 1b , visual scenes are usually complex and contain objects interposed in background clutter . When this is the case , feedforward processes alone are often insufficient for perceptual grouping ( Herzog & Clarke , 2014 ; Pelli et al. , 2004 ; Freeman et al. , 2012 ; Freeman & Pelli , 2010 ) , and it has been suggested that our visual system leverages feedback mechanisms to refine an initially coarse scene segmentation ( Ullman , 1984 ; Roelfsema & Houtkamp , 2011 ; Treisman & Gelade , 1980 ; Lamme & Roelfsema , 2000 ) . To rough approximation , there are two distinct types of perceptual grouping routines that are associated with feedback computations . One routine involves grouping low-level visual features with their neighbors according to Gestalt laws like similarity , good continuation , etc . ( Fig . 1b , top ; Jolicoeur et al. , 1986 ; 1991 ; Pringle & Egeth , 1988 ; Roelfsema et al. , 1999 ; Houtkamp & Roelfsema , 2010 ; Houtkamp et al. , 2003 ; Roelfsema et al. , 2002 ) . Another routine is object-based and mediated by †These authors contributed equally to this work . ‡DeepMind , London , UK . high-level expectations about the shape and structure of perceptual objects . In this routine , feedback refines a coarse initial feedforward analysis of a scene with high-level hypotheses about the objects it contains ( Fig . 1b , bottom ; Vecera & Farah , 1997 ; Vecera & O ’ Reilly , 1998 ; Vecera , 1993 ; Zemel et al. , 2002 ) . Both of these feedback routines are iterative and rely on recurrent computations . What are the neural circuits that implement Gestalt vs. object-based routines for perceptual grouping ? Visual neuroscience studies have suggested that these routines emerge from specific types of neural interactions : ( i ) horizontal connections between neurons within an area , spanning spatial locations and potentially feature selectivity ( Stettler et al. , 2002 ; Gilbert & Wiesel , 1989 ; McManus et al. , 2011 ; Bosking et al. , 1997 ; Schmidt et al. , 1997 ; Wannig et al. , 2011 ) , and ( ii ) descending top-down connections from neurons in higher-to-lower areas ( Ko & von der Heydt , 2018 ; Gilbert & Li , 2013 ; Tang et al. , 2018 ; Lamme et al. , 1998 ; Murray et al. , 2004 ; 2002 ) . The anatomical and functional properties of these feedback connections have been well-documented ( see Gilbert & Li 2013 for a review ) , but the relative contributions of horizontal vs. top-down connections for perceptual grouping remains an open question . Contributions Here we investigate the function of horizontal vs. top-down connections during perceptual grouping . We evaluate each network on two perceptual grouping “ challenges ” that are designed to be solved by either Gestalt or object-based visual routines . Our contributions are : • We introduce a biologically-inspired deep recurrent neural network ( RNN ) architecture featuring feedforward and feedback connections ( the TD+H-CNN ) . We develop a range of lesioned variants of this network for measuring the relative contributions of feedforward and feedback processing by selectively disabling either horizontal ( TD-CNN ) , top-down ( H-CNN ) , or both types of connections ( BU-CNN ) . • We introduce the cluttered ABC challenge ( cABC ) , a synthetic visual reasoning challenge that requires object-based grouping strategies ( Fig . 1c , bottom-right ) . We pair this dataset with the Pathfinder challenge ( Linsley et al . 2018b , Fig . 1c , top-right ) which features lowlevel Gestalt cues . Both challenges allow for a parametric control over image variability . With these challenges , we investigate whether each of our models can learn to leverage either object-level of Gestalt cues for grouping . • We find that top-down connections are key for grouping objects when semantic cues are present , whereas horizontal connections support grouping when they are not . Of the network models tested ( which included ResNets and U-Nets ) , only our recurrent model with the full array of connections ( the TD+H-CNN ) solves both tasks across levels of difficulty . • We compare model predictions to human psychophysics data on the same visual tasks and show that human judgements are significantly more consistent with image predictions from our TD+H-CNN than ResNet and U-Net models . This indicates that the strategies used by the human visual system on difficult segmentation tasks are best matched by a highly-recurrent model with bottom-up , horizontal , and top-down connections . 2 RELATED WORK . Recurrent neural networks Recurrent neural networks ( RNNs ) are a class of models that can be trained with gradient descent to approximate discrete-time dynamical systems . RNNs are classically featured in sequence learning , but have also begun to show promise in computer vision tasks . One example of this includes autoregressive RNNs that learn the statistical relationship between neighboring pixels for image generation ( Van Den Oord et al. , 2016 ) . Another approach , successfully applied to object recognition and super-resolution tasks , is to incorporate convolutional kernels into RNNs ( O ’ Reilly et al. , 2013 ; Liang & Hu , 2015 ; Liao & Poggio , 2016 ; Kim et al. , 2016 ; Zamir et al. , 2017 ) . These convolutional RNNs share kernels across processing timesteps , allowing them to achieve processing depth with a fraction of the parameters needed for a CNN of equivalent depth . Such convolutional RNN models have also been used by multiple groups as a foundation for vision models with biologically-inspired feedback mechanisms ( Lotter et al. , 2016 ; Spoerer et al. , 2017 ; Kietzmann et al. , 2019 ; Wen et al. , 2018 ; Linsley et al. , 2018b ; Nayebi et al. , 2018a ; Kar et al. , 2019 ) . Here , we construct convolutional-RNN architectures using the feedback gated recurrent unit ( fGRU ) proposed by Linsley et al . ( 2018a ) . The fGRU extends the horizontal gated recurrent unit ( hGRU , Linsley et al . 2018b ) , to implement ( i ) horizontal connections between units in a processing layer separated by spatial location and/or feature channel , and ( ii ) top-down connections from units in higher-to-lower network layers . The fGRU was applied to contour detection in natural and electron microscopy images , where it was found to be more sample efficient than state-of-the-art models ( Linsley et al. , 2020 ) . We leverage a modified version of this architecture to study the relative contributions of horizontal and top-down interactions for perceptual grouping . Synthetic visual tasks There is a long history of using synthetic visual recognition challenges for evaluating computer vision algorithms ( Ullman , 1996 ; Fleuret et al. , 2011 ) . There is a popular dataset of simple geometric shapes used to evaluate instance segmentation algorithms ( featured in Reichert & Serre 2014 ; Greff et al . 2016 ) , and variants of the cluttered MNIST dataset and CAPTCHA datasets have been helpful for testing model tolerance to visual clutter ( e.g. , Gregor et al . 2015 ; Sabbah et al . 2017 ; Spoerer et al . 2017 ; George et al . 2017a ; Jaderberg et al . 2015 ; Sabour et al . 2017 ) . Recent synthetic challenges have sought to significantly ramp up task difficulty ( Michaelis et al. , 2018 ) , and/or to parameterically control intra-class image variability ( Kim et al. , 2018 ; Linsley et al. , 2018b ) . The current work follows this trend of developing novel synthetic stimuli that are tightly controlled for low-level biases ( unlike e.g. , MNIST ) , and to design parametric controls ( e.g. , intra-class variability ) for task difficulty to try to strain network models . In the current study , we begin with the “ Pathfinder ” challenge ( Linsley et al. , 2018b ) , which consists of images with two white markers that may – or may not – be connected by a long path ( Fig . 2 ) . The challenge illustrated how a shallow network with horizontal connections could recognize a connected path by learning an efficient incremental grouping routine , while feedforward architectures ( CNNs ) with orders of magnitude more free parameters struggled to learn the same task . Our novel “ cluttered ABC ” ( cABC ) challenge complements the Pathfinder challenge by posing the same task on images that feature semantic objects rather than curves . Thus , cABC may be difficult to solve using the same incremental grouping routine as Pathfinder , which could capture both objects in a single group ( Fig . 2 ) . 3 SYNTHETIC PERCEPTUAL GROUPING CHALLENGES . Pathfinder and cABC challenges share a key similarity in their stimulus design : white shapes are placed on a black background along with two circular and white “ markers ” ( Fig . 2 ) . These markers are either placed on two different objects or the same object , and the task posed in both datasets is to discriminate between these two alternatives . The two challenges depict different types of objects . Images in the Pathfinder challenge contain two flexible curves , whereas the cABC challenge uses overlapping English-alphabet characters that are transformed in appearance and position . Local vs. object-level cues for perceptual grouping Differences in the types of shapes used in the two challenges make them ideally suited for different feedback grouping routines . The Pathfinder challenge features smooth and flexible curves , making it well suited for incremental Gestalt-based grouping ( Linsley et al. , 2018b ) . Conversely , the English-alphabet characters in the cABC challenge make it a good match for a grouping routine that relies on semantic-level object cues . The two synthetic challenges are designed to cause a high computational burden in models that rely on sub-optimal grouping routines . For instance , cluttered , flexible curves in the Pathfinder challenge are well-characterized by local continuity but have no set “ global shape ” or any semantic-level cues . This makes the task difficult for feedforward models like ResNets because the number of fixed templates that they must learn to solve it increases with the number of possible shapes the flexible paths may take . In contrast , letters in cABC images are globally stable but locally degenerate ( due to warping and pixelation ; see Appendix A for details of its design . ) , and letters may overlap with each other . Because the strokes of overlapping letters form arbitrary conjunctions , a feedforward network needs to learn a large number of templates to discriminate real from spurious stroke conjunctions . Overall , we consider these challenges two extremes on a continuum of perceptual grouping cues found in nature . On one hand , Pathfinder curves provide reliable Gestalt cues without any semantic information . On the other hand , cABC letters are semantic but do not have local grouping cues . As we describe in the following sections , human observers easily solve both challenges with minimal training , indicating that biological vision is capable of relying on a single routine for perceptual grouping . Parameterization of image variability It is difficult to disentangle visual strategies of neural network architecture choices using standard computer vision datasets . For instance , the relative advantages of visual routines implemented by horizontal vs. top-down connections can be washed out by neural networks latching onto trivial dataset biases . More importantly , the typical training regime for deep learning involves big datasets , and in this regime , the ability of neural networks to act as universal function approximators can make inductive biases less important . We overcome these limitations with our synthetic challenges . Each challenge consists of three limited-size datasets of different task difficulties ( easy , intermediate and hard ; characterized by increasing levels of intra-class variability ) . We train and test network architectures on each of the three datasets and measure how accuracy changes when the need for stronger generalization capabilities arises as difficulty increases . This method , called “ straining ” ( Kim et al. , 2018 ; Linsley et al. , 2018b ) , dissociates an architecture ’ s expressiveness from other incidental factors tied to a particular image domain . Architectures with appropriate inductive biases are more likely to generalize as difficulty/intra-class variability increases compared to architectures that rely on low-bias routines that can resemble “ rote memorization ” ( Zhang et al. , 2016 ) . Difficulty of Pathfinder challenge is parameterized by the length of target curves in the images ( 6- , 9- , or 14-length ; Linsley et al . 2018b ) . cABC challenge difficulty is controlled in two ways : first , by increasing the number of possible relative positional arrangements between the two letters . Second , by decreasing the correlation between random transformation parameters ( rotation , scale and shear ) applied to each letter in an image . For instance , on the “ Easy ” difficulty dataset , letter transformations are correlated : the centers of two letters are displaced by a distance uniformly sampled in an interval between 25 and 30 pixels , and the same affine transformation is applied to both letters . In contrast , letter transformations are independent : letters are randomly separated by 15 to 40 pixels , and random affine transformations are independently sampled and applied to each . We balance the variation of these random transformation parameters across the three difficulty levels so that the total variability of individual letter shape remains constant ( i.e. , the number of transformations applied to each letter in a dataset is the same across difficulty levels ) . Harder cABC datasets lead to increasingly varied conjunctions between the strokes of overlapping letters , making them well suited for models that can iteratively process semantic groups instead of learning fixed feature templates . See Appendix A for more details about cABC generation .

Summary: This paper proposed a dataset and designed a relevant network structure to analyze the function of horizontal and top-down connections for perceptual grouping. The used two datasets smartly isolate the requirements for exploiting Gestalt cues and object-based strategies. Appendix A detailed describes the cABC dataset, and the control experiments in Appendix B further validate the designing of the cABC. The proposed network flexibly integrates three types of connections and successfully solves both two challenges. The visualization results in Figure 4, S8 and S9 are insightful and also validate the intuitions. Overall the paper is clearly written and easy to follow.