Recurrent neural circuits for contour detection

1 INTRODUCTION . An open debate since the inception of vision science concerns why we experience visual illusions . Consider the class of “ contextual ” illusions , where the perceived qualities of an image region , such as its orientation or color , are biased by the qualities of surrounding image regions . A well-studied contextual illusion is the orientation-tilt illusion depicted in Fig . 1a , where perception of the central grating ’ s orientation is influenced by the orientation of the surrounding grating ( O ’ Toole & Wenderoth , 1977 ) . When the two orientations are similar , the central grating appears tilted slightly away from the surround ( Fig . 1a , top ) . When the two orientations are dissimilar , the central grating appears tilted slightly towards the surround ( Fig . 1a , bottom ) . Is the contextual bias of the orientation-tilt illusion a bug of biology or a byproduct of optimized neural computations ? Over the past 50 years , there has been a number of neural circuit mechanisms proposed to explain individual contextual illusions ( reviewed in Mély et al. , 2018 ) . Recently , Mély et al . ( 2018 ) proposed a cortical circuit , constrained by physiology of primate visual cortex ( V1 ) , that offers a unified explanation for contextual illusions across visual domains – from the orientation-tilt illusion to color induction . These illusions arise in the circuit from recurrent interactions between neural populations with receptive fields that tile visual space , leading to contextual ( center/surround ) effects . For the orientation-tilt illusion , neural populations encoding the surrounding grating can either suppress or facilitate the activity of neural populations encoding the central grating , leading to repulsion vs. attraction , respectively . These surround neural populations compete to influence encodings of the central grating : suppression predominates when center/surround are similar , and facilitation predominates when center/surround are dissimilar . The neural circuit of Mély et al . ( 2018 ) explains how contextual illusions might emerge , but it does not explain why . One possibility is that contextual illusions like the orientation-tilt illusion are “ bugs ” : vestiges of evolution or biological constraints on the neural hardware . Another possibility is that contextual illusions are the by-product of efficient neural routines for scene segmentation ( Keemink & van Rossum , 2016 ; Mély et al. , 2018 ) . Here , we provide computational evidence for the latter †These authors contributed equally to this work . ‡Currently at DeepMind , London , UK . possibility and demonstrate that the orientation-tilt illusion reflects neural strategies optimized for object contour detection . Contributions We introduce the γ-Net , a trainable and hierarchical extension of the neural circuit of Mély et al . ( 2018 ) , which explains contextual illusions . ( i ) The γ-Net is more sample efficient than state-of-the-art convolutional architectures on two separate contour detection tasks . ( ii ) Similar to humans but not state-of-the-art contour detection models , the γ-Net exhibits an orientation-tilt illusion after being optimized for contour detection . This illusion emerges from its preference for high-level object-boundary contours over low-level edges , indicating that neural circuits involved in contextual illusions also support sample-efficient solutions to contour detection tasks . 2 RELATED WORK . Modeling the visual system Convolutional neural networks ( CNNs ) are often considered the de facto “ standard model ” of vision . CNNs and their extensions represent the state of the art for most computer vision applications with performance approaching – and sometimes exceeding – human observers on certain visual recognition tasks ( He et al. , 2016 ; Lee et al. , 2017 ; Phillips et al. , 2018 ) . CNNs also provide the best fit to rapid neural responses in the visual cortex ( see Kriegeskorte 2015 ; Yamins & DiCarlo 2016 for reviews ) . Nevertheless , multiple lines of evidence suggest that biological vision is still far more robust and versatile than CNNs ( see Serre , 2019 , for a recent review ) . CNNs suffer from occlusions and clutter ( Fyall et al. , 2017 ; Rosenfeld et al. , 2018 ; Tang et al. , 2018 ) . They are also sample inefficient at learning visual relations ( Kim et al. , 2018 ) and solving simple grouping tasks ( Linsley et al. , 2018c ) . State-of-the-art CNNs require massive datasets to reach their impressive accuracy ( Lake et al. , 2015 ) and their ability to generalize beyond training data is limited ( Geirhos et al. , 2018 ; Recht et al. , 2018 ) . Cortical feedback contributes to the robustness of biological vision ( Hochstein & Ahissar , 2002 ; Wyatte et al. , 2014 ; Kafaligonul et al. , 2015 ) . Feedforward projections in the visual system are almost always matched by feedback projections ( Felleman & Van Essen , 1991 ) , and feedback has been implicated in visual “ routines ” that can not be implemented through purely feedforward vision , such as incremental grouping or filling-in ( O ’ Reilly et al. , 2013 ; Roelfsema , 2006 ) . There is a also a growing body of work demonstrating the potential of recurrent neural networks ( RNNs ) to account for neural recordings ( Fyall et al. , 2017 ; Klink et al. , 2017 ; Siegel et al. , 2015 ; Tang et al. , 2018 ; Nayebi et al. , 2018 ; Kar et al. , 2019 ; Kietzmann et al. , 2019 ) . Feedback for computer vision In contrast to CNNs , which build processing depth through a cascade of filtering and pooling stages with unique weights , RNNs process stimuli with filtering stages that reuse weights over “ timesteps ” of recurrence . On each discrete processing timestep , an RNN updates its hidden state through a nonlinear combination of an input and its the hidden state from its previous timestep . RNNs have been extended from their roots in sequence processing ( e.g. , Mozer 1992 ) to computer vision by computing the activity of RNN units through convolutional kernels . The common interpretation of these convolutional-RNNs , is that the input to each layer functions as a ( fixed ) feedforward drive , which is combined with layer-specific feedback from an evolving hidden state to dynamically adjust layer activity ( Linsley et al. , 2018c ; George et al. , 2017 ; Lotter et al. , 2016 ; Wen et al. , 2018 ; Liao & Poggio , 2016 ; Spoerer et al. , 2017 ; Nayebi et al. , 2018 ; Tang et al. , 2018 ) . In the current work , we are motivated by a similar convolutional-RNN , the horizontal gated recurrent unit ( hGRU , Linsley et al . 2018a ) , which approximates the recurrent neural circuit model of ( Mély et al. , 2018 ) for explaining contextual illusions . 3 RECURRENT NEURAL MODELS . We begin by reviewing the dynamical neural circuit of Mély et al . ( 2018 ) . This model explains contextual illusions by simulating interactions between cortical hypercolumns tiling the visual field ( where hypercolumns describe a set of neurons encoding features for multiple visual domains at a single retinotopic position ) . In the model , hypercolumns are indexed by their 2D coordinate ( x , y ) and feature channels k. Units in hypercolumns encode idealized responses for a visual domain ( e.g. , neural responses from the orientation domain were used to simulate an orientation-tilt illusion ; ( a ) xyk ) interactions between units in a layer ` , which receives feedforward drive from a center pathway encoding feature k ( e.g. , edges oriented at 0◦ or 22.5◦ ) at position ( x , y ) in an image . Blocks depict different layers , and arrowed connections denote top-down feedback . ( c ) A deep network schematic of the circuit diagram in ( b ) , which forms the basis of the γ-Net introduced here . Horizontal and top-down connections are implemented with feedback gated recurrent units ( fGRUs ) . Image encodings pass through these blocks on every timestep , from bottom-up ( left path ) to top-down ( right path ) , and predictions are read out from the fGRU closest to image resolution on the final timestep . This motif can be stacked to create a hierarchical model . Fig . 1b ) . Dynamics of a single unit at xyk obey the following equations ( we bold activity tensors to distinguish them from learned kernels and parameters ) : ηḢ ( S ) xyk + 2H ( S ) xyk = [ ξZxyk − ( αH ( F ) xyk + µ ) C ( S ) xyk ] + # Stage 1 : Recurrent suppression of Z τḢ ( F ) xyk + σ 2H ( F ) xyk = [ νC ( F ) xyk ] + , # Stage 2 : Recurrent facilitation of H ( S ) where C ( S ) xyk = ( W S ∗H ( F ) ) xyk # Compute suppression interactions C ( F ) xyk = ( W F ∗H ( S ) ) xyk . # Compute facilitation interactions Circuit activities consist of a feedforward drive , recurrent suppression , and recurrent facilitation , respectively denoted as Z , H ( S ) , H ( F ) ∈ RX×Y×K ( X is width , Y is height of the tensor , and K is its feature channels ) ∗ . The circuit takes its “ feedforward ” input Z from hypercolumns ( e.g. , orientation encodings from hypercolumn units ) , and introduces recurrent suppressive and facilitatory interactions between units , C ( S ) , C ( F ) ∈ RX×Y×K ( Fig . 1b ) . These interactions are implemented with separate kernels for suppression and facilitation , WS , WF ∈ RE×E×K×K , where E is the spatial extent of connections on a single timestep ( connectivity in this model is constrained by primate physiology ) . ∗Suppression refers to interactions that reduce unit activity , and facilitation refers to interactions that increase activity . These computations can cause illusory repulsion or attraction at the level of neural populations . These interactions are implemented through convolutions , allowing them to serially spread over timesteps of processing to connect units positioned at different spatial locations . The circuit outputs H ( F ) after reaching steady state . The circuit model of Mély et al . ( 2018 ) depends on competition between H ( S ) and H ( F ) to explain the orientation-tilt illusion . Competition is implemented by ( i ) computing suppression vs. facilitation in separate stages , and ( ii ) having non-negative activities , which enforces these functionally distinct processing stages . With these constraints in the circuit model , the strength of recurrent suppression – but not facilitation – multiplicatively increases with the net recurrent output . For the orientation-tilt illusion , suppression predominates when center and surround gratings have similar orientations . This causes encodings of the surround grating to “ repulse ” encodings of the center grating . On the other hand , facilitation predominates ( causing “ attraction ” ) when center and surround gratings have dissimilar orientations because it is additive and not directly scaled by the circuit output . Parameters controlling the circuit ’ s integration , suppression/facilitation , and patterns of horizontal connections between units are tuned by hand . Linear and multiplicative suppression ( i.e. , shunting inhibition ) are controlled by scalars µ and α , feedforward drive is modulated by the scalar ξ , and linear facilitation is controlled by the scalar ν . Circuit time constants are scalars denoted by η , , τ and σ . All activities are non-negative and both stages are linearly rectified ( ReLU ) [ · ] + = max ( · , 0 ) . Feedback gated recurrent units Linsley et al . ( 2018a ) developed a version of this circuit for computer vision applications , called the hGRU . In their formulation they use gradient descent ( rather than hand-tuning like in the original circuit ) to fit its connectivity and parameters to image datasets . The hGRU was designed to learn a difficult synthetic incremental grouping task , and a single layer of the hGRU learned long-range spatial dependencies that CNNs with orders-of-magnitude more weights could not . The hGRU replaced the circuit ’ s time constants with dynamic gates , converted the recurrent state H ( S ) for suppression into an instantaneous activity , and introduced a term for quadratic facilitation . The hGRU also relaxed biological constraints from the original circuit , including an assumption of non-negativity , which enforced competition between recurrent suppression vs. facilitation ( e.g. , guaranteeing that Stage 1 in the circuit model describes suppression of Zxyk ) . We extend the hGRU formulation in two important ways . First , like Mély et al . ( 2018 ) , we introduce non-negativity . This constraint was critical for Mély et al . ( 2018 ) to explain contextual illusions , and as we describe below , was also important for our model . Second , we extend the circuit into a hierarchical model which can learn complex contour detection tasks . Recent neurophysiological work indicates that contextual effects emerge from both horizontal and top-down feedback ( Chettih & Harvey , 2019 ) . Motivated by this , we develop versions of the circuit to simulate horizontal connections between units within a layer , and top-down connections between units in different layers . We call our module the feedback gated recurrent unit ( fGRU ) . We describe the evolution of fGRU recurrent units in H ∈ RX×Y×K , which are influenced by non-negative feedforward encodings Z ∈ RX×Y×K ( e.g. , a convolutional layer ’ s response to a stimulus ) over discrete timesteps · [ t ] : Stage 1 : GS = sigmoid ( US ∗H [ t− 1 ] ) # Compute channel-wise selection CS = WS ∗ ( H [ t− 1 ] GS ) # Compute suppression interactions S = [ Z− [ ( αH [ t− 1 ] + µ ) CS ] + ] + , # Suppression of Z Stage 2 : GF = sigmoid ( UF ∗ S ) # Compute channel-wise recurrent updates CF = WF ∗ S # Compute facilitation interactions H̃ = [ ν ( CF + S ) + ω ( CF ∗ S ) ] + # Facilitation of S H [ t ] = ( 1−GF ) H [ t− 1 ] + GF H̃ . # Update recurrent state Like the original circuit , the fGRU has separate stages for suppression ( S ) and facilitation ( H ) . In the first stage , the feedforward encodings Z are suppressed by non-negative interactions between units in H [ t− 1 ] ( an fGRU hidden state from the previous timestep ) . Suppressive interactions are computed with the kernel WS ∈ RE×E×K×K , where E describes the spatial extent of horizontal connections on a single timestep . This kernel is convolved with a gated version of the persistent hidden state H [ t−1 ] . The gate activity GS is computed by applying a sigmoid nonlinearity to a convolution of the kernel US ∈ R1×1×K×K with H [ t− 1 ] , which transforms its activity into the range [ 0 , 1 ] . Additive and multiplicative forms of suppression are controlled by the parameters µ , α ∈ RK , respectively . In the second stage , additive and multiplicative facilitation is applied to the instantaneous activity S. The kernels WF ∈ RE×E×K×K controls facilitation interactions . Additive and multiplicative forms of facilitation are scaled by the parameters ν , ω ∈ RK , respectively . A gate activity is also computed during this stage to update the persistent recurrent activity H. The gate activity GF is computed by applying a sigmoid to a convolution of the kernel UF ∈ R1×1×K×K with S. This gate updates H [ t ] by interpolating H [ t− 1 ] with the candidate activity H̃ . After every timestep of processing , H [ t ] is taken as the fGRU output activity . As detailed in the following section , the fGRU output hidden state is either passed to the next convolutional layer ( Fig . 1c , fGRU ( ` ) −→ conv ( ` +1 ) ) , or used to compute top-down connections ( Fig . 1c , fGRU ( ` +1 ) −→ fGRU ( ` ) ) . The fGRU has different configurations for learning horizontal connections between units within a layer or top-down connections between layers ( Fig . 1b ) . These two configurations stem from changing the activities used for a fGRU ’ s feedforward encodings and recurrent hidden state . “ Horizontal connections ” between units within a layer are learned by setting the feedforward encodings Z to the activity of a preceding convolutional layer , and setting the hidden state H to a persistent activity initialized as zeros ( Fig . 1c , conv ( ` ) −→ fGRU ( ` ) ) . “ Top-down connections ” between layers are learned by setting fGRU feedforward encodings Z to the persistent hidden state H ( ` ) of a fGRU at layer ` in a hierarchical model , and the hidden state H to the persistent activity H ( ` +1 ) of an fGRU at a layer one level up in the model hierarchy ( Fig . 1c , fGRU ( ` +1 ) −→ fGRU ( ` ) ) . The functional interpretation of the top-down fGRU is that it first suppresses activity in the lower layer using the higher layer ’ s recurrent horizontal activities , and then applies a kernel to the residue for facilitation , which allows for computations like interpolation , sharpening , or “ filling in ” . Note that an fGRU for top-down connections does not have a persistent state ( it mixes high and low-level persistent states ) , but an fGRU for horizontal connections does . γ-Net Our main objective is to test how a model with the capacity for contextual illusions performs on natural image analysis . We do this by incorporating fGRUs into leading feedforward architectures for contour detection tasks , augmenting their feedforward processing with modules for learning feedback from horizontal and top-down connections ( Fig . 1c ) . We refer to the resulting hierarchical models as γ-Nets , because information flows in a loop that resembles a γ : image encodings make a full bottom-up to top-down cycle through the architecture on every timestep , until dense predictions are read-out from the lowest-level recurrent layer of the network ( thus , information flows in at the top of the hierarchy , loops through the network , and flows out from the top of the hierarchy ) . In our experiments we convert leading architectures for two contour detection problems into γ-Nets : A VGG16 for BSDS500 ( He et al. , 2019 ) , and a U-Net for detection of cell membranes in serial electron microscopy images ( Lee et al. , 2017 ) . See Appendix A for an algorithmic description of γ-net .

Summary: The paper introduces a complex hierarchical recurrent model for contour detection loosely inspired by the organization of cortical circuits. Their model performs state-of-the-art on sample-limited versions of popular contour detection (BSDS500) and cell segmentation (SNEMI3D) datasets, and it reproduces the well-known tilt illusion when transfer-learning orientation estimation. Interestingly, "untraining" the tilt illusion degrades performance on contour detection.