Recurrent neural circuits for contour detection

1 INTRODUCTION . An open debate since the inception of vision science concerns why we experience visual illusions . Consider the class of “ contextual ” illusions , where the perceived qualities of an image region , such as its orientation or color , are biased by the qualities of surrounding image regions . A well-studied contextual illusion is the orientation-tilt illusion depicted in Fig . 1a , where perception of the central grating ’ s orientation is influenced by the orientation of the surrounding grating ( O ’ Toole & Wenderoth , 1977 ) . When the two orientations are similar , the central grating appears tilted slightly away from the surround ( Fig . 1a , top ) . When the two orientations are dissimilar , the central grating appears tilted slightly towards the surround ( Fig . 1a , bottom ) . Is the contextual bias of the orientation-tilt illusion a bug of biology or a byproduct of optimized neural computations ? Over the past 50 years , there has been a number of neural circuit mechanisms proposed to explain individual contextual illusions ( reviewed in Mély et al. , 2018 ) . Recently , Mély et al . ( 2018 ) proposed a cortical circuit , constrained by physiology of primate visual cortex ( V1 ) , that offers a unified explanation for contextual illusions across visual domains – from the orientation-tilt illusion to color induction . These illusions arise in the circuit from recurrent interactions between neural populations with receptive fields that tile visual space , leading to contextual ( center/surround ) effects . For the orientation-tilt illusion , neural populations encoding the surrounding grating can either suppress or facilitate the activity of neural populations encoding the central grating , leading to repulsion vs. attraction , respectively . These surround neural populations compete to influence encodings of the central grating : suppression predominates when center/surround are similar , and facilitation predominates when center/surround are dissimilar . The neural circuit of Mély et al . ( 2018 ) explains how contextual illusions might emerge , but it does not explain why . One possibility is that contextual illusions like the orientation-tilt illusion are “ bugs ” : vestiges of evolution or biological constraints on the neural hardware . Another possibility is that contextual illusions are the by-product of efficient neural routines for scene segmentation ( Keemink & van Rossum , 2016 ; Mély et al. , 2018 ) . Here , we provide computational evidence for the latter †These authors contributed equally to this work . ‡Currently at DeepMind , London , UK . possibility and demonstrate that the orientation-tilt illusion reflects neural strategies optimized for object contour detection . Contributions We introduce the γ-Net , a trainable and hierarchical extension of the neural circuit of Mély et al . ( 2018 ) , which explains contextual illusions . ( i ) The γ-Net is more sample efficient than state-of-the-art convolutional architectures on two separate contour detection tasks . ( ii ) Similar to humans but not state-of-the-art contour detection models , the γ-Net exhibits an orientation-tilt illusion after being optimized for contour detection . This illusion emerges from its preference for high-level object-boundary contours over low-level edges , indicating that neural circuits involved in contextual illusions also support sample-efficient solutions to contour detection tasks . 2 RELATED WORK . Modeling the visual system Convolutional neural networks ( CNNs ) are often considered the de facto “ standard model ” of vision . CNNs and their extensions represent the state of the art for most computer vision applications with performance approaching – and sometimes exceeding – human observers on certain visual recognition tasks ( He et al. , 2016 ; Lee et al. , 2017 ; Phillips et al. , 2018 ) . CNNs also provide the best fit to rapid neural responses in the visual cortex ( see Kriegeskorte 2015 ; Yamins & DiCarlo 2016 for reviews ) . Nevertheless , multiple lines of evidence suggest that biological vision is still far more robust and versatile than CNNs ( see Serre , 2019 , for a recent review ) . CNNs suffer from occlusions and clutter ( Fyall et al. , 2017 ; Rosenfeld et al. , 2018 ; Tang et al. , 2018 ) . They are also sample inefficient at learning visual relations ( Kim et al. , 2018 ) and solving simple grouping tasks ( Linsley et al. , 2018c ) . State-of-the-art CNNs require massive datasets to reach their impressive accuracy ( Lake et al. , 2015 ) and their ability to generalize beyond training data is limited ( Geirhos et al. , 2018 ; Recht et al. , 2018 ) . Cortical feedback contributes to the robustness of biological vision ( Hochstein & Ahissar , 2002 ; Wyatte et al. , 2014 ; Kafaligonul et al. , 2015 ) . Feedforward projections in the visual system are almost always matched by feedback projections ( Felleman & Van Essen , 1991 ) , and feedback has been implicated in visual “ routines ” that can not be implemented through purely feedforward vision , such as incremental grouping or filling-in ( O ’ Reilly et al. , 2013 ; Roelfsema , 2006 ) . There is a also a growing body of work demonstrating the potential of recurrent neural networks ( RNNs ) to account for neural recordings ( Fyall et al. , 2017 ; Klink et al. , 2017 ; Siegel et al. , 2015 ; Tang et al. , 2018 ; Nayebi et al. , 2018 ; Kar et al. , 2019 ; Kietzmann et al. , 2019 ) . Feedback for computer vision In contrast to CNNs , which build processing depth through a cascade of filtering and pooling stages with unique weights , RNNs process stimuli with filtering stages that reuse weights over “ timesteps ” of recurrence . On each discrete processing timestep , an RNN updates its hidden state through a nonlinear combination of an input and its the hidden state from its previous timestep . RNNs have been extended from their roots in sequence processing ( e.g. , Mozer 1992 ) to computer vision by computing the activity of RNN units through convolutional kernels . The common interpretation of these convolutional-RNNs , is that the input to each layer functions as a ( fixed ) feedforward drive , which is combined with layer-specific feedback from an evolving hidden state to dynamically adjust layer activity ( Linsley et al. , 2018c ; George et al. , 2017 ; Lotter et al. , 2016 ; Wen et al. , 2018 ; Liao & Poggio , 2016 ; Spoerer et al. , 2017 ; Nayebi et al. , 2018 ; Tang et al. , 2018 ) . In the current work , we are motivated by a similar convolutional-RNN , the horizontal gated recurrent unit ( hGRU , Linsley et al . 2018a ) , which approximates the recurrent neural circuit model of ( Mély et al. , 2018 ) for explaining contextual illusions . 3 RECURRENT NEURAL MODELS . We begin by reviewing the dynamical neural circuit of Mély et al . ( 2018 ) . This model explains contextual illusions by simulating interactions between cortical hypercolumns tiling the visual field ( where hypercolumns describe a set of neurons encoding features for multiple visual domains at a single retinotopic position ) . In the model , hypercolumns are indexed by their 2D coordinate ( x , y ) and feature channels k. Units in hypercolumns encode idealized responses for a visual domain ( e.g. , neural responses from the orientation domain were used to simulate an orientation-tilt illusion ; ( a ) xyk ) interactions between units in a layer ` , which receives feedforward drive from a center pathway encoding feature k ( e.g. , edges oriented at 0◦ or 22.5◦ ) at position ( x , y ) in an image . Blocks depict different layers , and arrowed connections denote top-down feedback . ( c ) A deep network schematic of the circuit diagram in ( b ) , which forms the basis of the γ-Net introduced here . Horizontal and top-down connections are implemented with feedback gated recurrent units ( fGRUs ) . Image encodings pass through these blocks on every timestep , from bottom-up ( left path ) to top-down ( right path ) , and predictions are read out from the fGRU closest to image resolution on the final timestep . This motif can be stacked to create a hierarchical model . Fig . 1b ) . Dynamics of a single unit at xyk obey the following equations ( we bold activity tensors to distinguish them from learned kernels and parameters ) : ηḢ ( S ) xyk + 2H ( S ) xyk = [ ξZxyk − ( αH ( F ) xyk + µ ) C ( S ) xyk ] + # Stage 1 : Recurrent suppression of Z τḢ ( F ) xyk + σ 2H ( F ) xyk = [ νC ( F ) xyk ] + , # Stage 2 : Recurrent facilitation of H ( S ) where C ( S ) xyk = ( W S ∗H ( F ) ) xyk # Compute suppression interactions C ( F ) xyk = ( W F ∗H ( S ) ) xyk . # Compute facilitation interactions Circuit activities consist of a feedforward drive , recurrent suppression , and recurrent facilitation , respectively denoted as Z , H ( S ) , H ( F ) ∈ RX×Y×K ( X is width , Y is height of the tensor , and K is its feature channels ) ∗ . The circuit takes its “ feedforward ” input Z from hypercolumns ( e.g. , orientation encodings from hypercolumn units ) , and introduces recurrent suppressive and facilitatory interactions between units , C ( S ) , C ( F ) ∈ RX×Y×K ( Fig . 1b ) . These interactions are implemented with separate kernels for suppression and facilitation , WS , WF ∈ RE×E×K×K , where E is the spatial extent of connections on a single timestep ( connectivity in this model is constrained by primate physiology ) . ∗Suppression refers to interactions that reduce unit activity , and facilitation refers to interactions that increase activity . These computations can cause illusory repulsion or attraction at the level of neural populations . These interactions are implemented through convolutions , allowing them to serially spread over timesteps of processing to connect units positioned at different spatial locations . The circuit outputs H ( F ) after reaching steady state . The circuit model of Mély et al . ( 2018 ) depends on competition between H ( S ) and H ( F ) to explain the orientation-tilt illusion . Competition is implemented by ( i ) computing suppression vs. facilitation in separate stages , and ( ii ) having non-negative activities , which enforces these functionally distinct processing stages . With these constraints in the circuit model , the strength of recurrent suppression – but not facilitation – multiplicatively increases with the net recurrent output . For the orientation-tilt illusion , suppression predominates when center and surround gratings have similar orientations . This causes encodings of the surround grating to “ repulse ” encodings of the center grating . On the other hand , facilitation predominates ( causing “ attraction ” ) when center and surround gratings have dissimilar orientations because it is additive and not directly scaled by the circuit output . Parameters controlling the circuit ’ s integration , suppression/facilitation , and patterns of horizontal connections between units are tuned by hand . Linear and multiplicative suppression ( i.e. , shunting inhibition ) are controlled by scalars µ and α , feedforward drive is modulated by the scalar ξ , and linear facilitation is controlled by the scalar ν . Circuit time constants are scalars denoted by η , , τ and σ . All activities are non-negative and both stages are linearly rectified ( ReLU ) [ · ] + = max ( · , 0 ) . Feedback gated recurrent units Linsley et al . ( 2018a ) developed a version of this circuit for computer vision applications , called the hGRU . In their formulation they use gradient descent ( rather than hand-tuning like in the original circuit ) to fit its connectivity and parameters to image datasets . The hGRU was designed to learn a difficult synthetic incremental grouping task , and a single layer of the hGRU learned long-range spatial dependencies that CNNs with orders-of-magnitude more weights could not . The hGRU replaced the circuit ’ s time constants with dynamic gates , converted the recurrent state H ( S ) for suppression into an instantaneous activity , and introduced a term for quadratic facilitation . The hGRU also relaxed biological constraints from the original circuit , including an assumption of non-negativity , which enforced competition between recurrent suppression vs. facilitation ( e.g. , guaranteeing that Stage 1 in the circuit model describes suppression of Zxyk ) . We extend the hGRU formulation in two important ways . First , like Mély et al . ( 2018 ) , we introduce non-negativity . This constraint was critical for Mély et al . ( 2018 ) to explain contextual illusions , and as we describe below , was also important for our model . Second , we extend the circuit into a hierarchical model which can learn complex contour detection tasks . Recent neurophysiological work indicates that contextual effects emerge from both horizontal and top-down feedback ( Chettih & Harvey , 2019 ) . Motivated by this , we develop versions of the circuit to simulate horizontal connections between units within a layer , and top-down connections between units in different layers . We call our module the feedback gated recurrent unit ( fGRU ) . We describe the evolution of fGRU recurrent units in H ∈ RX×Y×K , which are influenced by non-negative feedforward encodings Z ∈ RX×Y×K ( e.g. , a convolutional layer ’ s response to a stimulus ) over discrete timesteps · [ t ] : Stage 1 : GS = sigmoid ( US ∗H [ t− 1 ] ) # Compute channel-wise selection CS = WS ∗ ( H [ t− 1 ] GS ) # Compute suppression interactions S = [ Z− [ ( αH [ t− 1 ] + µ ) CS ] + ] + , # Suppression of Z Stage 2 : GF = sigmoid ( UF ∗ S ) # Compute channel-wise recurrent updates CF = WF ∗ S # Compute facilitation interactions H̃ = [ ν ( CF + S ) + ω ( CF ∗ S ) ] + # Facilitation of S H [ t ] = ( 1−GF ) H [ t− 1 ] + GF H̃ . # Update recurrent state Like the original circuit , the fGRU has separate stages for suppression ( S ) and facilitation ( H ) . In the first stage , the feedforward encodings Z are suppressed by non-negative interactions between units in H [ t− 1 ] ( an fGRU hidden state from the previous timestep ) . Suppressive interactions are computed with the kernel WS ∈ RE×E×K×K , where E describes the spatial extent of horizontal connections on a single timestep . This kernel is convolved with a gated version of the persistent hidden state H [ t−1 ] . The gate activity GS is computed by applying a sigmoid nonlinearity to a convolution of the kernel US ∈ R1×1×K×K with H [ t− 1 ] , which transforms its activity into the range [ 0 , 1 ] . Additive and multiplicative forms of suppression are controlled by the parameters µ , α ∈ RK , respectively . In the second stage , additive and multiplicative facilitation is applied to the instantaneous activity S. The kernels WF ∈ RE×E×K×K controls facilitation interactions . Additive and multiplicative forms of facilitation are scaled by the parameters ν , ω ∈ RK , respectively . A gate activity is also computed during this stage to update the persistent recurrent activity H. The gate activity GF is computed by applying a sigmoid to a convolution of the kernel UF ∈ R1×1×K×K with S. This gate updates H [ t ] by interpolating H [ t− 1 ] with the candidate activity H̃ . After every timestep of processing , H [ t ] is taken as the fGRU output activity . As detailed in the following section , the fGRU output hidden state is either passed to the next convolutional layer ( Fig . 1c , fGRU ( ` ) −→ conv ( ` +1 ) ) , or used to compute top-down connections ( Fig . 1c , fGRU ( ` +1 ) −→ fGRU ( ` ) ) . The fGRU has different configurations for learning horizontal connections between units within a layer or top-down connections between layers ( Fig . 1b ) . These two configurations stem from changing the activities used for a fGRU ’ s feedforward encodings and recurrent hidden state . “ Horizontal connections ” between units within a layer are learned by setting the feedforward encodings Z to the activity of a preceding convolutional layer , and setting the hidden state H to a persistent activity initialized as zeros ( Fig . 1c , conv ( ` ) −→ fGRU ( ` ) ) . “ Top-down connections ” between layers are learned by setting fGRU feedforward encodings Z to the persistent hidden state H ( ` ) of a fGRU at layer ` in a hierarchical model , and the hidden state H to the persistent activity H ( ` +1 ) of an fGRU at a layer one level up in the model hierarchy ( Fig . 1c , fGRU ( ` +1 ) −→ fGRU ( ` ) ) . The functional interpretation of the top-down fGRU is that it first suppresses activity in the lower layer using the higher layer ’ s recurrent horizontal activities , and then applies a kernel to the residue for facilitation , which allows for computations like interpolation , sharpening , or “ filling in ” . Note that an fGRU for top-down connections does not have a persistent state ( it mixes high and low-level persistent states ) , but an fGRU for horizontal connections does . γ-Net Our main objective is to test how a model with the capacity for contextual illusions performs on natural image analysis . We do this by incorporating fGRUs into leading feedforward architectures for contour detection tasks , augmenting their feedforward processing with modules for learning feedback from horizontal and top-down connections ( Fig . 1c ) . We refer to the resulting hierarchical models as γ-Nets , because information flows in a loop that resembles a γ : image encodings make a full bottom-up to top-down cycle through the architecture on every timestep , until dense predictions are read-out from the lowest-level recurrent layer of the network ( thus , information flows in at the top of the hierarchy , loops through the network , and flows out from the top of the hierarchy ) . In our experiments we convert leading architectures for two contour detection problems into γ-Nets : A VGG16 for BSDS500 ( He et al. , 2019 ) , and a U-Net for detection of cell membranes in serial electron microscopy images ( Lee et al. , 2017 ) . See Appendix A for an algorithmic description of γ-net .

Summary: The authors propose a new artificial neural network architecture that is derived from a human visual model (Mély et al., 2018). The original (human vision) model can explain some of the human visual illusions, specifically contextual ones. While the adaptation from this human visual model to artificial neural networks was previously done by (Linsley et. al., 2018a), in this paper the authors extend (Linsley et. al., 2018a) to better capture some of the constraints in the human visual model, and also to add a formulation that can also model top-down connections (across layers).