Contextual Recurrent Convolutional Model for Robust Visual Learning

1 INTRODUCTION . It has been long established that the primate ’ s ventral visual system has a hierarchical structure ( Felleman & Van Essen , 1991 ) including early ( V1 , V2 ) , intermediate ( V4 ) , and higher ( IT ) visual areas . Modern deep convolutional neural networks ( CNNs ) for image recognition ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ) trained on large image data sets like ImageNet ( Russakovsky et al. , 2015 ) imitate this hierarchical structure with multiple layers . There is a hierarchical correspondence between internal feature representations of a deep CNN ’ s different layers and neural representations of different visual areas ( Cichy et al. , 2016 ; Yamins & DiCarlo , 2016 ) ; lower visual areas ( V1 , V2 ) are best explained by a deep CNN ’ s internal representations from lower layers ( Cadena et al. , 2017 ; Khaligh-Razavi & Kriegeskorte , 2014 ) and higher areas ( IT , V4 ) are best explained by its higher layers ( Khaligh-Razavi & Kriegeskorte , 2014 ; Yamins et al. , 2014 ) . Deep CNNs explain neuron responses in ventral visual system better than any other model class ( Yamins & DiCarlo , 2016 ; Kriegeskorte , 2015 ) , and this success indicates that deep CNNs share some similarities with the ventral visual system , in terms of architecture and internal feature representations ( Yamins & DiCarlo , 2016 ) . However , there is one key structural component that is missing in the standard feedforward deep CNNs : contextual feedback recurrent connections between neurons in different areas ( Felleman & Van Essen , 1991 ) . These connections greatly contribute to the complexity of the visual system , and may be essential for the success of the visual systems in reality ; for example , there are evidences that recurrent connections are crucial for object recognition under noise , clutter , and occlusion ( O ’ Reilly et al. , 2013 ; Spoerer et al. , 2017 ; Rajaei et al. , 2018 ) . In this paper , we explored a variety of model with different recurrent architectures , contextual modules , and information flows to understand the computational advantages of feedback circuits . We are interested in understanding what and how top-down and bottom-up contextual information can be combined to improve in performance in visual tasks . We investigated VGG16 ( Simonyan & Zisserman , 2014 ) , a standard CNN that coarsely approximate the ventral visual hierarchical stream , and its recurrent variants for comparison . To introduce feedback recurrent connections , we divided VGG16 ’ s layers into stages and selectively added feedback connections from the groups ’ highest layers to their lowest layers . At the end of each feedback connection , there is a contextual module ( Section 3.2 ) that refines the bottom-up input with gated contextual information . We tested and compared several networks with such contextual modules against VGG16 in several standard image classification task , as well as visual tasks in which refinement under feedback guidance is more likely to produce some beneficial effects , such as object recognition under degraded conditions ( noise , clutter and occlusion ) and fine-grained recognition . We found that our network could outperform all the baseline feedforward networks and surpassed them by a large margin in finegrained and occlusion tasks . We also studied the internal feature representations of our network to illustrate the effectiveness of the structure . While much future work has to be done , our work can still be an important step to bridge the gap between biological visual systems and state-of-the-art computer vision models . 2 RELATED WORK . Although recurrent network modules including LSTM ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Unit ( Cho et al. , 2014 ) have been widely used in temporal prediction ( Wang et al. , 2017c ) and processing of sequential data ( e.g . video classification ( Donahue et al. , 2015 ) ) , few studies have been done to augment feedforward CNNs with recurrent connections in image-based computer vision tasks . Image classification . Standard deep CNNs for image classification suffer from occlusion and noise ( Wang et al. , 2017a ; b ; Zhang et al. , 2017 ) , since heavy occlusion and noise severely corrupt feature representations at lower layers and therefore cause degradation of higher semantic layers . With the inclusion of feedback connections , a model can “ rethink ” or refine its feature representations at lower layers using feedback information from higher layers ( Li et al. , 2018 ) ; after multiple rounds of feedback and refinement , input signals from distracting objects ( noise , irrelevant objects , etc . ) will be suppressed in the final feature representation ( Cao et al. , 2015 ) . Li et al . ( 2018 ) used the output posterior possibilities of a CNN to refine its intermediate feature maps ; however , their method requires posterior possibilities for refinement and thus can not be applied in scenarios where supervision is absent . Jetley et al . ( 2018 ) used more global and semantic features at higher convolutional layers to sharpen more local feature maps at lower layers for image classification on CIFAR datasets ; however , our own experimentation suggests that this method only works when the higher and lower layers have a relatively small semantic gap ( similarly sized receptive fields ) ; on high- resolution dataset like ImageNet , large semantic gaps between higher and lower layers make this method difficult to work . Other computer vision tasks . Linsley et al . ( 2018 ) designed a model with explicit horizontal recurrent connections to solve contour detection problems , and Spoerer et al . ( 2017 ) evaluated the performance of various models with recurrent connections on digit recognition tasks under clutter . The tasks evaluated in these studies are rather simple and contrived , and it remains to be seen whether their models and conclusions can apply to real world computer vision problems . ( Li et al. , 2018 ) uses posterior possibilities at the last fully connected layer to select intermediate feature map representations ; however , the posterior possibility vector is not informative enough and the input of the feedback connection is totally fixed , which makes it less flexible to fully mimic the recurrent connections in the visual system . Overall , feedback and recurrent connections are present in multiple layers of the visual hierarchy , and this study constrains feedback connections to the output classification layer only . It is worth noting that a recent study ( Nayebi et al. , 2018 ) is motivated by recurrent connections in the brain as well ; however , their work focuses on exploring the computational benefits of local recurrent connections while ours focuses on feedback recurrent ones . Thus , we believe that our work is complementary to theirs . 3 METHODS . In this section , we will describe the overall architecture of our proposed model and discuss some design details . 3.1 OVERALL MODEL ARCHITECTURE . The main structure of our Contextual Recurrent Convolutional Network ( CRCN ) is shown in Figure 1 . A CRCN model is a standard feedforward convolutional network augmented with feedback connections attached to some layers . At the end of each feedback connection , a contextual module fuses top-down and bottom-up information ( dashed red lines in Figure 1 ) to provide refined and sharpened input to the augmented layer . Given an input image , the model generates intermediate feature representations and output responses in multiple time steps . At the first time step ( t = 0 in Figure 1 ) , the model passes the input through the feedforward route ( black arrows in Figure 1 ) as in a standard CNN . At later time steps ( t > 0 in Figure 1 ) , each contextual module fuses output representations of lower and higher layers at the previous step ( dashed red lines in Figure 1 ) to generate the refined input at the current time step ( red lines in Figure 1 ) . Mathematically , we have O ( t ) k = { fk ( O ( t ) k−1 ) if t = 0 or k /∈ SG ck ( O ( t−1 ) k−1 , O ( t−1 ) h ( k ) ) if t > 0 and k ∈ SG , ( 1 ) where SG is the index set of layers augmented with feedback connections and contextual modules , ck ( · , · ) ( detailed in Eqs . ( 2 ) ) is the contextual module for layer k , O ( t ) k denotes the output of layer k at time t , h ( · ) is a function that maps the index of an augmented layer to that of its higher feedback layer , and fk ( · ) denotes the ( feedforward ) operation to compute the output of layer k given some input . 3.2 CONTEXTUAL MODULE DESIGN . The key part of the Contextual Recurrent Convolutional Network model is the contextual module at the end of each feedback connection . Figure 3 shows one possible design of the contextual module , which is inspired by traditional RNN modules including LSTM ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Unit ( Cho et al. , 2014 ) . In this scheme , a gate map is generated by the concatenation of the bottom-up and the ( upsampled ) top-down feature map passing through a 3× 3 convolution ( black circle with “ C ” and black arrows with circle ) . Then a tanh function is applied to the map to generate a gate map . The gate map then controls the amount of contextual information that can go through by a point-wise multiplication ( red lines ) . To make the information flow more stable , we add it with bottom-up feature map ( black circle with “ + ” ) . The equations are presented in Eqs . ( 2 ) . Then we use this new feature representation to replace the old one and continue feedforward calculation as described in Section 3.1 . O ( t ) k = gate ∗Upsample ( O ( t−1 ) h ( k ) ) +O ( t−1 ) k ( 2a ) gate = Tanh ( Conv3×3 ( Concat ( Upsample ( O ( t−1 ) h ( k ) ) , O ( t−1 ) k−1 ) ) ) ( 2b ) 3.3 LOCATION OF RECURRENT CONNECTIONS . Since there exists a gap between the semantic meanings of feature representations of bottom-up and top-down layers , we argue that recurrent connection across too many layers can do harm to the performance . Therefore , we derive three sets of connections , conv3 2 to conv2 2 , conv4 2 to conv3 3 , and conv5 2 to conv4 3 respectively . It is worth noting that all these connections go across pooling layers , for pooling layers can greatly enlarge the receptive field of neurons and enrich the contextual information of top-down information flow . For information flow in networks with multiple recurrent connections , take the network structure in Figure 2 as an example . The part between conv2 2 and conv5 2 will be unrolled for a certain number of times . To make the experiments setting consistent , we used model with two recurrent connections ( loop1 + loop2 ) in all the tasks . 4 EXPERIMENTS AND ANALYSIS . We first tested the Contextual Recurrent Convolutional model on standard image classification task including CIFAR-10 , CIFAR-100 , ImageNet and fine-grained image classification dataset CUB-200 . To display the robustness of our model , we showed its performance on noise image classification , adversarial attack and occluded images . We found that our model achieved considerate performance gain compared with the standard feedforward model on all these tasks . Notice that our proposed models are based on VGG16 with 2 recurrent connection ( loop1+loop2 in Figure 2 ) in all the tasks .

Summary: The paper proposes to add "recurrent" connections inside a convolution network with gating mechanism. The basic idea is to have higher layers to modulate the information in the lower layers in a convolution network. The way it is done is through upsampling the features from higher layers, concatenating them with lower layers and imposing a gate to control the information flow. Experiments show that the model is achieving better accuracy, especially in the case of noisy inputs or adversarial attacks.