Title: CONTEXT-GATED CONVOLUTION

Abstract: As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature. Many efforts have been recently devoted to complementing CNNs with the global modeling ability, especially by a family of works on global feature interaction. In these works, the global context information is incorporated into local features before they are fed into convolutional layers. However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs. Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context. As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features. Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

1. INTRODUCTION:

Convolutional Neural Networks (CNNs) have achieved significant successes in various tasks, e.g., image classification (He et al., 2016a; Huang et al., 2017) , object detection (Girshick et al., 2014; Ren et al., 2015) , image translation (Zhu et al., 2017) , action recognition (Carreira & Zisserman, 2017) , sentence/text classification (Zhang et al., 2015; Kim, 2014) , machine translation (Gehring et al., 2017) , etc. However, the sliding window mechanism of convolution makes it only capable of capturing local patterns, limiting its ability of utilizing global context. Taking the 2D convolution on the image as one example, as Figure 1a shows, the standard convolution only operates on the local image patch and thereby composes local features.

According to the recent research on neuroscience (Gilbert & Li, 2013) , neurons' awareness of global context is important for us to better interpret visual scenes, stably perceive objects and effectively process complex perceptual tasks. Many methods (Vaswani et al., 2017; Wang et al., 2017a; b; Park et al., 2018; Hu et al., 2018; Chen et al., 2019; Cao et al., 2019; Bello et al., 2019) have been proposed recently to introduce global context modeling modules into CNN architectures. In this paper, such a family of works is named as global feature interaction methods. As Figure 1b shows, these methods modulate intermediate feature maps by incorporating the global context with the local feature representation. For example, in Non-local modules (Wang et al., 2017b) , local features are reassembled according to global correspondence, which augments CNNs with the global context modeling ability.

As was discussed by Gilbert & Li (2013) , the global context influences neurons processing information in two distinct ways: "various forms of attention, including spatial, object oriented and feature oriented attention" and "rather than having a fixed functional role, neurons are adaptive processors, changing their function according to behavioral context". The previous work (Vaswani et al., 2017; Wang et al., 2017a; b; Park et al., 2018; Hu et al., 2018; Chen et al., 2019; Cao et al., 2019; Bello et al., 2019) of global feature interaction methods, shown in Figure 1b , only modifies intermediate features, namely, inputs of neurons, which corresponds to the first way. However, to the best of our knowledge, the other efficient and intuitive way, i.e., explicitly modulating the convolution kernels according to context, has not been exploited yet. Motivated by this, we will model convolutional layers as "adaptive processors" and explore how to leverage global context to guide the composition of local features in convolution operations.

In this paper, we propose Context-Gated Convolution (CGC), as Figure 1c shows, a new perspective of complementing CNNs with the awareness of the global context. Specifically, our proposed CGC learns a series of mappings to generate gates from the global context feature maps to modulate convolution kernels accordingly. With the modulated kernels, standard convolution is performed on input feature maps, which is enabled to dynamically capture representative local patterns and compose local features of interest under the guidance of global context. Our contributions are in three-fold.

• To the best of our knowledge, we make the first attempt of introducing the contextawareness to convolutional layers by modulating the weights of them according to the global context. 

2. CONTEXT-GATED CONVOLUTION:

2.1 PRELIMINARY Without loss of generality, we consider one sample of 2D case. The input to a convolutional layer is a feature map X ∈ R c×h×w , where c is the number of channels, and h, w are respectively the height and width of the feature map. In each convolution operation, a local patch of size c × k 1 × k 2 is collected by the sliding window to multiply with the kernel W ∈ R o×c×k1×k2 of this convolutional layer, where o is the number of output channels, and k 1 , k 2 are respectively the height and width of the kernel. Therefore, only local information within each patch is extracted in one convolution operation. Although in the training process, the convolution kernels are learned from all the patches from all the images in the training set, the kernels are not adaptive to the current context during inference time.

2.2. MODULE DESIGN:

In order to handle the aforementioned drawback of standard convolution, we propose to incorporate the global context information during the convolution process. Different from the existing approaches that modify the input features according to the context, e.g., a global correspondence of (1) and G (2) from C and O to construct the gate G. , denote convolution and element-wise multiplication operations, respectively. ⊕ is shown in Equation 1.

feature representations, we attempt to directly modulate the convolution kernel under the guidance of the global context information.

One simple and straightforward way of modulating the convolution kernel W with global context information is to directly generate a gate G ∈ R o×c×k1×k2 of the same size with W according to global context. Assuming that we generate the gate from a context vector v ∈ R l using a linear layer without the bias term, the number of parameters is l × o × c × k 1 × k 2 , which is extremely catastrophic when we modulate the convolution kernel of every convolutional layer. For modern CNNs, o and c can be easily larger than 100 or even 1,000, which makes o × c the dominant term in the complexity. Inspired by previous works on convolution kernel decomposition (Howard et al., 2017; Chollet, 2017) , we propose to decompose the gate G into two tensors G (1) ∈ R c×k1×k2 and G (2) ∈ R o×k1×k2 , so that the complexity of o × c can thereby significantly break down.

However, directly generating these two tensors is still not acceptable. Supposing that we generate them with two linear layers, the number of parameters is l × (o + c) × k 1 × k 2 , which is at the same scale with the number of parameters of the convolution kernel itself. The bottleneck now is jointly modeling channel-wise and spatial interactions, namely l and

(o + c) × k 1 × k 2 , considering that v ∈ R l is encoded from the input feature map X ∈ R c×h×w .

Inspired by depth-wise separable convolutions (Howard et al., 2017; Chollet, 2017) , we propose to model the spatial interaction and the channel-wise interaction separately to further reduce complexity.

In this paper, we propose one novel Context-Gated Convolution (CGC) to incorporate the global context information during the convolution process. Specifically, our proposed CGC consists of three modules: the Context Encoding Module, the Channel Interacting Module, and the Gate Decoding Module. As Figure 2 shows, the Context Encoding Module encodes global context information in each channel into a latent representation C via spatial interaction; the Channel Interacting Module projects the latent representation to the space of output dimension o via channel-wise interaction; the Gate Decoding Module produces G (1) and G (2) from the latent representation C and the projected representation O to construct the gate G via spatial interaction. The details of them are described in the following.

Context Encoding Module. To extract contextual information, we first use a pooling layer to reduce the spatial resolution to h × w and then feed the resized feature map to the Context Encoding Module. It encodes information from all the spatial positions for each channel, and extracts a latent representation of the global context. We use a linear layer with weight E ∈ R h ×w ×d to project the resized feature map in each channel to a latent vector of size d. Inspired by the bottleneck structure from (He et al., 2016a; Hu et al., 2018; Wang et al., 2017b; Vaswani et al., 2017) , we set d = k1×k2 2 to extract informative context, when not specified. The weight E is shared across different channels. A normalization layer and an activation function come after the linear layer. There are c channels, so the output of the Context Encoding Module is C ∈ R c×d .

Channel Interacting Module. It projects the feature representations C ∈ R c×d to the space of output dimension o. Inspired by (Ha et al., 2016) , we use a grouped linear layer I ∈ R 

EQUATION

where σ(•) denotes the sigmoid function. Now we have G with the same size of the convolution kernel W, which is generated from the global context by our lightweight modules. Then we can modulate the weight of a convolutional layer by element-wise multiplication to incorporate rich context information:

EQUATION

With the modulated kernel, a standard convolution process is performed on the input feature maps, where the context information can help the kernel capture more representative patterns and also compose features of interest.

Complexity. The computational complexity of our three modules is

O(c × d × h × w + c × o/g + c × d × k 1 × k 2 + o × d × k 1 × k 2 + o × c × k 1 × k 2 )

, where h , w can be set independent of h, w.

It is negligible compared to convolution's O(o × c × k 1 × k 2 × h × w).

Except the linear time of pooling, the complexity of these three modules is independent of the input's spatial size. The total number of parameters is

O(d × h × w + c × o/g 2 + 2 × d × k 1 × k 2 ), which is negligible compared to standard convolution's O(o × c × k 1 × k 2 ).

Therefore we can easily replace the standard convolution by our proposed CGC with a very limited computation and parameter increment, therefore enabling neurons to be adaptive to global context.

2.3. DISCUSSIONS:

We are aware of previous works on dynamically modifying the convolution operation (Dai et al., 2017; Wu et al., 2019; Jia et al., 2016; Jo et al., 2018; Mildenhall et al., 2018) . However, two key factors distinguish our approach from those works: whether the information guiding convolution is collected globally and how it changes the parameters of convolution. Dai et al. (2017) proposed to adaptively set the offset of each element in a convolution kernel, and Wu et al. (2019) proposed to dynamically generate the weights of convolution kernels. However, in their formulations, the dynamic mechanism for modifying convolution kernels only takes local patches or segments as inputs, so it is only adaptive to local inputs, which limits their ability of leveraging rich information in global context. According to experiments in Section 3.4, our proposed CGC significantly outperforms Dynamic Convolution (Wu et al., 2019) with the help of global context awareness.

Another family of works on dynamic filters (Jia et al., 2016; Jo et al., 2018; Mildenhall et al., 2018) generates weights of convolution kernels using features extracted from input images by another CNN feature extractor. The expensive feature extraction process makes it more suitable for generating a few filters, e.g., in the case of low-level image processing. It is impractical to generate weights for all the layers in a deep CNN model in this manner. However, our CGC takes input feature maps of a convolutional layer and makes it possible to dynamically modulate the weight of each convolutional layer, which systematically improves CNNs' global context modeling ability. 

3. EXPERIMENTS:

In this section, we demonstrate the effectiveness of our proposed CGC in incorporating 1D, 2D, and 3D context information in 1D, 2D, and (2+1)D convolutions. We conduct extensive experiments on image classification, action recognition, and machine translation, and observe that our CGC consistently improves the performance of modern CNNs with negligible parameter increment on four benchmark datasets: ImageNet (Russakovsky et al., 2015) , CIFAR-10 ( Krizhevsky et al., 2009) , Something-Something (v1) (Goyal et al., 2017) , and IWSLT'14 De-En (Cettolo et al.) .

3.1. IMPLEMENTATION DETAILS:

All of the experiments are based on PyTorch (Paszke et al., 2017) . All the linear layers are without bias terms. We follow common practice to use Batch Normalization (Ioffe & Szegedy, 2015) for computer vision tasks, or Layer Normalization (Ba et al., 2016) for natural language processing tasks, and ReLU (Nair & Hinton, 2010) as the activation function. Note that we learn different sets of scaling and shifting factors for C that is fed to the identity connection and for C that is fed to the Channel Interacting Module. We use average pooling with h = k 1 and w = k 2 , when not specified. Note that we only replace convolution kernels with a spatial size larger than 1. For those Point-wise convolutions, we take them as linear layers and do not modulate them. To reduce the size of I, we fix c/g = 16 when not specified. We initialize all these layers as what He et al. (2015) did for computer vision tasks and as what Glorot & Bengio (2010) did for natural language processing tasks. 

3.2. IMAGE CLASSIFICATION:

Experiment Setting. Following previous works (He et al., 2016a) on ImageNet, we train models on ImageNet 2012 training set, which contains about 1.28 million images from 1,000 categories, and report the results on its validation set, which contains 50,000 images. We replace all the 3 × 3 convolutions in ResNet-50 (He et al., 2016a) with our CGC and train the network from scratch. Note that for the first convolutional layer, we use I ∈ R 3×64 for the Channel Interacting Module. We follow common practice (He et al., 2016a) and apply minimum training tricks to isolate the contribution of our CGC. CIFAR-10 contains 50K training images and 10K testing images in 10 classes. We follow common practice (He et al., 2016b) to train and evaluate the models. We take ResNet-110 (He et al., 2016b) (with plain blocks) as the baseline model and test other possibilities of generating the gate G. All the compared methods are trained based on the same training protocol1 . The details are provided in appendix. For evaluation, we report Top-1 and Top-5 accuracy of a single crop with the size 224 × 224 for ImageNet and 32 × 32 for CIFAR-10, respectively.

Performance Results. As Table 1 shows, our CGC significantly improves the performance of baseline models on both ImageNet and CIFAR-10. On ImageNet, our CGC improves the Top-1 accuracy of ResNet-50 by 1.12% with only 0.03M more parameters and 6M more FLOPs, which verifies our CGC's effectiveness of incorporating global context and its efficiency. We also find that GC-ResNet-50 is hard to train from scratch unless using the fine-tuning protocol reported by Cao et al. (2019) , which indicates that modifying features may be misleading in the early training process. Although our CGC introduces a few new parameters, our model converges faster and more stably compared to vanilla ResNet-50, as is shown in Figure 3 . We suppose that this is because the adaptiveness to global context improves the model's generalization ability and the gating mechanism reduces the norm of gradients back-propagated to the convolution kernels, which leads to a smaller Lipschitz constant and thus better training stability (Santurkar et al., 2018; Qiao et al., 2019) .

Ablation Study. In order to demonstrate the effectiveness of our module design, ablation studies are conducted on CIFAR-10, as illustrated in Table 2 . Specifically, we ablate many variants of our CGC and find our default setting a good trade-off between parameter increment and performance gain. The experiments on the combination of G (1) and G (2) show that our decomposition approach in Equation 1 is a better way to construct the gate. For channel interacting, we find that using a full linear model with g = 1 achieves better performance with more parameters, as is expected. We try removing the bottleneck structure and set d = k 1 × k 2 , and the performance drops, which validates the necessity of the bottleneck structure. Shared Norm indicates learning the same set of scaling and shifting factors for C in the following two branches. Two Es indicates that we learn another E to encode C only for the Channel Interacting Module. We also try sharing D for generating G (1) and G (2) , using larger resized feature maps and using max pooling instead of average pooling. All the results support our default setting. We also test different numbers of layers to replace standard convolutions with our CGC. The result indicates that the more, the better.

3.3. ACTION RECOGNITION:

Baseline Methods. For the action recognition task, we adapt three baselines to evaluate the effectiveness of our CGC: TSN (Wang et al., 2016) , P3D-A (Qiu et al., 2017) (details are in appendix), and TSM (Lin et al., 2018) . Because our CGC's effectiveness of introducing 2D spatial context to CNNs has been verified in image classification, in this part, we focus on its ability of incorporating 1D temporal context and 3D spatial-temporal context. For the 1D case, we apply our CGC to temporal convolutions in every P3D-A block. For the 3D case, we apply our CGC to spatial convolutions in P3D-A or 2D convolutions in TSN or TSM; the pooling layer produces c × k × k × k cubes, the Context Encoding Module encodes k × k × k feature maps into a vector of length k 3 /2 and the Gate Decoding Module generates o × c × t × k × k gates. Note that for the first convolutional layer, we use I ∈ R 3×64 for the Channel Interacting Module.

Experiment Setting. The Something-Something (v1) dataset has a training split of 86,017 videos and a validation split of 11,522 videos, with 174 categories. We follow (Qiao et al., 2019) to train on the training set and report evaluation results on the validation set. We follow Lin et al. (2018) to process videos and augment data. Since we only use ImageNet for pretraining, we adapt the code base of TSM but the training setting from Qiao et al. (2019) . We train TSN-and TSM-based models for 45 epochs (50 for P3D-A), start from a learning rate of 0.025 (0.01 for P3D-A), and decrease it by 0.1 at 26 and 36 epochs (30, 40, 45 for P3D-A). For TSN-and TSM-based models, the batch size is 64 for 8-frame models and 32 for 16-frame models, and the dropout rate is set to 0.5. P3D-A takes 32 continuously sampled frames as input and the batch size is 64, and the dropout ratio is 0.8. We use the evaluation setting of Lin et al. (2018) for TSN-and TSM-based models and the evaluation settings of Wang et al. (2017b) for P3D-A. All the models are trained with 8-GPU machines.

Performance Comparisons. As Table 3 shows, our CGC significantly improves the performance of baseline CNN models, compared to Non-local (Wang et al., 2017b) . As aforementioned, Non- (Vaswani et al., 2017) 39.47M 34.41 DynamicConv (Wu et al., 2019) 38.69M 35.16 LightConv (Wu et al., 2019) 38.14M 34.84 LightConv + Dynamic Encoder (Wu et al., 2019) local modules modify the input feature maps of convolutional layers by reassembling local features according to the global correspondence. We apply Non-local blocks in the most effective way as is reported by Wang et al. (2017b) . However, we observe that its performance gain is not consistent when training the model from scratch, which again indicates that modifying features according to the global correspondence may be misleading in the early training process. When applied to TSM, it even degrades the performance. Our proposed CGC consistently improves the performance of all the baseline models. When applied to TSM, our CGC yields the state-of-the-art performance, when without Kinetics (Carreira & Zisserman, 2017) pretraining, with only RGB modality and with negligible parameter increment.

3.4. MACHINE TRANSLATION:

Baseline Methods. The LightConv proposed by Wu et al. (2019) achieves better performances with a lightweight convolutional model, compared to Transformer (Vaswani et al., 2017) . We take it as the baseline model and augment their Lightweight Convolution with our CGC. Note that the Lightweight Convolution is a grouped convolution L ∈ R H×k with weight sharing, so we remove the Channel Interacting Module since we do not need it to project latent representations. We resize the input sequence S ∈ R c×L to R H×3k with average pooling. For those sequences shorter than 3k, we pad them with zeros. Since the decoder decodes translated words one by one at the inference time, it is unclear how to define global context for it. Therefore, we only replace the convolutions in the encoder.

Experiment Setting. We follow Wu et al. (2019) to train all the compared models with 160K sentence pairs and 10K joint BPE vocabulary. We use the training protocol of DynamicConv (Wu et al., 2019) provided in Ott et al. (2019) . The widely-used BLEU-4 (Papineni et al., 2002) is reported for evaluation of all the models. We find that it is necessary to set beam width to 6 to reproduce the results of DynamicConv reported in (Wu et al., 2019) , and we fix it to be 6 for all the models.

Performance Comparisons. As Table 4 shows, replacing Lightweight Convolutions in the encoder of LightConv with our CGC significantly outperforms LightConv and LightConv + Dynamic Encoder by 0.37 and 0.18 BLEU, and yields the state-of-the-art performance. As is discussed previously, Dynamic Convolution leverages a linear layer to generate the convolution kernel according to the input segment, which lacks the awareness of global context. This flaw may lead to sub-optimal encoding of the source sentence and thus the unsatisfied decoded sentence. However, our CGC incorporates global context of the source sentence and helps significantly improve the quality of the translated sentence. Moreover, our CGC is much more efficient than Dynamic Convolution because of our module design. Our CGC only needs 0.01M extra parameters, but Dynamic Convolution needs 30× more.

4. RELATED WORKS:

There has been much effort in augmenting CNNs with context information. They can be roughly categorized into three types: first, adding backward connections in CNNs (Stollenga et al., 2014; Zamir et al., 2017; Yang et al., 2018) to model the top-down influence (Gilbert & Li, 2013) like humans' visual processing system; second, modifying intermediate feature representations in CNNs according to attention mechanism (Vaswani et al., 2017; Wang et al., 2017b) ; third, dynamically generating the parameters of convolution according to local or global information (Jia et al., 2016; Noh et al., 2016; Dai et al., 2017; Wu et al., 2019) .

For the first category of works, it is still unclear how the feedback mechanism can be effectively and efficiently modeled in CNNs. For example, Yang et al. (2018) proposed an Alternately Updated Clique to introduce feedback mechanisms into CNNs. However, compared to standard CNNs, the complex updating strategy increases the difficulty for training them as well as the latency at the inference time. The second category of works is the global feature interaction methods. They (Vaswani et al., 2017; Wang et al., 2017a; b; Park et al., 2018; Hu et al., 2018; Chen et al., 2019; Cao et al., 2019; Bello et al., 2019) were proposed recently to modify local features according to global context information, usually by a global correspondence, i.e. the self-attention mechanism. There are also works on reducing the complexity of self-attention mechanism (Parmar et al., 2018; Child et al., 2019) . However, this family of works only considers changing the input feature maps. The third type of works is more related to our work. As is discussed before, our approach is distinct from them in two key factors.

5. CONCLUSION:

In this paper, motivated by neuroscience research on neurons as "adaptive processors", we proposed Context-Gated Convolution (CGC) to incorporate global context information into CNNs. Different from previous works which usually modifies input feature maps, our CGC directly modulates convolution kernels under the guidance of global context information. We proposed three modules to efficiently generate a gate to modify the kernel. As such, our CGC is able to extract representative local patterns according to global context. The extensive experiment results show consistent performance improvements on various tasks. There are still a lot of future works that can be done. For example, ew could design task-specific gating modules to fully uncover the potential of the proposed CGC.

Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 695-712, 2018.

A APPENDIX:

A.1 DETAILS OF TRAINING SETTINGS ON IMAGENET AND CIFAR-10

For ImageNet, we use 224 × 224 random resized cropping and random horizontal flipping for data augmentation. Then we standardize the data with mean and variance per channel. We use a standard cross-entropy loss to train all the networks with a batch size of 256 on 8 GPUs by SGD with a weight decay of 0.0001 and a momentum of 0.9 for 100 epochs. We start from a learning rate of 0.1 and decrease it by a factor of 10 every 30 epochs. For CIFAR-10, we use 32 × 32 random cropping with a padding of 4 and random horizontal flipping. We use a batch size of 128 and train on 1 GPU. We decrease the learning rate at the 81st and 122nd epochs, and ends training after 164 epochs.

A.2 DETAILS ABOUT P3D-A:

Based on ResNet-50, we add a temporal convolution with k = 5, stride = 2 after the first convolutional layer. For convolutional layers in residual blocks, we follow Wang et al. (2017b) to add 3 × 1 × 1 convolution (stride is 1) after every two 1 × 3 × 3 convolutions. We only inflate the max pooling layer after the first convolutional layer with a temporal kernel size of 3 and a stride of 2 without adding any other temporal pooling layers. Note that all the aforementioned convolutional layers come with a Batch Normalization layer and a ReLU activation function. the same class, adjusting the kernels adaptively is beneficial for correct classification, which is consistent with the neuroscience research that motivates our CGC.

:

The code is based on https://github.com/bearpaw/pytorch-classification

Figures and Tables:

Figure 1: (a) Standard convolution only composes local information. (b) Global feature interaction methods modify input feature maps by incorporating global information. (c) Our CGC, in a fundamentally different manner, modulates convolution kernels under the guidance of global context. denotes convolution.

g is the number of groups. The weight I is shared among different dimensions of d and different groups. A normalization layer and an activation function come after the linear layer. The final output of the Channel Interacting Module is O ∈ R o×d . Gate Decoding Module. It takes both C and O as inputs, and decodes the latent representations to the spatial size of convolution kernels. We use two linear layers whose weights D c ∈ R d×k1×k2 and D o ∈ R d×k1×k2 are respectively shared across different channels in C and O. Then each element in the gate is produced by

Figure 3: The training curves of ResNet-50 and ResNet-50 + CGC (ours) on ImageNet.

Figure 5: Visualization of the difference matrix between inter-class distances and intra-class distances of the last gate in the network on ImageNet validation set. (Best viewed on a monitor when zoomed in)

Our proposed CGC consists of three components, namely the Context Encoding Module, the Channel Interacting Module, and the Gate Decoding Module. The Context Encoding Module encodes global context information into a latent representation C; the Channel Interacting Module transforms C to O with output dimension o; the Gate Decoding Module produces G

Table content: <table><tr><td>𝐗</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>⊛</td></tr><tr><td colspan="2">Convolution</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>𝐖</td></tr><tr><td>Pooling Context 𝑐 × ℎ × 𝑤 Gating</td><td>𝑐 × ℎ′ × 𝑤′</td><td colspan="2">Context Encoding</td><td>𝐶 𝑐 × 𝑑</td><td colspan="2">Channel Interacting</td><td>𝑜 × 𝑑 𝑂</td><td/><td>Gate Decoding</td><td/><td>𝑜 × 𝑐 × 𝑘 1 × 𝑘 2 𝐆</td><td>⊙</td></tr><tr><td/><td>𝑐 × ℎ′𝑤′</td><td>Linear</td><td>Normalization</td><td>Activation</td><td>𝑐 × 𝑑</td><td>Grouped Linear</td><td>Normalization</td><td>Activation</td><td>𝑜 × 𝑑 𝑐 × 𝑑</td><td>Linear</td><td>𝑜 × 𝑘 1 × 𝑘 2 𝐆 (𝟏) 𝑐 × 𝑘 1 × 𝑘 2 𝐆 (𝟐)</td><td>𝐖</td></tr><tr><td>Figure 2:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Image classification results on ImageNet and CIFAR-10. Param indicates the number of parameters in the model. ∆MFLOPs is the increment of the number of multiplication-addition operations compared to ResNet-50 (4 GFLOPs) for ImageNet models and ResNet-110 (256 MFLOPs) for CIFAR-10 models. Bold indicates the best result.

Table content: <table><tr><td>Dataset</td><td/><td>Models</td><td/><td/><td/><td/><td>Param</td><td>∆MFLOPs Top-1(%) Top-5(%)</td></tr><tr><td/><td/><td colspan="5">SE-ResNet-50 (Hu et al., 2018)</td><td>28.09M 8</td><td>77.18</td><td>93.67</td></tr><tr><td/><td/><td colspan="5">BAM-ResNet-50 (Park et al., 2018)</td><td>25.92M 83</td><td>76.90</td><td>93.40</td></tr><tr><td/><td/><td colspan="5">GC-ResNet-50 (Cao et al., 2019)</td><td>28.11M 8</td><td>73.90</td><td>91.70</td></tr><tr><td/><td/><td colspan="4">ResNet-50 (He et al., 2016a)</td><td/><td>25.56M -</td><td>76.18</td><td>92.91</td></tr><tr><td>ImageNet</td><td/><td colspan="4">ResNet-50 + CGC(Ours)</td><td/><td>25.59M 6</td><td>77.30</td><td>93.66</td></tr><tr><td/><td/><td colspan="6">CBAM-ResNet-50 (Woo et al., 2018) 28.09M 15</td><td>77.34</td><td>93.69</td></tr><tr><td/><td/><td colspan="5">CBAM-ResNet-50 + CGC(Ours)</td><td>28.12M 21</td><td>77.68</td><td>93.68</td></tr><tr><td>CIFAR-10</td><td/><td colspan="4">ResNet-110 (He et al., 2016b) ResNet-110 + CGC(Ours)</td><td/><td>1.73M 1.80M</td><td>-2</td><td>93.96 94.86</td><td>99.73 99.82</td></tr><tr><td>0</td><td>20</td><td>40</td><td>Epoch</td><td>60</td><td>80</td><td>100</td></tr></table>

Ablation studies on CIFAR-10. Param indicates the number of parameters in the model. ∆MFLOPs is the increment of the number of multiplication-addition operations compared to. Bold indicates our default setting.

Table content: <table><tr><td>Models</td><td colspan="2">Param ∆MFLOPs Top-1 Acc (%)</td></tr><tr><td>ResNet-110 (He et al., 2016b)</td><td>1.73M -</td><td>93.96</td></tr><tr><td>ResNet-110 + CGC</td><td>1.80M 1.681</td><td>94.86</td></tr><tr><td>ResNet-110 + CGC only G (1)</td><td>1.75M 1.447</td><td>94.53</td></tr><tr><td>ResNet-110 + CGC only G (2)</td><td>1.78M 1.472</td><td>94.41</td></tr><tr><td>ResNet-110 + CGC G (1)  *  G (2)</td><td>1.80M 1.681</td><td>94.59</td></tr><tr><td>ResNet-110 + CGC g = 1</td><td>1.96M 1.681</td><td>94.97</td></tr><tr><td>ResNet-110 + CGC d = k1 × k2</td><td>1.81M 1.741</td><td>94.61</td></tr><tr><td>ResNet-110 + CGC Shared Norm</td><td>1.79M 1.681</td><td>94.72</td></tr><tr><td>ResNet-110 + CGC Two Es</td><td>1.80M 1.871</td><td>94.53</td></tr><tr><td>ResNet-110 + CGC Shared D</td><td>1.79M 1.681</td><td>94.78</td></tr><tr><td colspan="2">ResNet-110 + CGC h = 2k1, w = 2k2 1.81M 1.681</td><td>94.68</td></tr><tr><td>ResNet-110 + CGC MaxPool</td><td>1.80M 1.681</td><td>94.44</td></tr><tr><td>ResNet-110 + CGC (res1,2,3)</td><td>1.80M 1.678</td><td>94.55</td></tr><tr><td>ResNet-110 + CGC (res2,3)</td><td>1.78M 1.052</td><td>94.43</td></tr><tr><td>ResNet-110 + CGC (res3)</td><td>1.76M 0.622</td><td>94.26</td></tr></table>

Action recognition results on Something-Something (v1). Backbone indicates the backbone network architecture. Param indicates the number of parameters in the model. Frame indicates number of frames used for evaluation. Bold indicates the best result.

Table content: <table><tr><td>Models</td><td>Backbone</td><td>Param</td><td>Frame</td><td colspan="2">Top-1(%) Top-5(%)</td></tr><tr><td>TRN (Zhou et al., 2018)</td><td>BNInception</td><td>18.3M</td><td>8</td><td>34.4</td><td>-</td></tr><tr><td>TRN (Lin et al., 2018)</td><td>ResNet-50</td><td>31.8M</td><td>8</td><td>38.9</td><td>68.1</td></tr><tr><td>ECO (Zolfaghari et al., 2018)</td><td colspan="2">BNInc+Res18 47.5M</td><td>8</td><td>39.6</td><td>-</td></tr><tr><td>ECO (Zolfaghari et al., 2018)</td><td colspan="2">BNInc+Res18 47.5M</td><td>16</td><td>41.4</td><td>-</td></tr><tr><td>ECOEnLite (Zolfaghari et al., 2018)</td><td colspan="2">BNInc+Res18 150M</td><td>92</td><td>46.4</td><td>-</td></tr><tr><td>TSN (Wang et al., 2016)</td><td>ResNet-50</td><td colspan="2">23.86M 8</td><td>19.00</td><td>44.98</td></tr><tr><td colspan="2">TSN + Non-local (Wang et al., 2017b) ResNet-50</td><td colspan="2">31.22M 8</td><td>25.73</td><td>55.17</td></tr><tr><td>TSN + CGC (Ours)</td><td>ResNet-50</td><td colspan="2">24.07M 8</td><td>32.58</td><td>60.06</td></tr><tr><td>P3D (Qiu et al., 2017)</td><td>ResNet-50</td><td colspan="3">25.38M 32 × 30 45.17</td><td>74.61</td></tr><tr><td>P3D + Non-local (Wang et al., 2017b)</td><td>ResNet-50</td><td colspan="3">32.73M 32 × 30 45.88</td><td>74.94</td></tr><tr><td>P3D + CGC 1D (Ours)</td><td>ResNet-50</td><td colspan="3">25.39M 32 × 30 46.14</td><td>75.92</td></tr><tr><td>P3D + CGC 3D (Ours)</td><td>ResNet-50</td><td colspan="3">25.61M 32 × 30 46.35</td><td>75.97</td></tr><tr><td>P3D + CGC 1D &amp; 3D (Ours)</td><td>ResNet-50</td><td colspan="3">25.62M 32 × 30 46.53</td><td>76.00</td></tr><tr><td>TSM (Lin et al., 2018)</td><td>ResNet-50</td><td colspan="2">23.86M 8</td><td>44.65</td><td>73.94</td></tr><tr><td colspan="2">TSM + Non-local (Wang et al., 2017b) ResNet-50</td><td colspan="2">31.22M 8</td><td>43.91</td><td>72.18</td></tr><tr><td>TSM + CGC (Ours)</td><td>ResNet-50</td><td colspan="2">24.07M 8</td><td>46.00</td><td>75.11</td></tr><tr><td>TSM (Lin et al., 2018)</td><td>ResNet-50</td><td colspan="2">23.86M 16</td><td>46.61</td><td>76.18</td></tr><tr><td>TSM + CGC (Ours)</td><td>ResNet-50</td><td colspan="2">24.09M 16</td><td>47.87</td><td>77.22</td></tr></table>

Machine translation results on IWSLT'14 De-En. Param indicates the number of parameters in the model. Bold indicates the best result.

Table content: <table><tr><td>Models</td><td>Param</td><td>BLEU-4</td></tr><tr><td>(Deng et al., 2018)</td><td>-</td><td>33.08</td></tr><tr><td>Transformer</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes a novel and interesting idea of modulating convolutional kernels with global context. They use depthwise factorized channel-wise interaction to reduce complexity and parameters. While an interesting idea, I found it hard to see how the global context would usefully modulate the kernels. While Figure 1 illustrates the idea, it is not clear how the modulated kernel is better suited for the task relative to the original kernel. It would be nice if you could show a simple/toy example where your modulation clearly modifies the kernel in a way that is intuitively useful. Overall, I believe the motivation for the method could be made clearer - Why is this the best way to incorporate global context? It is also unclear to me how the proposed network relates to the motivating and cited Neuroscience literature. It is nice that the computational complexity is carefully tracked. The CGC network does seem to converge faster but the benefits to validation accuracy seem modest in Table 1/Fig 3. It would be interesting to further explore what is improving the convergence speed -i.e. to test your supposition that it *improves the model*s generalization ability and the gating mechanism reduces the norm of gradients back-propagated to he convolution kernels ...*. The experiments are well described and extensive and some of them do give some more substantial performance improvements, but overall I think the paper does not explain the motivation for the algorithm sufficiently for ICLR publication.