Contextual Convolutional Networks
Authors: Shuxian Liang
Xu Shen
Tongliang Liu
Xian-Sheng Hua
Venue: ICLR 2023 poster (2023)
Keywords: Convolutional Neural Networks

Abstract: This paper presents a new Convolutional Neural Network, named Contextual Convolutional Network, that capably serves as a general-purpose backbone for visual recognition. Most existing convolutional backbones follow the representation-to-classification paradigm, where representations of the input are firstly generated by category-agnostic convolutional operations, and then fed into classifiers for specific perceptual tasks (e.g., classification and segmentation). In this paper, we deviate from this classic paradigm and propose to augment potential category memberships as contextual priors in the convolution for contextualized representation learning. Specifically, top-k likely classes from the preceding stage are encoded as a contextual prior vector. Based on this vector and the preceding features, offsets for spatial sampling locations and kernel weights are generated to modulate the convolution operations. The new convolutions can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation without additional supervision. The qualities of Contextual Convolutional Networks make it compatible with a broad range of vision tasks and boost the state-of-the-art architecture ConvNeXt-Tiny by 1.8% on top-1 accuracy of ImageNet classification. The superiority of the proposed model reveals the potential of contextualized representation learning for vision tasks. Code is available at: \url{https://github.com/liang4sx/contextual_cnn}.


Published as a conference paper at ICLR 2023
CONTEXTUAL CONVOLUTIONAL NETWORKS
Shuxian Liang!?; Xu Shen?, Tongliang Liu', Xian-Sheng Hua!t 'Zhejiang University, Alibaba Cloud Computing Ltd., "yydney AI Centre, The University of Sydney
shuxian. lsx8zju. edu. cn, shenxuustc@@maill com, tongliang. liuu sysddeyy..... au, huaxianshengeee com
ABSTRACT
This paper presents Convolutional Neural Network, named Contextual Con- volutional Network, that capably serves as a general-purpose backbone for visual recognition. Most existing convolutional backbones follow the representation-to- classification paradigm, where representations of the input are firstly generated by category-agnostic convolutional operations, and then fed into classifiers for specific perceptual tasks (e.g.. classification and segmentation). In this paper, we deviate from this classic paradigm and propose to augment potential category mem- berships as contextual priors in the convolution for contextualized representation learning. Specifically, top-k likely classes from the preceding stage are encoded contextual prior vector. Based this vector and the preceding features, off- sets for spatial sampling locations and kernel weights are generated to modulate the convolution operations. The new convolutions can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back puck-propagation without additional supervision. The qualities of Contextual Con- volutional Networks make it compatible with a broad range of vision tasks and boost the state-of-the-art architecture ConvNeXt-Tiin by 1.8% top-1 accuracy of ImageNet classification. The superiority of the proposed model reveals the po- tential of contextualized representation learning for vision tasks. Code available at: https: / wtt.:cccccccooo.mo hwtp:///////////////oeeettttttte _al__nn]
INTRODUCTION
Beginning with the AlexNet (Krizhevsky et al., 2012) and its revolutionary performance the ImageNet image classification challenge, convolutional neural networks (CNNs) have achieved significant success for visual recognition tasks, such as image classification (Deng et al. 2009). instance segmentation (Zhou et al., 2017) and object detection (Lin et al. 2014). Lots powerful CNN backbones proposed to improve the performances, including greater scale (Simonyan
Zisserman 2014; (Szeedyy et al.] [015]] He et al., 2016),, more extensive connections (Huang al.[ Xie al. 2017.Sum et al. 2019 Yang et al.) | 2018), and more sophisticated forms of
convolution (Dai et al. 2017 Zhu et al. 2019 Yang et al. 2019). Most of these architectures follow
the pppresenttioon-toclasiiieeeee by category-agnostic convolutional operations, and then fed into classifiers for specific perceptual tasks. Consequently, all inputs are processed by consecutive static convolutional operations and expressed as universal representations.
paradigm, where representations of the input are firstly generated
In parallel, in the neuroscience community, evidence accumulates that human visual system integrates both bottom-up processing from the retina and top-down modulation from higher-order cortical areas (Rao & Ballard 1999 Lee & Mumford, 2003 Friston 2005). On the one hand, the bottom-up processing is based on feedforward connections along a hierarchy that represents progressively more complex aspects visual scenes (Gilbert & Sigman 2007). This property is shared with pinseentttioo-to-lsliiiiiee
the aforementioned 2014). On the other hand, recent findings suggest that the top-down modulation affects the bottom-up processing in way that enables the neurons to carry more information about the stimulus being
paradigm (Zeiler & Fergus 2014 Yamins et al.)
"This work was done when the author was visiting Alibaba as a research intern.
Corresponniig author.

---

Published as a conference paper at ICLR 2023
predictions
Roggl Sodi- Vigglo
Noethoese (GT)
Bick
cluck
chanch
Inkcedde
Cam-
top-s predictions
(CT)
Imag
heshhese
cooc
D
charch
tooatile
(a) 3 3 contextual convolution
(b) visualization of learned features
Figure 1: Left: Illustration of 3 x 3 contextual convolution. Given preceding instance features and top-k likely classes, sampling offsets and weight offsets are generated via non-linear layers. These offsets are added to regular grid sampling locations and static kernel weights of standard convolution, respectively. Right: Grad-CAM visualization (Selvaraju et al. (017) of the learned features of ConvNeXt-T (Liu et al. 2022) and our model. Grad-CAM is used to interpret the learned features by highlighting corresponding regions that discriminate the predicted class from other classes.
discriminated (Gilbert & Li 2013). For example, recordings in the prefrontal cortex reveal that the same neuron can be modulated to express different categorical representations as the categorical context changes (e.g.. from discriminating animals to discriminating cars) (Cromer et al. 2010 Gilbert & Li, 2013). Moreover, words with categorical labels (e.g., "chair") can set visual priors that alter how visual information is processed from the very beginning. allowing for more effective representational separation of category members and nonmembers (Lupyan & Ward 2013 Boutonnet & LLpyyan 2015). The top-down modulation can help to resolve challenging vision tasks with complex scenes or visual distractors. This property is however not considered by recent CNN backbones yet.
Motivated by the top-down modulation with categorical context in the brain, we present novel archi- tecture, namely Contextual Convolutional Networks (Contextual CNN), which augments potential category memberships as contextual priors in the convolution for representation learning. Specifically, the top-k likely classes by far are encoded as a contextual vector. Based on this vector and preceding features, offsets for spatial sampling locations and kernel weights are generated to modulate the convolutional operations in the current stage (illustrated in Fig. Tap. The sampling offsets enable free form deformation of the local sampling grid considering the likely classes and the input instance, which modulates where to locate information about the image being discriminated. The weight offsets allow the adjustment of specific convolutional kernels (e.g. "'egges" "corners"), which modulates how to extract discriminative features from the input image. Meanwhile, the considered classes are reduced from à to m (m < k) and fed to the following stage for further discrimination. By doing so, the following stage of convolutions is conditioned on the results of the previous, thus rendering convolutions dynamic in a smart way.
The proposed contextual convolution can be used drop-in replacement for existing convolutions in CNNs and trained end-to-end without additional supervision. Serving as a general-purposed backbone, the newly proposed Contextual CNN is compatible with other backbones or methods in a broad range of vision tasks, including image classification, video classification and instance segmentation. Its performance surpasses the counterpart models by a margin of +1.8% top-1 accuracy (with 3% additional computational cost) on ImageNet-1K K (Deng et al. 2009). 2.3% top-1 accuracy on Kinetics-400 (Kay et al. (0017). +1.1% box AP and + +10% mask AP on COCO (Lin et al. 2014). demonstrating the potential of contextualized representation learning for vision tasks. The qualitative results also reveal that Contextual CNN can take on selectivity for discriminative features according to the categorical context, functionally analogous to the top-down modulation of the human brain. As shown in Figure lb] the counterpart model presents high but wrong score for "boathouse" w.r.t. the groundtruth class "pier" based on features of the oceanside house, which are shared across images of both classes. In contrast, the proposed model predicts correctly by generating features of the long walkway stretching from the shore to the water, which is crucial cue to discriminate "pier" from "boathouse", We hope that Contextual CNN's strong performance on various vision problems can promote the research on new paradigm of convolutional backbone architectures.
2

---

Published as a conference paper at ICLR 2023
2
RELATED WORKS
Classic CNNs. The exploration of CNN architectures has been an active research area. VGG nets (Simonyan & Zisserman 2014) and GoogLeNet (Szegedy et 2015) demonstrate the benefits of increasing depth. ResNets (He et al. 2016) verify the effectiveness of learning deeper networks via residual mapping. Highway Network adopts a gating mechanism adjust the routing of short connections between layers. More recently, some works include more extensive connections to further improve the capacity of CNNs. For example, DenseNet (Huang et al., 2017) connects each layer to every other layer. ResNeXt (Xie et 2017) aggregates set of transformations via grouped convolutions. SENet (Hu et al.
2018) recalibrates channel-wise feature response using 2020) connects the high-to-low resolution convolution
global information. HRNet (Wang et al.
streams in parallel. FlexConv (Romero et al.
(022) learns the sizes of convolutions from training data.
Other recent works improve the efficiency of CNNs by introducing depthwise separable convolutions (Howard et al. 2017) and shift operation (Wu et al. 2018a).
Dynamic CNNs. Different from the classic CNNs, dynamic CNNs adapt their structures or parameters to the input during inference, showing better accuracy or efficiency for visual recognition. One line of work drops part of an existing model based on the input instance. For example, some works skip convolutional layers on a per-input basis, based on either reinforcement learning [Wang et al.) 2018
Wu et al. 2018b) or early-exit mechanism (Huang et al.] 2018). The other line of work uses dynamic kernel weights or dynamic sampling offsets for different inputs. Specifically, some works aggregate multiple convolution kernels using attention (Yang al., 2019; Chen et al. 2020) or channel fusion (Li ct al. 2021). WeightNet Ma et al. (2020) unifies kernel aggregation and the channel excitation via grouped fully-connected layers. Dai et al. (2017) and Zhu et al. (2019) learn different sampling offsets of convolution for each input image. The proposed Contextual CNN shares some similar high level spirit with these dynamic CNNs. A key difference of our method is that we explicitly adopt potential category memberships contextual priors constrain the adaptive inference.
There are some recent CNN architectures which have shared the same term "context" (Duta et al. 2021 Marwood & Baluja 2021p. The differences of our method lie in two folds. First, the context in our work refers to the category priors (i.e., top-k likely classes) of each input while theirs refer to the boarder receptive field of the convolutions. Second, our work modulates convolutional kernels (via weight/sampling offsets) according to the category priors. They adopt multi-level dilated convolutions and use soft attention on spatial channel dimensions, respectively. Neither of them leverages the category priors, nor modulates the convolutions to extract category-specific features.
3 CONTEXTUAL CONVOLUTIONAL NETWORKS
In this section, we describe the proposed Contextual Convolutional Networks. The overall architecture introduced in Section 3.1 In Section 3.2 present contextualizing layers, which generate contextual priors for Contextual CNN. In Section 3.3 we present contextual convolution blocks, which modulate convolutions according the contextual priors. Without loss of generality, this section is based on the ConyNeXt-T version (Liu et al. 2022), namely Contextual Con-NeXt-T. Detailed architecture specifications of Contextual CNN are the supplementary material.
3.1 OVERALL ARCHITECTURE
An overview of Contextual CNN is presented in Figure 2 Consider an input RGB image of size H W x 3, where H is the height and W is the width. Ni is the number of all possible classes for the image (for example, N, is 1000 for ImageNet-1K (Deng et al. 2009).. A set of class embeddings
ε = f ej No of E Rdd is generated from an embedding layer of size N₁ x d. These class embeddings are constant in the whole network and form the basis of contextual priors.
Following common practice in prior vision backbones (He et al. 2016 Xie et al. 2017: Radosavovic al. 2020 Liu et al. 2022), Contextual CNN consists of stem ("Stem" that preprocesses the input image, a network body ("Stagel-4") that generates the feature maps of various granularities and a final head that predicts the output class. These components are introduced as follows.
3

---

Published as a conference paper at ICLR 2023
H W
M.
If W
H W
H W 12
16'"1h
Sigee
Stage2
Stage3
Visilla
Contectual
Contectaal
Contestral Conv
She
Okrpui Clos
Irage
Chen
Connetraiin
Coev
Costestalion
Coav
Contetration
Block
Block
Block
Black
xt
Ny Choose
Clacaes
N₁ Classss
Classes
N, Classes
Class
Figure 2: The architecture of a Contextual Convolutional Network (Contextual ConyNext-T). For simplicity, we denote the downsampling layers at "Stage2-4" by "down.".
The stem. Consistent with the standard design ConyNeXt-T, the stem of Contexual ConyNeXt-T uses 4 stride convolution. The stem results in 4x downsampling of the input image, while the output features have C' = 96 channels.
The body. Several vanilla convolutional blocks are applied on the output features of the stem. Maintaining the resolution of H W these blocks have an output channel size of C' and share the same architecture as their ConvNeXt counterparts. These blocks are referred to as "Stagel" in this work. A contextualizing layer (described in Section 3.2) is used afterwards to extract the contextual prior from the most likely classes. It reduces the number of considered classes from N1 to N₂ and merges the embeddings of the output N2 classes as contextual prior. In parallel, vanilla convolutional layer used to 2x downsample the resolution to x g and doubles the channel size to 2C. Taking the contextual prior and the downsampled feature map inputs, few contextual convolution blocks (described in Section 3.3) are applied for feature extraction. The contextualizing layer, the downsampling layer and the following blocks are denoted "Stage 2". The procedure is repeated twice, "Stage 3" and "Stage 4" Noting that "Stage 3" has a feature map resolution of TO I6. maintains 4C channels and reduces the number of considered classes to Ng. "Stage 4" has feature map resolution of 32 32: maintains 8C channels and reduces the number of considered classes to N4. As the number of considered classes reduces gradually, the contextual prior conveys more fine-grat rreeggaaieed categorical information (shown in Fig. [aa,, which modulates the higher-level contextual convolutions to extract more discriminative features accordingly. Details of choosing the numbers of considered classes (N2, N3 and N₂) are in the supplementary material.
I :
Hf W
The head. The head of Contextual CNN shares the same procedure of reducing considered classes in the contextualizing layers (i.c., classifying in Section 3.22). And it finally reduces the number of classes from N₂ to 1. The final one class is used the output class for the input image.
The loss. The number of considered classes are reduced progressively in Contextual CNN (Ni -> N₂ N₂ N₂ + 1). For stage (i E 2, 4)). adopt cross entropy loss over the corresponding classification scores 8; (introduced in Section (3.2p. Following Radford et al. (2021) given the set of considered classes N₁ (LNa N.), the loss is calculated by:
L, = -T T (y E NG) log exp (8, (v) /T)
Σn exp (st (N²) (Nz) /T) /T)
where an indicator function, y denotes the ground-truth class and T a learnable temperature parameter. The overall loss of Contextual CNN is computed by:
L= (Cr + L2 Ls) L1.
where a a weight scalar. is empirically set as 0.15 for all experiments in this paper. More discussions about in the supplementary material.
3.2 CONTEXTUALIZINS LAYERS
Each contextualizing layer consists of two steps. The first step, dubbed classifying, reduces the number of considered classes from N; N where i is the index of the corresponding stage. The
4

---

Published as a conference paper at ICLR 2023
Features
Nettaen
Notlinear
AEL
(oedjac
PC
1x] Cityr
RALR RELL
LN AELU
Konhwddtageaf Closs
Eubcolinp
R
7a7-dCors
dxdl
Stepl: Clossifylgg
Stepl: Merging
Weight Officts aw Ior
Samplias Officts 20oxboo
Contextualizing
(b) Modulation
Figure 3; Illustration of (a) contextualizing and (b) modulation using contextual prior and features.
second step. dubbed merging, merges the embeddings N,+1 considered classes into a contextual prior for later contextual convolution blocks.
Classifying. As shown in Figure 3al given the N, considered classes, a set of class embeddings
& c) ee...... e collected from the aforementioned embedding layer. Following Radford et al. 22021) to compare with these embeddings, the visual features x from the proceding stage are projected to the embedding space. The projection involves a global average pooling and two fully-connected (FC) layers. Layer Normalization (LN) (Ba et al. 2016) and ReLU (Nair & Hinton 2010) are used between the FC layers. The output of the projection is a visual feature vector with dimension d. Then, cosine similarity is computed between the L2- n--normalized visual feature vector and the 1.2-normaiized embeddings of the N; classes. The resulting similarity vector 8 is used as the classification scores of the N, classes for loss calculation. The top-N highest scoring classes in s; are collected and propagated to the following merging step as well as the next stage.
Merging. Given No+1 output classes from the classifying step, merge their embeddings E1+1
e (o.. ... e++1 N.+1] ) into the contextual prior C E Rdx1, Specifically, the merging operation uses two fully connected layers (followed by LN and ReLU) and ID global average pooling layer between them. The merging layers different between stages. The generated context prior C summarizes the similarities and differences between the considered classes. It acts as task information for the extraction of more discriminative features in the following contextual convolution blocks.
3.3 CONTEXTUAL CONVOLUTION BLOCK
vanilla ConvNeXt block contains one 7 x 7 depthwise convolution and two x pointwise convolutions. A contextual convolution block in Contextual ConvNeXi-T's built by replacing the 7 x 7 depthwise convolution by a 7 x contextual convolution with other layers unchanged.
For vanilla 7 x depthwise convolution, consider the convolution kernel of size e 7 where e is the imput channel size") For each position on the output feature map y, the convolution first samples K 49 locations over the input feature map x using regular grid g, then sums all sampled values weighted by w:
K
((p)
Σ w(k) k=1
x(p + g(k)).
(3)
In contextual convolutions, we augment the kernel weight w with weight offsets Aw, and augment the grid g with sampling offsets Ap:
y(p) Σ (w(())+A((H) x(p + g(k) Ap(k)).
(4)
Weight offsets. The weight offsets Aw allow the adaptive adjustment of the convolution weights according to the contextual priors. As illustrated Figure 3b to obtain Aw, we squeeze the input
Depthwise convolutions operate on per-channel hasis and do change the channel size of features.
5

---

Published as a conference paper at ICLR 2023
map via global average pooling (GAP) and then concatenate the resulting feature vector with the contextual prior C. Two FC layers with LN and ReLU between them are applied afterwards to generate Aw. Notably, configure the size of Aw x c x 7, same as the dimensions of w, to allow the summation in equation
Sampling offsets. Following Dai al. (2017) and Zhu et al. (2019), the sampling offsets Ap applied to enable free-form deformation of the sampling grid. In our case, we compute the sampling offsets considering inspired by [iing et al. (2022), we first expand the contextual prior C to the same spatial shape of x. then concatenate them along the channel dimension. The resulting maps are then fed to nonlinear convolutional block consisting of one 1 x convolution (followed by LN and ReLU) and one 7 x 7 depthwise convolution (with the same kernel size and dilation as the replaced vanilla convolution). The output sampling offsets have the same resolution as the input features. The channel size 2K corresponds to K 2D offsets.
only the input features but also the contextual priors. As shown in Figure 3b)
To balance accuracy and efficiency, only one of every three blocks is replaced by contextual convolu- tion block Contextual ConvNeXt-T. More details are the supplementary material.
EXPERIMENTS
In the following, Contextual CNN is compared with the state the arts (SOTAs) on three tasks, i.e., image classification, video classification and instance segmentation. We then ablate important design elements and analyze internal properties the method via exemplification and visualizations.
4.1 IMAGE CLASSIHCATTINN ON IMAGENET-IK -K
Settings. For image classification, benchmark Contextual CNN on ImageNet-1K (Deng et al. 2009). It contains 1.28M training images and 50K validation images from 1, 000 classes. The top-1 accuracy single crop of size 224 224 is reported. To compare with SOTAs, instantiate Contextual CNN using the recent method ConvNeXt (Liu et al., 2022), dubbed Contextual ConvNeXt. Following Touvron et al. 20021: Liu al. (2021 2022). train the model for 300 epochs using an AdamW W optimizer (Loshchilov & Hutter 2017) with learning rate of 0.001. The batch size we use 096 and the weight decay 0.05. We adopt the same augmentation and regularization strategies as Liu et al. (2022) in training. Unless otherwise specified, the channel size of class embeddings d is 256. The numbers considered classes for the four stages are: N₂ 1000, N₂ = 500, N33 200 and N4 = 50. More details and discussions are in the supplementary material.
Table 1: Comparison with the state-of- of-tte-rrts on ImageNet- "FLOPs" denotes multiply-add operations. Following Liu et al. (2021), inference throughput is measured V100 GPU.
image size
throughput (images/s)
top-1
model
#param.
FLOPs
acc.
RegNetY-4G (Radosavovic al.
2020
224
21M
4.0G
1156.7
80.0
RegNer Y-&G (Radosavovic et al.
2020
2242
39M
8.0G
591.7
81.7
RegNetY-16G
Radosavovic al.
2020
2242
84M
16.0G
334.7
82.9
Swin-T Liu et al.
2021
224
28M
4.5G
757.9
81.3
Swin-S
Liu et
2021
224
50M
8.7G
436.7
83.0
Swin-B
Liu et al.
2021
224
88M
15.4G
286.6
83.5
ConvNeXt-T
(Lu al. 2022
2242
29M
4.5G
774.7
82.1
ConvNeXt-S
Liu et al.
2022
2242
50M
8.7G
447.1
83.1
ConvNeXt-B
Liu et al.
2022
2242
89M
15.4G
292.1
83.8
Contextual ConvNeXt-T (ours)
2242
32M
4.6G
770.6
83.9
Contextual ConvNeett-S (ours)
2242
53M
8.9G
445.2
84.6
Contextual ConyNeXt-B (ours)
2242
93M
15.8G
291.0
85.2

---

Published as a conference paper at ICLR 2023
Results. Table presents the ImageNet-1K results of various SOTA architectures, including Reg- Net (Radosavovic et al., 2020), Swin Transformer (Liu et al. 2021) and ConvNeXt (Liu et al.)
2022). Contextual ConvNeXt outperforms all these architectures with similar complexities, e.g.. 11.8%/41.5%% 1.44+14% .4% vs. ConwNeXt-T/S/BB The above results verify the effectiveness of Contex- tual CNN for large-scale image classification, showing the potential of contextualized represention learning. Inspired by Swin Transformer, to compare efficiency with hurdware-optimizze classic CNNs, we adopt an efficient batch computation approach for contextual convolutions (detailed in $3 of the supplementary material). Thus, in addition to the noticeably better performances, Contextual ConvNeXt also enjoys high inference throughput comparable ConyNeXt.
4.2 EMPIRICAL EVALUATION DOWNSTREAM TASKS
Table Kinetics-400 video action classifcation results using TSM (Lin al. reproduced results using the same training recipe.
2019).
denotes the
backbone
image size
#frames
FLOPs
top-1
top-5
ResNet50
224
16
65G
74.7
91.4
ResNer101
224
16
125G
75.9
92.1
Contextual ResNet50 (ours)
2242
16
68G
77.0
93.1
Table 3: COCO object detection and segmentation results using Mask-RCNN. Following Liu et al. (202T). FLOPs calculated with image size 1280 800.
backbone
FLOPs
Apboz
Aphor
Apboz
Apmask
Apmask
Apmask
Swin-T (Liu et al. 2021 267G
46.0
68.1
50.3
41.6
65.1
44.9
ConvNeXt-T (Lu et al. 2022
262G
46.2
67.9
50.8
41.7
65.0
44.9
Contextual NeX- (ours)
267G
47.3
69.0
52.2
42.7
66.2
45.6
Video classification on Kinetics-400. Kinetics-400 (Kay et al., 2017 is a large-scale video action classification dataset with 240K training and 20K validation videos from 400 classes. We fine-tune TSM (Lin et al., 2019) model with Contextual CNN based on ResNet50 (He et al., (016).. For a fair comparison, the model first pretrained ImageNet-1K following the original ResNet50, then fine-tuned on Kinetics-400 following TSM. More details are in the supplementary material.
Table lists the video action classification results of Contextual ResNet50 and ResNet50 / R1. It is observed that Contextual ResNet50 is +2.3%%++177 better on top- top-1/5 accuracy than ResNet50 with similar computation cost. Moreover, Contextual ResNet50 even better than the more so- phisticated backbone ResNet101. These results verify that the Contextual CNN architecture can be effectively extended general visual recognition tasks like video classification.
Instance segmentation on COCO. The instance segmentation experiments are conducted on COCO (Lin et al. 2014), which contains 118K training and 5K validation images. Following Swin Trans- former, we fine-tune Mask R-CNN (He et al. (017) on the COCO dataset with the aforementioned Contextual ConvNeXt backbones. The training details are deferred to the supplementary material.
Table 3 shows the instance segmentation results of Swin Transformer, ConvNeXt and Contextual ConvNeXt. With similar complexity, our method achieves better performance than ConvNeXt and Trin-Transformer in terms of both box and mask AP. This demonstrates that the superiority of Contextual CNN's s contextualized representations still hold for downstream dense vision tasks, indicating that Contextual CNN is able to serve general-purpoon backbone for computer vision.
4.3 ABLATION STUDY
We ablate major design elements Contextual CNN using ImageNet-1K 1K dataset. All ablation models based on ResNet50 (He et al., 2016). Details of the architecture are in the supplementary material.
Block design (Table We empirically analyze the effect the proposed components in our work: contextualizin layers and contextual convolutions. First, compared with vanilla ResNet50 (a/), sim-
7

---

Published as a conference paper at ICLR 2023
Table Ablations on the proposed components. "Ctx" denotes contextualizing layers. "CixConv" denotes contextual convolutions. "Deform" denotes deformable convolutions V2 (Zhu et al. 2019
model
#param.
FLOPs
top-1 acc.
al R50
25.5M
4.1G
76.58
Ctx a2
25.7M
4.1G
77.18
a3
CtxConv
27.2M
4.2G
77.59
a4
Deform
25.7M
4.2G
77.24
a5 Ctx CixConv
27.9M
4.3G
79.35
ply using contextual prior C as an extra input (via expanding and concatenating) to the convolutions (a2) shows slightly better result (.0.).)) This reveals that feeding the contextual prior without modulations provides limited gains for representation learning. Second, applying contextual convolu- tions alone (a3) leads to a + 1.01% gain on top-1 accuracy while applying deformable convolutions (Zuu et al. 2019) alone shows a +0.66% gain (a4). The results imply that modulating convolutions according only imput features improves the expressive power of learned representations. This spirit shared between contextual convolutions, deformable convolutions and other forms of dynamic convolutions. Third, combining the proposed components,
observe significant gain of +2.77% compared to the classic CNN (a5 vs. al) and a gain of +1.76%/+2.1 +161142.11% compared to the dynamic CNNs (a5 vs. a3la4). The results verify that the categorical context provides important cues about the more discriminative directions to modulate the convolutions, highlighting the advantage of using potential category memberships as contextual priors.
Table 5: Ablations on contextual modulations.
weight offset
sampling offset
#param.
FLOPs
top-1
bl
25.5M
4.1G
77.18
b2
27.5M
4.2G
78.30
b3
25.9M
4.3G
78.44
b4
27.9M
4.3G
79.35
Modulation design (Table We investigate how the two forms of contextual modulations in con- volutions affect the performance. Using weight/sampling offsets alone brings in +1.12%/41.26% improvement (b2/b3 vs. bl), which indicates that either form of the modulations leverages the categorical context some degree. Combining the two forms leads to more significant gain of +2.17% (b4 vs. bl). This implies that the two forms of modulations complete each other.
Table Ablations on contextualizing. d the dimension of class embeddings.
Stagel Stage2
Stage3 Stage4
#param.
FLOPs
top- I acc.
cl
256
25.5M
4.1G
76.58
c2
256
26.7M
4.1G
77.86
c3
256
27.7M
4.2G
78.63
04
256
27.9M
4.3G
79.35
c5
v
256
28.0M
4.3G
77.99
c6
64
27.3M
4.2G
77.59
c7
128
27.5M
4.3G
78.44
c8
512
28.3M
4.3G
78.89
Contextualizing design (T. We study the influence of two hyperparameters: the number of contextual stages and the dimension of class embeddings d. First, the model with 3 contextual stages ("Stage2-4") yields the best top-1 accuracy compared to those with more or less contextual stages (c3 cOlcllc2le4). This suggests that contextual modulation is more effective middle-level and high-level features. Then, by increasing the dimension of class embeddings d, the accuracy improves steadily and saturates at 256 (c4 c6lc7lc8). This reveals the strategy of setting d in practice.
8

---

Published as a conference paper at ICLR 2023
4.4
MORE ANALYSIS
image
ConvNeXL-TT jacy wal
image
-on-NeXI-T
OUTS
Junky (CT2:
clecticc fan 22%
52800 lester (GT), 00%
Iadky (01))
JESy wile
beake (GT) 19%6
clannz frn: 80%
Figure 4: Comparison Grad-CAM visualization results. The Grad-CAs Crad visualization is calculated for the last convolutional oulputs. "GT" denotes the ground truth class of the image.
Chapsi disoo Slopcc
W. 520
M,
(a) Selected classes from "Stage I" to "Stage4"
(b) Confusion matrix
Figure 5: Left: The t-SNE distributions of the class embeddings on ImageNet-IK. The points in blue denote the selected classes of the corresponding stage and the point in red denotes the groud-truth class. Right: Comparison of class embeddings between ConvNeXt-T and our Contextual ConvNext-1 on ImageNet-1K, K. For ConvNeXt-T, we normalize the weights of the classifier as its class embeddings (1000 768). The confusion matrix denotes the similarity matrix class embeddings (1000 1000).
Analysis of generated features. We adopt Grad-CAM (Selvaraju et al. 2017) to interpret the learned features by highlighting the regions that discriminate different classes (shown Figure 4 In the first case, the counterpart (ConyNeXt-T)) generates features w.r.t. the face of the animal, which shares similar appearances between "husky" and "wolf". Our model generates features of the ears that are long and stand upright, which are the key factor to differentiate "husky" from "wolf" (offset and triangular). In the second case, the counterpart generates features w.r.t. the base and the center hub of the object, both are shared between "space heater" and "electric fan". In contrast, our model generates features the fan region, which is very different between the two categories (filaments vs. vanes). In summary. the counterpart generates shared patterns of the most likely classes, which help select these classes out of 1,000 classes but fail to further differentiate them. This behavior is reasonable since the Contextual CNN, in contrast, takes few most likely
convolutions in the model category-agnoste classes as contextual priors and learns to generate more discriminative features w.r.t. these classes. Thus, our model superior in resolving the above challenging cases that confuse the counterpart.
Analysis of class embeddings. Figure 5a visualizes the stage- wise classifying of Contextual CNN. Specifically, first adopt I-SNE (Van der Maaten & Hinton 2008) visualize the class embeddings of the model. We then highlight the selected classes from "Stagel" to "Stage4". The results suggest that the contextual prior progressively converges semantic neighbors of the groundtruth class. Figure 5b compares the confusion matrix of class embeddings between ConvNext-TT and Contextual ConvNeXt-T. Following Chen al. (2019), we uses the weights of the last fully-connected layer its class embeddings. One observe that the class embeddings learned by of model exhibit more effective separation of category memberships.
CONCLUSION
This paper presents Contextual Convolutional Network, new CNN backbone which leverages a few most likely classes contextual information for representation learning. Contextual CNN surpasses the performance of the counterpart models on various vision tasks including image classification, video classification and object segmentation, revealing the potential of contextualized representation learning for computer vision. We hope that Contextual CNN's strong performance on various vision problems promote the research new paradigm of convolutional backbone architectures.

---

Published as a conference paper at ICLR 2023
A ETHICS STATEMENT
First, most modern visual models, including Swin Trasnformer, ConvNeXt and the proposed Con- textual CNN, perform best with their huge model variants well with large-scale training. The accuracy-driven practice consumes a great amount of energy and leads to an increase in carbon emissions. One important direction is encourage efficient methods in the field and introduce more appropriate metrics considering energy costs. Then, many of the large-scale datasets have shown biases in various ways, raising the concern of model fairness for real-world applications. While our method enjoys the merits of large-scale training, reasonable approach to data selection is needed to avoid the potential biases of visual inference.
B REPRODUCIBILITY STATEMENT
We make the following efforts ensure the reproducibility of this work. First, we present detailed architecture specifications of Contextual ConvNeXt and Contextual ResNet50 in $1 of the supple- mentary material. Second, we provide detailed experimental settings (c.g.. the training recipes) of Contextual CNN for all involved tasks in $2 of the supplementary material. Third, code of Contextual CNN is available at: /tttps://
com/ 1wwng4sx/////eee htw.aaaggg.//ooottttataal..co
C ACKNOWLEDGMENTS
This work was (partially) supported by the National Key R&D Program of China under Grant 2020AAA0103901.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 160.00667700450. 2016.
arXiv preprint
Bastien Boutonnet and Gary Lupyan. Words jump-start vision: A label advantage in recognitton... Journal of Neuroscience, 2015.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In CVPR, 2020.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang. and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR, 2019.
Jason A Cromer, Jefferson Roy, and Earl Miller. Representation of multiple, independent categories in the primate prefrontal cortex. Neuron, 2010.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Ionut Cosmin Duta, Mariana Iuliana Georgescu, and Radu Tudor Ionescu, Contextual convolutional neural networks. ICCV, 2021.
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, 2005.
Charles D Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuro- science, 2013.
Charles D Gilbert and Mariano Sigman. Brain states: top-down influences in sensory processing. Neuron, 2007.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04661,, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks, In CVPR, 2017.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classificatton. In ICLR, 2018
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1wXXvv:700.066000 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurIPS, 2012.
Tai Sing Lee and David Mumford. Hierarchical bayesian inference in the visual cortex. Journal of the Optical Society of America. A, Optics, image science, and vision, 2003.
Yunsheng Li, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Ye Yu, Lu Yuan, Zicheng Liu, Mei Chen, and Nuno Vasconcelos. Revisiting dynamic convolution via matrix decomposition. arXiv preprint 2IXx::2003.08566 2021.
Shuxian Liang, Xu Shen, Jianqiang Huang, and Xian-Sheng Hua. Delving into details: Synopsis-to- detail networks for video recognition. In ECCV, 2022.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrlll, and Saining Xie. convnet for the 2020s. In CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. urXiv:1711.055 2017.
arXiv preprint
Gary Lupyan and Emily Ward. Language can boost otherwise unseen objects into visual awareness. Proceedings of the National Academy of Sciences, 2013.
Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. In ECCV, 2020.
David Marwood and Shumeet Baluja. Contextual convolution blocks. In BMVC, 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICMI., 2010.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. In CVPR, 2020.
11

---

Published as a conference paper at ICLR 2023
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classica receptive-fieke effects. Nature neuroscience, 1999.
David W. Romero, Robert-Jan Bruintjes, Jakub M. Tomczak, Erik J. Bekkers, Mark Hoogendoorn, and Jan C. van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In ICLR, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In ICCV, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Hugo Touyron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 2008.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI, 2020.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrlll, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. ECCV, 2018.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative spatial convolutions. In CVPR, 2018a.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths residual networks. CVPR, 2018b.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James J DiCarlo. performaane-optiimin hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 2014.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In NeurIPS, 2019.
Jiwei Yang, Xu Shen, Xinmei Tian, Houqiang Li, Jianqiang Huang, and Xian-Sheng Hua. Local convolutional neural networks for person re-identification. In ACM MM, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. CVPR, 2019.
12


Reviews:
[{"review_id": "vExOSmtEcyl", "reviewer": "Reviewer_2om9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of computer vision with convolutional neural networks (CNNs). The authors propose to contextualize the CNN by incorporating category priors. This is achieved by adding two new features to the CNN architecture. \n\nSome convolutional layers are contextualized by deforming the convolution grid using predicted spatial offsets, and by dynamically changing the kernel weights using predicted weight offsets. The offsets are predicted from the visual features extracted at the previous layer and a context prior. \n\nThe introduced context prior is computed by a new context layer which performs two tasks. First, from the visual features are projected to the space of classes embeddings, and a cosine similarity is measured to retain the top-n classes, which will be used in the subsequent contextualized convolutions. Then, the embeddings for the retained classes are merged to compute the context prior. \n\nThrough experiments on ImageNet and two adaptation tasks, the author show that the proposed model outperforms existing methods with a limited increase of computational complexity on GPU. An ablation study shows the importance of the various features proposed.", "summary_of_the_review": "The idea of using some context to compute the offsets of a deformable convolution is interesting and could be seen as a form of attention. For some application, it might even be more interesting to be able to select the list of classes to give to the network.\n\nCombining this idea with the conditioning of a dynamic convolution is equally interesting.\n\nAdditionally to the GPU results, the efficiency on CPU would be nice to have, compared to an efficient convolution operation, as well as memory considerations, although this potential drawback is shared with deformable and dynamic convolutions.\n\nOverall, I found the paper easy to read, the analysis convincing and the idea simple and interesting, worth sharing with the community.", "strength_and_weaknesses": "Strenghs:\n  - The idea is simple and well-explained, and extends and combines nicely the notions of deformable convolutions and dynamic convolutions with context\n  - The experiments report better results with a limited impact on computational efficiency\n  - The method is simple enough and looks easy to integrate\n\nThe main weakness of the idea, compared to the conventional CNN backbones lies in the adaptation of such a model to different tasks. It might be useful to only adapt the top layer(s) for a new task. In the presented experiments, it looks like the whole network needs to be fine-tuned. One experiments is reported in which class embeddings are frozen. It would be interesting to also measure the accuracy after only fine-tuning the top layer.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kYXrRib5yS", "reviewer": "Reviewer_y6tk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new Convolutional Neural Network, Contextual Convolutional Network, for visual recognition. Most existing convolutional backbones follow the representation-toclassification paradigm, where input representations are first generated by category-agnostic convolutional operations and then fed into perceptual classifiers (e.g., classification and segmentation).\nTop-k likely classes are encoded as a contextual prior vector. Based on this vector and the previous features, convolution offsets and kernel weights are generated. The new convolutions can easily replace their plain counterparts in existing CNNs and be trained end-to-end without supervision.\n", "summary_of_the_review": "The paper presented a modified architecture for CNN and it can be improved by doing some changes on presentation and adding more explanation on architecture and experiments to make the work more mature!", "strength_and_weaknesses": "Strength \nmodified Convolutional Neural Network, named Contextual Convolutional Network\ndeviate from this classic paradigm and propose to augment potential category memberships as contextual priors in the convolution for contextualized representation learning\ntrained end-to-end by standard back-propagation without additional supervision\n\nWeaknesses\nThe method is not very novel and can not be regarded as a new CNN, \nIt is a modification of a current method that provides better results in same cases\nthe method is not well presented and it can be included more details about architecture\nMore explanations on the experiments and the discussion on results\nNo discussion about computation and the architecture performance\nNo comparison with other CNN(classic and dynamic as their definition in paper) for performance\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g4gzlRoHDT", "reviewer": "Reviewer_CrqV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a neural network called contextual convolutional network, which adds classification layers at multiple stages and uses the top (most probable) class embeddings in following stages. The proposed layers / blocks are added to ConvNeXt models and tested on several mainstream / large-scale datasets (ImageNet, MS COCO, Kinetics-400).", "summary_of_the_review": "Currently, I believe the weaknesses slightly outweigh the strengths.", "strength_and_weaknesses": "Strength:\n- The paper is fairly easy to follow.\n- The experiments are conducted on representative datasets.\n\nWeaknesses:\n- The authors seemed to have missed some competing works bearing almost identical names [A, B] based on the idea of leveraging the context during convolution. The authors should point the differences / novelty with respect to these existing works. To me, it seems that the proposed convolution can be seen as a generalization of the convolution proposed in [A]. I also believe a head-to-head comparison with [A, B] is in order. At the moment, I have doubts about the novelty of the method.\n- An analysis of the failure cases is missing. The authors should also point out when the proposed method is expected to fail.\n- Are the performance improvements statistically significant? Are the improvements consistent over multiple runs? Such questions need to be answered to validate the shown improvements.\n\nMinor issues / language corrections:\n- The citation for SEnet is missing.\n- “have a output” => “have an output”.\n\n[A] Duta et al., \"Contextual Convolutional Neural Networks\". In: ICCVW, 2021.\n[B] Marwood et al., \"Contextual Convolution Blocks\". In: BMVC, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3nse2oL94rA", "reviewer": "Reviewer_oBzi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to increasingly refine at each CNN stage the knowledge about the class of a given image and use this information to build a more discriminative model. The more discriminative model is build with two contributions: i) add variations to the static CNN filters that are classes specific (based on the set of classes that are selected so far) and on the sampling location similar to deformable CNN. Both contributions seem important to boost classification results on imageNet and also other tasks,", "summary_of_the_review": "I consider this paper a solid and balanced contribution in the field of convolutional neural networks.\nIt has:\n- A promising contribution\n- Goof and clear presentation\n- Complete related work (in my knowledge)\n- Excellent results & ablation studies", "strength_and_weaknesses": "\\+ The paper is well written and clear to understand\n\\+ Related work is complete\n\\+ Experimental results are thorough with the important ablation studies. \n\n\\- The convolution offsets is the same as deformable convolutions (Dai et al. 2017). How does the proposed method compares with Deformable convolutions?\n\\- The proposed approach has as hyperparameters the number of classes to consider on each layer. Did the authors think about considering it as a single scaling factor instead of an independent value for each stage?\n\\- In my understanding, in the contribution and title I would put more emphasis on the fact that the following stage of convolution depend on the results of the previous, thus rendering convolutions dynamic in a smart way.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster