MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
Authors: Duy Kien Nguyen
Vedanuj Goswami
Xinlei Chen
Venue: ICLR 2021 Poster (2021)
Keywords: visual counting
visual question answering
common object counting
visual reasoning
modulated convolution

Abstract: This paper focuses on visual counting, which aims to predict the number of occurrences given a natural image and a query (e.g. a question or a category). Unlike most prior works that use explicit, symbolic models which can be computationally expensive and limited in generalization, we propose a simple and effective alternative by revisiting modulated convolutions that fuse the query and the image locally. Following the design of residual bottleneck, we call our method MoVie, short for Modulated conVolutional bottlenecks. Notably, MoVie reasons implicitly and holistically and only needs a single forward-pass during inference. Nevertheless, MoVie showcases strong performance for counting: 1) advancing the state-of-the-art on counting-specific VQA tasks while being more efficient; 2) outperforming prior-art on difficult benchmarks like COCO for common object counting; 3) helped us secure the first place of 2020 VQA challenge when integrated as a module for ‘number’ related questions in generic VQA models. Finally, we show evidence that modulated convolutions such as MoVie can serve as a general mechanism for reasoning tasks beyond counting.

Published as a conference paper at ICLR 2021
MOVIE: REVISITING MODULATED CONVOLUTIONS FOR VISUAL COUNTING AND BEYOND
Duy-Kien Nguyen, Vedanuj Goswami, Xinlei Chen Facehook Al Research (FAIR)
ABSTRACT
This paper focuses visual counting, which aims to predict the number of oc- currences given a natural image and query (e.g. a question or a category). Un- like most prior works that use explicit, symbolic models which can be computa- tionally expensive and limited in generalization, we propose simple and effec- tive alternative by revisiting modulated convolutions that fuse the query and the image locally. Following the design of residual bottleneck, we call our method MoVie, short for Modulated con Volutional bottlenecks. Notably, MoVie reasons implicitly and holistically and only needs a single forward-pass, during inference. Nevertheless, MoVie showcases strong performance for counting: I) advancing the state-of-the-art on counting-specific VQA tasks while being more efficient; 2) outperforming prior-art on difficult benchmarks like COCO for common ob- ject counting; 3) helped us secure the first place of 2020 VQA challenge when integrated as a module for 'number' related questions in generic VQA models. Finally, show evidence that modulated convolutions such MoVie can serve as a general mechanism for reasoning tasks beyond counting.
INTRODUCTION
We focus on visual counting: given a natural image and query, it aims to predict the correct number of occurrences in the image corresponding to that query. The query is generic, which can be a natural language question (e.g. 'how many kids are on the sofa') or a category name (e.g. 'car'). Since visual counting requires open-ended query grounding and multiple steps of visual reasoning (Zhang et al., 2018), is a unique testhed to evaluate machine's 's ability to understand multi-modal data.
Mimicking how humans count, most existing counting modules (Trott et al., 2018) adopt an intuition-driven reasoning procedure, which performs counting iteratively by mapping candidate image regions to symbols and count them explicitly based on relationships (Fig. 1, top-left). While interpretable, modeling regions and relations repeatedly can be expensive in computation (Jiang et al., 2020). And more importantly, counting is merely single visual reasoning task- if we sider the full spectrum of reasoning tasks (e.g. logical inference, spatial configuration). it is probably infeasible to manually design specialized modules for every of them (Fig. 1, bottom-left).
In this paper, we aim to establish a simple and effective alternative for visual counting without explicit, symbolic reasoning. Our work is built on two research frontiers. First, on the synthetic CLEVR dataset (Johnson et al., 2017), it was shown that using queries to directly modulate convolu- tions can lead to major improvements in the reasoning power of Convolutional Network (ConyNet) (e.g. achieving near- seareperfect 94% on counting) (Perez et al., 2018). However, it was difficult to trans- fer this finding to natural images. partially due the dominance of bottom up attention features that represent images with regions (Anderson et al., 2018). 8). Interestingly, recent analysis discovered that plain convolutional features can be as powerful as region features (Jiang al., 2020), which becomes second step-stone for our approach to compare fairly against region-based counting modules.
Motivated by fusing multi-modalities locally for counting, the central idea behind our approach is to revisit convolutions modulated by query representations. Following ResNet (He et al., 2016), we choose bottleneck as our basic building block, with each bottleneck being modulated once. Multiple bottlenecks are stacked together to forn our final module. Therefore, we call method Mo Modulated con Volutional bottlenecks. Inference for MoVie is performed by a simple, feed-forward pass holistically the feature map, and reasoning is done implicitly (Fig. 1, top-right).

---

Published as a conference paper at ICLR 2021
Question 1: [ccouninaa how thinos there?
Question 1: (ccntiing)
how many thinos ane there?
spplet symbert courter predku wcrk)
Image
right métioos
Mtep
3kop
Image
conAtiLtionl fumun
Yonaler?
tandder?
Question 2: (spatial reazoninal what the animal the front doing?
Question 2: (spatia) reasening) what is the animall the front doinn??
estlig
We study visual counting. Different from previous works that perform explicit, symbolic counting (left), we propose an implicit, holistic counter, MoVie, that directly modulates convolutions (right) and can outperform state-of-the-ar methods on multiple benchmarks. Its simple design also allows potential generalization beyond counting to other visual reasoning tasks (bottom).
Figure I:
MoVie demonstrates strong performance. First, improves the state-of-the-art on several VQA- based counting benchmarks (HowMany-QU (Trott et al., 2018) and TallyQA (Acharya et al., 2019)), while being more efficient. also works well on counting common objects, significantly outper- forming all previous approaches on challenging datasets like COCO (Lin et al., 2014), Furthermore, we show Vie can easily plugged into generic VQA models and improve the 'number' category on VQA 2.0 (Goyal et al., 2017)- and with the help of Vie, won the first place of 2020 VQA challenge, achieving 76.36% overall accuracy on the VQA v2.0 test-challenge server. To better understand this implicit model, we present detailed ablative analysis and visualizations of MoVie, and notably find improves upon its predecessor Fil.M (Perez et al., 2018) across all the counting benchmarks we experimented, with a similar computation cost.
Finally, we validate the feasibility of MoVie for reasoning tasks beyond counting (Fig. 1, bottom- right) by its near-perfect accuracy on CLEVR and competitive results on GQA (Hudson & Manning, 2019a). These evidences suggest that modulated convolutions such as MoVie can potentially serve as a general mechanism for visual reasoning. Code will be made available.
RELATED WORK
Here discuss related works to the counting module and works related to the task.
Explicit counting/reasoning modules. (Trott al., 2018) was among the first to treat counting differently from other types of questions, and cast the task as a sequential decision making problem optimized by reinforcement learning. A similar argument for distinction was presented in (Zhang et al., 2018), which took a step further by showing their fully-differentiable method can be attached generic VQA models as a module. However, the idea of modular design for VQA was not new notably, several seminal works (Andreas et al., 2016; Hu et al., 2017) have described learnable procedures to construct networks for visual reasoning, with reusable modules optimized for partic- ular capabilities (e.g. count, compare). Our work differs from such works in philosophy, they put more emphasis (and likely bias) on interpretation whereas we seek data-driven, general-purpose components for visual reasoning.
Implicit reasoning modules. Besides modulated convolutions (Perez et al., 2018; De Vries et al., 2017), another notable work is Relation Network (Santoro et al., 2017), which learns to repre- sent pair-wise relationships between features from different locations through simple MLPs, and showcases humpr- performance on CLEVR. The counter from TallyQA (Acharya et al., 2019) followed this idea and built two such networks one among foreground regions, and one between foreground and background. However, their counter is still based on regions, and neither general- ization VQA module nor other counting/reasoning tasks is shown.
Because existing VQA benchmarks like VQA 2.0 also include counting questions, generic VQA models (Fukui al,, 2016; Yu et al., 2019) without explicit counters also fall within the scope of
hhttps:/vvisn sual qa. org/ roe. html
2

---

Published as a conference paper at ICLR 2021
'implicit' ones when it comes to counting. However, key distinction of MoVie is that we fuse multi-modal information locally, which will be covered in detail in Sec. 3.1 next.
Specialized counting. Counting for specialized objects has a number of practical applica- tions (Marsden et al., 2018), including but are not limited to cell counting (Xie al., 2018), crowd counting (Sindagi & Patel, 2017), vehicle counting (Onoro-Rubio & López-Sastre, 2016), wild-life counting (Arteta et al., 2016), etc. While less general, they are important computer vision appli- cations to respective domains, e.g. medical and surveillance. Standard convolution filters are used extensively in tftt-offthhe-ar models (Cheng et al., 2019) to produce density maps that approximate the target count number in a local neighborhood. However, such models are designed to deal ex- clusively with a single category of interest, and usually require point-level supervision (Sindagi & Patel, 2017) in addition to the ground-truth overall count number for training.
Another line of work on specialized counting is psychology syychollgy-inspired (Cutini & Bonato, 2012), which focuses the phenomenon coined 'subitizing' (Kaufman et al., 1949), that humans and animals can immediately tell the number of salient objects in a scene using holistic cues (Zhang et al., 2015). It is specialized because the number of objects is usually limited to be small (e.g. up to 4 (Zhang al., 2015)).
General visual counting. Lifting the restrictions of counting one category or a few items at a time, more general task settings for counting have been introduced. Generalizing to multiple semantic classes and more instances, common object counting (Chattopadhyay et al., 2017) as a task has been explored with a variety of strategies such as detection (Ren et al., 2015), ensembling (Galton, 1907), or segmentation (Laradji et al., 2018). The most recent development in this direction (Cholakkal et al., 2019) also adopts a density map based approach, achieving state-of-the-art with weak, image- level supervision alone. Even more general is the setting of open-ended counting, where the counting target is expressed in natural language questions (Acharya et al., 2019). This allows more advanced "reasoning" task to be formulated involving objects, attributes, relationships, and more. Our module is designed for these general counting tasks, with the modulation coming from either a question or class embedding.
3 COUNTING WITH MODULATIONS
3.1 MODULATED CONVNET
Motivation. Why choosing modulated convolutions for counting? Apart from empirical evidence on synthetic dataset (Perez et al., 2018), we believe the fundamental motivation lies in the convo- lution operation itself. Since ConvNets operate on feature maps with spatial dimensions (height and width), the extra modulation - in our case the query representation is expected to be applied densely to all locations of the map in fully-convolutional manner. This likely suits visual count- ing well for at least two reasons. First, counting (like object detection) is a translation -esssaationequuiariann problem: given fixed-sized local window, the outcome changes as the input location changes. Therefore, a local fusion scheme like modulated convolutions is more preferred compared to ex- isting fusion schemes (Fukui et al., 2016; Yu et al., 2017), which typically applied after visual features are pooled into a single global vector. Second, counting requires exhaustive search over all possible locations, which puts the dominating bottom-up attention features (Anderson et al., 2018) that sparsely sample image regions at a disadvantage in recall, compared to convolutional features that output responses for each and every location.
Pipeline and module. In Fig. 2 (a) we show the overall pipeline. The output convolutional features from standard ConvNet (e.g. ResNet) are fed into the MoVie module at the top. The module consists of four modulated convolutional bottlenecks (Fig. 2 (b)). Each bottleneck receives the query extra input to modulate the feature map and outputs another sized feature map. The final output after several stages of local fusion is average pooled and fed into a two-layer MIP classifier
In fact, the module can be placed in earlier stages, or even be split across different stages. However, we didn't t find it help much but incurs extra computation overhead that computes convolutional features twice if used module for generic VQA models. That's why we stick the top.
3

---

Published as a conference paper at ICLR 2021
snaver
D element-wee additon
M
element-viae mutiliaaioon
module
bottlensck
188-4
Picghia
Lollkneck--
3
mes
cottlnnck-22
1xT
COTV's e pool
botteneckk
modi
vn8ge
(a) pipeline
(b) modulc
(c) bottleneck
(d) modulation
Figure 2: Overview MoVie which can be placed on top of any convolutional features. The mod-
ule consists of several modulated convolutional bottlenecks. Each bottleneck is simple modified version of the residual bottleneck, with an additional modulation block before the first convolution. The output of the module is average pooled and fed into a two-layer MLP for the final answer.
(with ReLUU to predict the answer. Note that we do not apply fusion between query and the global pooled vector: all the interactions between query and image occur in modulated bottlenecks locally.
Bottleneck. As depicted in Fig. 2 (c), MoVie closely follows the original ResNet bottleneck design, which is both lightweight and effective for learning visual representations (He al., 2016). The only change is that: before the first 1 x convolution layer, we insert modulation block that takes the query representation as a side information to modulate feature maps, which we detail next.
Modulation. We start with Feature-wise Linear Modulation (FiLM) (Perez al., 2018) and intro- duce notations. Since the modulation operation is the same across all feature vectors, we simplify by just focusing on one single vector VERC on the feature map, where the channel size. FiLM modulates with linear transformations per channel and the output VELM is:
(1) where & is element-wise multiplication, @ is element-wise addition (same normal vector ad- dition). Intuitively, 7ER scales feature vector and BERC does shifting. Both y and B are conditioned on the query representation Q through FC weights (W,, WaJeR
VFLM X 7) + 8,
One crucial detail that stabilizes FiLM training is to predict the difference Ay rather than y itself, where 1014 and is all-one vector. This essentially creates residual connection,
VILM = [y @ (1 e Ay)] @
@ [(v & Ay) + 8]
We can then view FFLM (v ) as a residual function for modulation, conditioned jointly This perspective creates opportunities for explore
and and will be added back other, potentially better forms of F(v, q) for counting.
The modulation block for MoVie is shown in Fig. 2 (d). The modulation function is defined FMOvie & (V(vvA)) where WERCxC is learnable weight matrix that be easily converted to a 1x1 convolution in the network. Intuitively, instead of using the output of vœ2r directly, this weight matrix learns output C' inner products between and Ay weighted individually by each column in W. Such increased richness allows the model to potentially capture more intricate relationships. Our final formulation for MoVie is:
MoVie ① W & A7).
Note that we also removed / and thus saved the need for W Compared to FILM, the parameter count of Mo Vie can be fewer, depending the relative size of channel C and query dimension D.

---

Published as a conference paper at ICLR 2021
Scale robustness. One concern using convolutional feature maps directly for counting is on its sensitiveness to imput image scales, as ConvNets are fundamentally not vvale-ivvrriant. Region- based counting models (Trott et al., 2018; Zhang et al., 2018) tend to be more robust to scale changes, as their features are computed on fixed-shaped (e.g- 7x7) features regardless of the sizes of the corresponding bounding boxes in pixels (Ren et al., 2015). We find two implementation details helpful to remedy this issue. First is to always keep the input size fixed. Given an image, we resize and pad it global maximum size, rather than maximum size within the batch (which can vary based on aspect-ratios of sampled images). Second, we employ multi-scale training, by uniformly sampling the target image size from pre-defined set (e.g. shorter size 800 pixels). Note that input size and image size can be decoupled, and the gap is filled by zero-padding.
Query representation. In our experiments, query representations qEIP have two types: ques- tions (Trott et al., 2018) and categories (Chattopadhyay et al., 2017).
For questions, use LSTM and Self Attention (SA) layers (Vaswani et al., 2017) for question en- coding (Yu et al., 2019). Specifically, a question (or sentence in NLP) consisting of N words is first converted into sequence (PP=.. 2 of N 300-dim Glo word embeddings (Pennington et al., 2014), which are then fed into one-directional LSTM followed by a stacked of L=4 layers of self attention:
Q LSTM(QQ),
(3)
Q' SA
(4)
1E (2...., L + 1} is the D=512 dimensional embedding for each word where Q'=(wq.... ...., w in the question after the (1-1)-th SA layer.
Given the final Q1+,1 to get the conditional vector q. we resort to summary attention mecha- nism (Nguyen & Okatani, 2018). A two-layer 512-dim MLP with ReLU con-linearity is applied to compute an attention score Sn for each word representation "N I++1 We normalize all scores by soft- max to derive attention weights on and then compute an aggregated representation q via a weighted summation over Q QL+11
For categories, we use an embedding matrix C. = (ci,...,Cy of N category embeddings where On E IR P. The query representation q of n-th category: L2-Norm(c,).
3.2 MOVIE AS COUNTING MODULE
An important property of a question- puaste visual counter is to see if it can be integrated as module for generic VQA models (Zhang et al., 2018). At a high-level, state-of-the-ar VQA models (Jiang et al., 20 Yu et al., 2019) follow a common paradigm: Obtain an image representation iERO" and question representation R.,,, then apply fusion scheme (e.g. (Fukui et al., 2016)) to produce the final answer a. A VQA loss L against ground truth à is computed to train the model (Fig. (a)).
While a naïve approach that directly appends pooled features veR" from MoVie to (Fig. 3 (b)) likely suffers from feature co-adaptation (Hinton et al., 2012), we follow a three-branch training scheme (Wang et al., 2019; Qi et al., 2020) by adding two auxiliary, training-only branches: One trains the original VQA model just with i, and the other trains a normal Mo with v and MLP. All three branches are assigned equal loss weight. The fusion parameters for i and the joint branch (i+v) are not shared. This setup forces the network to learn powerful representations within and v. as they have to separately minimize the VQA loss. During testing, we only use the joint branch, leading to significant improvements especially on 'number' related questions for VQA without sac- rificing inference speed (Fig. 3 (c)).
EXPERIMENTS
We conduct a series of experiments to validate the effectiveness of MoVie. By default, we use Adam (Kingma& & Ba, 2015) optimizer, with batch size 128 and base learning rate le 4, momentum 0.9 and 0.98. We start training by linearly warming up learning rate from 2.5e for 3 epochs (Yu al., 2019). The rate is decayed by 0. x after 10 epochs and we finish training after 13 epochs.
5

---

Published as a conference paper at ICLR 2021
L. a)
(. a)
L(a a)
L(a,.
luse
fuse
4
fuse
fuse
MLP
+
i+
(a)
(b)
(c)
Figure 3: MoVie counting module for VQA. (a) A high-level overview of the current VQA systems, image and question are fused to predict the answer a. (b) A naive approach to include match dimensions) to i. (c) Our final design to train with two auxiliary losses on and v, while during testing only using the joint branch ay. Shaded areas are used for both train and test; white areas are train-only.
MoVie counting module: directly add pooled features (with one
Design
B
+
ACC f
RMSE
# bottlenecks
ACC f
RMSE
VEIM
/
/
57.1
2.67
1
57.2
2.65
MoVie
x
58.4
2.60
2
58.0
2.63
/
58.2
2,63
3 58.2
2.62
VMoVi variants
x
X
57.9
2.64
4 58.4
2.60
58.5
2.63
5
58.5
2.63
(a) MoVie vs. FiLM. 6: residual connection.
(b) Number of modulated bottlenecks.
Fixed input size
Test size
ACC +
RMSE +
Train size
Test size
ACC f
RMSE $
400
22.1
4.70
400
53.9
2.92
x
600
36.0
3.16
800
600
57.3
2.69
800
56.2
2.68
800
58.4
2.60
400
53.9
2.92
400
56.5
2.78
/
600
57.3
2.69
(400,600,800)
600
58.8
2.66
800
58.4
2.60
800
58.8
2.59
(c) Fixed vs. batch-dependent input size.
(d) Multi-scale single-scale training.
Table 1:
Ablative analysis on HowMany QA val set. MoVie modulation design outperforms FiLM under fair comparisons; both fixing input size and multi-scale training improve robustness to scales.
4.1 VISUAL COUNTING
For visual counting, we have two tasks: open-ended counting with question queries where we ab- lated our design choices, and counting common object with class queries.
Open-ended counting benchmarks. Two datasets are used to counting with question queries, First HowMany-QA (Trott et al., 2018) where the train set questions extracted from VQA 2.0 train and Visual Genome (VG) (Krishna et al., 2017). The val and test sets are taken from VQA 2.0 val set. Each ground-truth answer
number between 0 20. Extending HowMany-QQ., the TallyQA (Acharya et al., 2019) dataset augments the train set by adding synthetic counting questions automatically generated from COCO annotations. They also split the test set into two parts: test- simple and rest-complex, based on whether the question requires advanced reasoning capability. The answers range between and 15. For both datasets, accuracy (ACC, higher-better) and standard RMSE (lower-better) (Acharya et al., 2019) are metrics used for evaluation.
We first conduct ablative analysis on How QA for important design choices in MoVie. Here, use fixed ResNet-50 features pre-trained VG (Jiang et al., 2020), and train MoVie on HowMany--AA train, evaluate on val. The results are summarized in Tab.
Modulation design. In Tab. la. compare MoVie design with FiLM (Perez et al., 2018): All other settings (e.g. the number of bottlenecks) are fixed and only V is replaced with VELM in module (see Eq. and 2). We also experimented other variants of Vavie by switching on/off

---

Published as a conference paper at ICLR 2021
#params (M)
FLOPs (G)
MowMany QA
BallyQA-Simphi
TallyQA-Comp
Method
Backbone
ACC
RMSE +
ACC 1 RMSF +
ACC RMSE +
Count module (2018)
R-101
44.6
54.7
2.59
70.5
1.15
50.9
1.58
IRLC (2018)
R-101
44.6
56.1
2.45
TallyQA (2019)
R-101+152
104.8
1883.5
60.3
2.35
71.8
1.13
56.2
1.43
TallyQA (FG-Only)
R-101
44.6
1790.9
69.4
1.18
51.8
1.50
Moeie
R-50
25.6
176.1
61.2
2.36
70.8
1.09
54.1
1.52
MoVie
R-101
44.6
306.9
62.3
2.30
73.3
1.04
56.1
1.43
MoVie
X-101
88.8
706.3
64.0
2.30
74.9
1.00
56.8
1.43
Table 2:
Open-ended counting Howmany-QA and TallyQA test set. MoVie outperforms prior arts with lower #parameters and FLOPs. R: ResNet (He et al., 2016); X: ResNeXt (Xie al., 2017).
and the residual connection (+). The results indicate that: 1) The added linear mapping (W) is helpful for counting - all MoVie variants outperform original FiLM; 2) the residual connection also plays role and benefits accuracy; 3) With W and @D, 3 is less essential for counting performance.
Number of bottlenecks. We then varied the number of modulated bottlenecks, with results shown in Tab 1b. We find the performance saturates around 4, but stacking multiple bottlenecks useful compared to using single one. We observe the same trend with FiLM but with weaker performance.
Scale robustness. The last two tables Ic and 1d ablate our strategy to deal with input scale changes. In Tab. Ic we set single scale (800 pixels shorter side) for training, and vary the scales during test- ing. If we only pad images to the largest size within the batch, scale mismatch drastically degenerate the performance (top rows). By padding the image to the maximum fixed input size, substantially improves the robustness to scales (bottom 3 rows). Tab. 1d additionally adds multi-scale training, showing it further helps scale robustness. We use both strategies for the rest of the paper.
Open-ended counting results. We report test set on counting questions for both How -AA and TallyQA in Tab. Even with ResNet-50, MoVic already achieves strong results, e.g. outperform- ing previous work on HowMany- -A in accuracy and TallyQA (FG-Only). With a ResNeXt-101 backbone (Xie et al., 2017), we can surpass all the previous models by a large margin e.g. 44% in absolute accuracy Howwanny-QA The same also holds on TallyQA, where perform better for both simple and complex questions. Note we also have lower parameter counts and FLOPs
Visualization. We visualize the activation maps produced by the last modulated bottleneck for sev- eral complex questions in TAllyQA (Fig. 4). Specifically, we compute the normalized fg-norm map per-location on the feature map (Malinowski et al., 2018). The attention on the question is also vi- sualized by using attention weights (Nguyen & Okatani, 2018) from the question encoder. First two rows give successful examples. On the left, MoVie able to focus on the relevant portions of the image produce the correct count number, and can extract key words from the question. We further modified the questions to be more general (right), and even with a larger number to count, MoVie Vie can give the right answers. Four failure cases are shown in the last two rows, where the model either fails to predict the correct answer, or produces wrong attention maps to begin with.
Common object counting. Second, we apply Mo to common object counting, where the query is an object category. Following standard practice, we choose ResNet-50 pre-trained on Ima- geNet (Deng al., 2009) as backbone which is also fine-tuned. Due to the skewed distribution between zero and non-zero answers, we perform balanced sampling during training.
Results on COCO (Lin et al., 2014) are summarized Tab. 3, where we compared against not only the state-of the- counting approach CountSeg (Cholakkal et al., 2019), but also the latest improved version of Faster R-CNN, In addition to RMSE, three variants are introduced in (Chattopadhyay et al., 2017) with different focuses. Mo Vie outperforms all the methods on three metrics and comes as a close second for the remaining one without using any instance-level supervision. For more details and results Pascal VOC (Everingham al., 2015), appendix.

---

Published as a conference paper at ICLR 2021
How many red trains are in the picture
How many trains are in the picture
Pred: 1, Ans: 1
Pred: 2, Ans: 2
How many animals are laying down
How many animals are there
Pred: 1. Ans:
Pred: 8, Ans: 8
raiO
ZEESS
How many of the birds are the
How many road sign poles are there Pred: 2, Ans: 3
Pred: 2. Ans: 1
How many of the pillows are gray
How many seals are on the bench
Pred: 2, Ans: 0
Pred: 0, Ans:
Figure 4: Visualizations of attention maps on images and questions for several complex examples in TallyQA. First rows show successful cases, and last two shows four failure ones. Best viewed in color on a computer screen. See text for detailed explanations.
Instance supervision
Method
RMSE
RMSE-nz 4
reI-RMSE $
rel-RMSE-nz
L.CResFCN (2018)
0.38
2.20
0.19
0.99
glance-noft-21. (2017)
X
0.42
2.25
0.23
0.91
CountSeg (2019)
X
0.34
1.89
0.18
0.84
Faster R-CNN (2019)
V
0.35
1.88
0.80
MoVie
0.30
1.49
0.19
Table Common object counting on COCO test with various RMSE metrics. MoVie outperforms prior state-of-the-arts without instance-level supervision. All models use ResNet-50 backbone pre- trained on ImageNet.
4.2 VISUAL QUESTION ANSWERING
Next, we explore MoVie as counting module for generic VQA models. We choose MCAN (Yu et al., 2019), the 2019 VQA challenge winner, target model (see appendix for Pythia (Jiang et al., 2018), the 2018 winner). For fair comparisons, we only use single-scale training, and adopt the learning rate schedule from MCAN to train MoVie using fixed ResNet-50 features.
The top section of Tab. shows our analysis of different designs to incorporate Mo Vie into MCAN- Small (Sec. 3.2). We train all models VQA 2.0 train and report the breakdown scores val. Trained individually, MoVie outperforms MCAN in 'number' questions by decent margin, but lags behind in other questions. Directly adding features from MoVie (naïve fusion) shows limited im-
8

---

Published as a conference paper at ICLR 2021
Method
Test set
Yes/No
Number
Other t
Overall
MoVie
82.48
49.26
54.77
64.46
MCAN-Sma (2019)
83.59
46,71
57.34
65.81
val
MCAN-Sman MoVic (naive fusion)
83.25
49.36
57.18
65.95
MCAN-Smale MoVie
84,01
50,45
57.87
66.72
MCAN- Large X-101 (2020)
88.46
55.68
62.85
72.59
test-dev
MCAN-Large X-101 + MoVie
88.39
57.05
63.28
72.91
Table 4: Top:
Different ways to use MoVie as a counting module for VQA models measured by VQA score (Antol et al., 2015). Bottom: MoVie especially helps 'number' questions on test-dev.
Method
BottomUp (2018)
MAC (2018)
NSM* (2019b)
LCGN (2019)
Mo
Humans
Overall f
49.7
54.1
63.2
56.1
57.1
89.3
Binary
66.6
71.2
78.9
73.5
91.2
Open f
34.8
38.9
49.3
42.7
87.4
Table Reasoning on GQA test set show the generalization of Mo beyond counting (*: uses scene-graph supervision (Krishna et al., 2017)) Metrics follow (Hudson & Manning, 2019a).
provement, while our final three-branch fusion scheme is much more effective: increasing accuracy on all types with a strong emphasis on 'number' while keeping added cost minimum (see appendix).
In the bottom section of Tab. 4, we further verified the effectiveness of Mo Vie and the fusion scheme on the test-dev split of VQA 2.0, switching the base VQA model to MCAN-Large, the backbone ResNeXt- 101, and using more data (VQA 2.0 train+val and VG) for training. We find Mo Vie consistently strengthens the counting ability. Thanks to MoVie, our entry won the first place of 2020 VQA challenge, achieving a test-sid score of 76.29 and test-chullenge 76.36. The improvement over Tab. 4 mainly comes from better grid features and model ensembling..
4.3 BEYOND COUNTING
Finally, to explore the capability of our model beyond counting, we evaluate Mo on the CLEVR dataset (Johnson et al., 2017). We train our model with a ImageNet pre-trained ResNet-101 for 45 epochs, and report a (near-perfect) test set accuracy of 97.42% similar to the observation made in FiL.. This suggests is the general idea of modulated convolutions that helped achieving strong performance on CLEVR, rather than specific forms presented in (Perez et al., 2018).
Since CLEVR synthetic, we also initiate an exploration of MoVie on the recent natural-image reasoning dataset, GQA (Hudson & Manning, 2019a). We use ResNeXt-101 from VG and report competitive results Tab. without using extra supervisions like scene-graph (Krishna al., 2017). Despite simpler architecture compared to models like MAC (Hudson & Manning, 2018) 8), we demon- strate better overall accuracy with larger gain on open questions.
5 CONCLUSION
In this paper, we propose a simple and effective model named MoVie for visual counting by revisit- ing modulated convolutions that fuse queries images locally. Different from previous works that perform explicit, symbolic reasoning, counting is done implicitly and holistically in MoVie and only needs single forward- pass during inference. We significantly outperform state-of-the of-the-arts on three major benchmarks in visual counting, namely HowMany MoVie can easily incorporated as a module for general VQA models like MCAN to improve accuracy on 'number' related questions on VQA 2.0. The strong performance helped us secure the first place in the 2020 VQA challenge. Finally, we show Mo Vie can be directly extended to perform well on datasets like CLEVR and GQA, suggesting modulated convolutions general mechanism can be useful for other reasoning tasks beyond counting.
Tally-QA and COCO; and show that
Phtpps: / /r ve. google.cooooooooo.oooooooo com/ foom/fi11 file/d/1 190R6xHQ7A1 ______ we_.2y91moooo.. moRo4ufqqu F5n /vi vilee ex provides additional details.

---

Published as a conference paper at ICLR 2021
REFERENCES
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyga: questions. In AAAI, 2019.
Answering complex counting
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In CVPR, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Damrlll and Dan Klein. Neural module networks. In CVPR, 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Carlos Arteta, Victor Lempitsky, and Andrew Zisserman. Counting in the wild. In ECCV, 2016.
Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R. Selvaraju, Dhruv Batra, and Devi Parikh. Counting everyday objects in everyday scenes. CVPR, 2017.
Zhi-Qi Cheng. Jun-Xiu Li, Qi Dai, Xiao Wu, and Alexander Hauptmann. Learning spatial awareness to improve crowd counting. In ICCV, 2019.
Hisham Cholakkal, Guolei Sun, Fahad Shahbaz Khan, and Ling Shao. Object counting and instance segmentation with image-level supervision. In CVPR, 2019.
Simone Cutini and Mario Bonato. Subitizing and visual short-term memory in human and human species: a common shared system? Frontiers in Psychology, 2012.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NeurIPS, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint 0tt.::/606000844/ 2016.
Francis Galton. One vote, one value. Nature, 1907.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. CVPR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. 2012.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017.
Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-condition graph net- works for relational reasoning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
10

---

Published as a conference paper at ICLR 2021
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine rea- soning. arXiv preprint w8Xvv1103.00367. 2018.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019a.
Drew A. Hudson and Christopher D. Manning. Learning by abstraction: The neural state machine.
In NeurlPS, 2019b.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Mille and Xinlei Chen. In defense of grid features for visual question answering. arXiv preprint arXiv:2001. 1.361,, 2020.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Mareus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0.1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:180.009956 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Edna L. Kaufman, Miles W Lord, Thomas Whelan Reese, and John Volkmann. The discrimination of visual number. The American journal of psychology, 1949.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Issam H Laradji, Negar Rostamzadeh, Pedro O Pinheiro, David Vazquez, and Mark Schmidt. Where are the blobs: Counting by localization with point supervision. ECCV, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. CVPR, 2017.
Mateusz Malinowski, Carl Doersch, Adam Santoro, and Peter W. Battaglia. Learning visual question answering by bootstrapping hard attention. In ECCV, 2018.
Mark Marsden, Kevin McGuinness, Suzanne Little, Ciara Keogh, and Noel E O'Cnnorr People, penguins and petri dishes: Adapting object counting models to new visual domains and object types without forgetting. In CVPR, 2018.
Duy-Kien Nguyen and Takayuki Okatani. Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering. In CVPR, 2018.
Daniel Onoro-Rubio and Roberto J Lopez-Sastre. Towards perspective-free object counting with
deep learning. In ECCV, 2016.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection point clouds with image votes. arXiv preprint arXiv 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In NeurlPS, 2017.
11

---

Published as a conference paper at ICLR 2021
Vishwanath A. Sindagi and Vishal M. Patel. A survey of recent advances in emn- crowd counting and density estimation. PR Letters, 2017.
enn-basee based single image
Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. In ICLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Weiyao Wang. Du Tran, and Matt Feiszli. What makes training multi-modal networks hard? arXiv preprint arXiv:1905. 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2, 2019. URL htt tuub. com/ weweebookreeeearrceeeeeeett.e
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Weidi Xie, J Alison Noble, and Andrew Zisserman. Microscopy cell counting and detection with fully convolutional regression networks. Computer methods biomechanics and biomedical engineering: Imaging & Visualization, 2018.
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV, 2017.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. CVPR, 2019.
Jianming Zhang, Shugao Ma, Mehrnoosh Sameki, Stan Sclaroff, Margrit Betke, Zhe Lin, Xiaohui Shen, Brian Price, and Radomir Mech. Salient object subitizing. In CVPR, 2015.
Yan Zhang, Jonathon S. Hare, and Adam Prenee-Bennett. Learning count objects in natural images for visual question answering. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
We use Pytorch to implement our model modular framework for vision and language multi- modal research from Facebook Al Research (FAIR).
Question representation. was shown in Natural Language Processing (NLP) research that adding SA layers helps to produce informative and discriminative language representations (De- vlin et al., 2019); and we also empirically observe better results (0.7% improvement accuracy and 0.1 reduction in RMSE according to analysis on HowMany- (Trott et al., 2018) 8) val set). We cap all the questions to the same length N as in common practice and pad all-zero vectors to shorter questions (Nguyen & Okatani, 2018).
Our design for the self-attention layer closely follows (Vaswani et al., 2017) and uses multi-head attentions (h=8 heads) with each head having D dimensions and attend with separate set of keys, queries, and values. Layer normalization and feed-forward network are included without position embeddings.
Object detection based counting. We train Faster R-CNN (Ren et al., 2015) with feature pyramid networks (Lin et al., 2017) using the latest implementation on Detectron2 (Wu et al., 2019). For fair comparison, we also use ResNet-50 backbone (He et al., 2016) pre-trained ImageNet, the same for our counting module. The detector is trained on the train2014 split of COCO images, which referred as the train set for common object counting (Chattopadhyay network for 90K iterations, reducing learning rate by 0. x at 60K and 80K iterations starting from base learning rate of 0.02. The batch size set 16. Both left-right flipping and scale augmentation (randomly sampling shorter-side from (640, 672, 704, 736, 768, 800)) are used. The reference AP (Ren et al., 2015) on COCO val2017 split is 37.1. We directly convert the testing output of the detector to the per-category counting numbers.
al., 2017). We train the
B DEFINITION OF RMSE VARIANTS
Besides accuracy and RMSE, 5 object counting (Chattopadhyay et al., 2017) additionally proposed several variants of RMSE evaluate system's counting ability. For convenience, also include them here. The standard RMSE is defined as:
M 1 M
RMSE
Σ 1=1
(6 c2)
(5)
where & is ground-truth, & is prediction, and N is the number of examples. Focusing more on non-zero counts, RMSE-nZ tries to evaluate model's counting ability harder examples where the answer is at least one:
RMSE-IZ nz
Σ
(a: - c;
(6)
Mns
iE (1]62200)
where Maz is the number of examples where ground-truth is non-zero. To penalize the mistakes when the count number is small (as making mistake of 1 when the ground-truth is 2 is more serious than when the ground-truth 100), rel-RMSE is proposed as:
M
(a
c))²
rel-RMSE
M
+1
And finally, rel-RMSE-nZ is used calculate the relative RMSE for non-zero examples both challenging and aligned with human perception.
"https://mmf.r
13

---

Published as a conference paper at ICLR 2021
Instance supervision
Method
RMSE
RMSE-nZ
rel-RMSE J rel-RMSE-uL
LC-ResFON (2018)
0.31
1.20
0.17
0.61
LC-PSPNet (2018)
/
0.35
1.32
0.20
0.70
glance-noft-21 (2017)
X
0.50
1.83
0.27
0.73
CountSeg (2019)
X
0.29
1.14
0.17
0.61
MoVie
X
0.36
1.37
0.18
0.56
Table 6: Common object counting on VOC test set with various RMSE metrics.
# Train params (M) Test params (M) Train mem (G) Train speed (s/iter)
MCAN-Large (2019)
218.2
218.2
10.5
0.84
MCAN-Large MoVie
260.2
241.2
11.7
0.89
Table 7: Adding MoVie as a module to MCAN. Training speed is ~5% slower, and the additional parameters during testing is minimal (~10%) uses the joint branch only.
C COMMON OBJECTS COUNTING ON VOC
As mentioned in Sec. 4.1 of the main paper, present the performance of MoVie test split of Pascal VOC counting dataset in Tab. 6. Different from COCO, the VOC dataset much smaller with 20 object categories (Everingham et al., 2015). We results to the state-of-the-ar method, CountSeg (Cholakkal et al., 2019) in two relative metrics (rel- RMSE and rel-RMSEEnz) and falls behind in RMSE and RMSE-nz. In contrast, shown in Tab. 3, MoVie outperforms CountSeg on COCO with a significant margin on three RMSE metrics. The performance difference in two datasets suggests that MoVie scales better than CountSeg in terms of dataset size and number of categories. Moreover, the maintained advantage on relative metrics indicates the output of MoVie better aligned with human perception (Chattopadhyay al., 2017).
that MoVie achieves comparable
D COUNTING MODULE FOR VQA ARCHITECTURES
As mentioned in 3.2, integrate MoVie into VQA models counting module. Naturally, such an integration leads to changes in model size, training speed etc. We report the added computational costs in Tab. for our three-branch fusion scheme into MCAN. We see the training speed only 55% slower, and the additional parameters used during testing is kept minimal (~ (~10%)) Note that since the integration of Vie mainly benefits 'number' questions, is different from general model size increase.
Similar MCAN, we also conducted experiments incorporating MoVie Pythia (Jiang et al., 2018), the 2018 VQA challenge winner, where we trained using the VQA 2.0 traintval, and eval- uated on test-dev using the server. We observe even more significant improvements on Pythia for "number' related questions (Tab. 8). MoVie also improves the performance of the network in all other categories, verifying that our MoVie generalizes to multiple VQA architectures.
E VISUALIZATION OF WHERE MOVIE HELPS
When MoVie is used counting module for generic VQA tasks, we fuse features pooled from MoVie and the features used by state-of-the-art VQA models (e.g. MCAN (Yu et al., 2019)) to jointly predict the answer. Then a natural question arise: where does Mo help? To answer this question, we want visualize how important MoVie and the original VQA features contribute to the final answer produced by the joint model. We conduct this study for each the 55 question types listed in the VQA 2.0 dataset (Antol et al., 2015) for better insights.
Specifically, suppose a fused representation in the joint branch is denoted as oi+(i where is from the VQA model, is from MoVie, and f(-) is the function consisting of layers applied after the features are summed up. We can compute two variants of this representation o: one without v: (i)) and without i: f-)= The similarity score is then computed between two pairs
14

---

Published as a conference paper at ICLR 2021
Method
Test set
Yes/No t
Number 1
Other f
Overall t
MCAN-Large X-101 (2020)
88.46
55.68
62.85
72.59
lest-dev
MCAN-Large X-101 + MoVic
88,39
57.05
63.28
72.91
Pythia X-101 (2018)
84.13 85.15
45.98 53.25
58.76
67.76
test-dev
Pythia X-101 + MoVic
59.31
69.26
Table 8: VQA accuracy of Pythia with and without Mo on VQA 2.0 test-dev set.
For
exte
wo
Figure 5: Visualization of where MoVie helps MCAN for different question types on VQA 2.0 val set. We compute the probability by assigning each question to MoVie based on similarity scores (see Sec. E for detailed explanations). The top contributed question types are counting related, confirming that statc-of-the-ar VQA models that perform global fusion are not idcally designed for counting, and the value of MoVic with local fusion. Best viewed on a computer screen with zoom.
Figure 6: Similar to Fig. 5 but with Pythia. Best viewed on a computer screen with zoom.
via dot-product: p==o²o o-v and p. p o.j. Given one question, assign score of to MoVie pt>Pv. and otherwise 0. The scores within each question type are then averaged, and produces the probability of how MoVie is chosen over the base VQA model for that particular question type.
We take two models as examples. One is MCAN-Small (Yu et al., 2019) + Mo (three-banchh and the other replaces MCAN-Small with Pythia (Jiang et al., 2018). The visualizations
15

---

Published as a conference paper at ICLR 2021
shown in Fig. 5 and Fig. 6, respectively. We sort the question types based on how much Mo Vie has contributed, i.. the "probability'. Some observations:
Mo shows significant contribution in the counting questions for both MCAN and Pythia: the top three question types are consistently 'how many people are in', "how many people are", and 'how many', this evidence strongly suggests that existing models that fuse fea- tures globally between vision and language are not well suited for counting questions, and confirms the value of incorporating Mo Vie (that performs fusion locally) as a counting module for generic VQA models;
The 'Yes/No' questions are likely benefited from Mo well, since the contribution of Mo spreads several question types belong to that category (e.g. 'are', 'are they', 'do', etc.) - this maybe because counting also includes verification-of-exiittnn questions such image":
as 'are there people wearing hats
For Pythia, we also find it likely helps 'color' related questions (e.g. 'what color are the', 'what color', etc.) and some other types this strengthens our exploration that our model contributes beyond counting capability.
16


Reviews:
[{"review_id": "zRj_la3Ya8s", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Description:\n\nThis work presents Modulated conVolutional bottlenecks (MoVie) to focus on visual counting where multi-modal information is fused locally. By providing motivation of modulation for visual counting, this paper presents the MoVie module which consists of four modulated convolutional bottlenecks. This paper generalizes FiLM (Perez et al 2018) as a residual function for modulation and introduces a modulation block for MoVie. MoVie can be easily incorporated in any generic VQA on top of any convolutional feature map (eg shown for two models Pythia and MCAN). Extensive ablation studies are presented with SOTA results reported for multiple tasks: open-ended counting with question queries (Howmany-QA and TallyQA) and counting common objects with class queries, apart from the VQA challenge 2.0. \n\nStrength:\n- This paper is well-motivated with well-justified compelling results. \n- Generalization of FiLM as a residual function for modulation is really interesting and using a learnable weight matrix is intuitive. \n- Extensive interesting ablation studies have been provided - number of modulated bottlenecks, modulation design, scale robustness, etc. \n- SOTA on multiple counting tasks plus VQA challenge 2.0. \n- Extensive analysis has been provided with interesting visualization maps. \n\nWeakness:\n- One might argue that anonymity is not properly maintained by mentioning “we won first place in the VQA challenge 2020”. In the future, it is expected to set a precedence by appropriately delexicalizing and stating “we achieve X position in the challenge” (similar to anonymizing Github links)\n- In Section 3.2, three branch training is not motivated clearly. The ablation study without three-branch training would help identify the main signal in the training and further strengthen the claims about the generalization of MoVie. \n- Implementation details related to the resources, framework, training days, etc. would help in reproducibility apart from the promised code release.  \n\nQuestions:\n- In ML retrospectives Perez et al (2019) noted that the FiLM model could easily overfit the data. Was a similar situation observed with MoVie module? Could the authors further specify how much the results varied for different regularization? \nhttps://ml-retrospectives.github.io/neurips2019/accepted_retrospectives/2019/film/\n- In footnote 1, the authors note that the addition of MoVie bottleneck only at the top improved the performance. However, Perez et al. (2019) noted that “Without any neural layers following a FiLM layer, FiLM would have a very limited capacity”. Could the authors comment on this observation?\n\nSuggestions/Comments:\n- Please take care of using citep compared to citet (natbib style) appropriately. In the current version, it is really confusing in the main text. \n- It would help to bold the best results in Table 1, 4 and 5\n- Please be consistent with the usage of “Tab” and “Tables”. \n- Please mention the metric used in Table 4 and 5. \n- The backbone used for MoVie in Table 3,4 and 5 should be clearly stated. \n- Section 4.2 We use choose -> We choose\n\nThis is a good paper and it would be interesting to see the discussions to further improve the paper. \n\n--------------------------------------------------------------------------------------------------------------------------------------------------------\nPost Rebuttal update:\n\nThanks to the authors for providing relevant details and fellow reviewers for nice discussions. Original rating is maintained. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "z3lkVVd8ys4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to count general objects for a query. Inspired by Perez et al. (2018), the authors design a convolutional neural network consisting of modulated bottlenecks. While the other methods utilize an explicit approach with an object detector, the proposed method is a data-driven and implicit one. Additionally, the authors highlight that MoVie can be integrated with other reasoning tasks. The authors report the experimental results for object counting, visual question answering (VQA), and GQA.\n\n**Pros**\n- The proposed approach using modulated convolutions seems reasonable. The motivation is clarified in Sec. 3.1.\n- The generalization beyond counting is a good insight for visual reasoning tasks.\n- Experimental results show the state-of-the-art performance on counting, VQA, and GQA. Perhaps the authors can compare computational times between the proposed method and the existing methods with symbolic models because MoVie seems faster than them. Table 7 shows the additional computational cost of MoVie, but the reviewer is also interested in comparisons to symbolic models.\n\n**Cons**\n- One of the highlighted result is the generalization beyond counting. However, Table 5 only shows the performance with the performance of MoVie in comparison to the other methods. An additional ablation study is necessary to show the contribution of MoVie itself. Table 4 shows Movie boosts the performance not only for the number questions but also for Yes/No questions and the other types for validation data; however, the performance gain is moderate for test-dev data. \n- Table 2 shows that the combination of X-101 and MoVie achieves the best performance; as shown, the backbone network is different from the existing methods. The authors should use the same backbone network for a fair comparison.\n- The reason why the authors omit beta from Eq. (1) is unclear as far as observed from Table 2. If both beta and residual connections are employed, ACC is slightly better, while RMS is worse than that of MoVie. The significance is also unclear on this table.\n\n**Minor comments**\n- Use of \"ConvNet\" should be avoided since it is confusing whether \"ConvNet\" is a general convolutional neural network or a particular method or a library.\n\n**Overall rating**\nThe reviewer is leaning toward acceptance because the motivation is clear and because the proposed method is effective in several experiments. The reviewer would like the authors to solve the cons above to improve the rating.\n\n**Additional comment after rebuttal**\nThanks to to the sufficient answers and results, the first rating is maintained.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_oplS_SfF_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper addresses the counting-based VQA scenarios, as well as general object counting problems. On the VQA side, following [1], they rely on grid-based features/predictions, vs region-based annotation/prediction. Then following [2, 3], they employ LSTM with a self attention layer for the question encoding. As part of the contribution of the paper, they modified FiLM (Feature-wise Linear Modulation) formulation to form an alternative modulation for the task of counting, which they call it MoVie. Here is how the paper reformulate FiLM:\n$$\\bar{v}_{FiLM}=\\left [ v\\otimes \\left ( 1\\oplus\\Delta \\gamma  \\right ) \\right ]\\oplus \\beta =v\\oplus\\left [ (v\\otimes\\Delta \\gamma)\\oplus\\beta \\right ]$$\n, where $v$ represents feature vector, $\\gamma$ and $\\beta$ are both conditioned on query representation, and $\\left ( v \\otimes \\Delta \\gamma\\right )\\oplus \\beta$ as the residual in FiLM formulation. \n\nDifferently to FiLM residual function, in MoVie, they added a weight matrix, $W$, to learn $C$ inner products between $v$ and $\\delta \\gamma$ weighted individually by each column in $W$, to capture more information. In addition, they removed the residual $\\beta$. $\\bar{v}_{MoVie}=v \\oplus W^{T}(v \\otimes \\Delta \\gamma)$. Empirically they showed the presence of $\\beta$ is not essential in MoVie formulation. \n\nBy removing $\\beta$ from the formula, depending on the dimensions of the $W_{\\beta}$, and $W$, MoVie could be more or less efficient than the FiLM original formulation. \n\nS: strength W: weakness/ room for improvement \n\nS1: It is intuitive but not trivial that the modification in the FiLM formulation may improve the performance in VQA object counting-related tasks. So the empirical experiments were comprehensive to confirm the assumption, and it is a valuable founding.\n \nS2. W1: They showed MoVie’s performance beyond counting tasks and on more general VQA datasets such as GQA and CLEVR. However, I would like to see how replacing FiLM with MoVie changes the performance.\n \nW2: It is valuable to see the visualization of some failure and successful cases of the model (Figure 4). However, in order to give a better insight on the MoVie advantages vs FiLM, I suggest presenting some cases that FiLM fails but MoVie is able to respond to the query/count correctly. \n \nW3. In the paper, in multiple places it is stated that the approach is more efficient compared to alternative approaches. However it is not really clear to me how it is considered as more efficient. For example, in Table 2, the number of parameters for MoVie is relatively higher than other approaches. Could you elaborate on the efficiency of the model?\n \nW4. For the general counting task, it is presented that MoVie outperforms other methods in the object counting literature on the COCO dataset. However in appendix, it is presented that the model doesn’t have competitive results on the PASCAL VOC dataset not only compared to [4] but also to some other papers in literature. The reason that is mentioned in the paper is related to the scale of the dataset [5]. Object counting datasets are usually compared based on their performance while objects are heavily occluded or in crowd counting tasks. Since the presented failure tasks in Figure 4, are both examples of occlusions, I am wondering how the model performs on popular and challenging crowd counting datasets, such as Penguins, Trancos, or Shanghai datasets. \n \nW5. Image-level object counting, implicit counting, and multimodal reasoning aspects of the tasks were all presented in other papers in the literature (and well-cited within the paper). It is not clear to me besides improving the FiLM formula, and employing it for VQA counting-related and counting tasks, what are the other contributions of the paper? \n\nReferences:\n\n[1] Jiang et al, In Defense of Grid Features for Visual Question Answering, CVPR 2020.\n\n[2] Vaswani et al, Attention is all you need, NeurIPS 2017.\n\n[3] Yu et al, Deep modular co-attention networks for visual question answering, CVPR 2019.\n\n[4] Cholakkal et al, Object counting and instance segmentation with image-level supervision, CVPR 2019.\n\n[5] Laradji et al, Where are the blobs: Counting by localization with point supervision, ECCV 2018.\n\n######################################\n\nReason for the decision: \nMentioned points marked as S and W. Adding more clarity to the contributions specific to the paper and comparisons as mentioned,  could change my score.\n\n#####################################\nAfter rebuttal: (from 5 to 6)\n\nThanks for the clarifying the points in your response. I am happy to change my score to Accept the paper as most my concerns are addressed and I believe the paper is a good fit in this conference. \n\nA comment to the authors that didn't impact my score but raised some concerns:\n\nMy concern is about mentioning the first place in VQA challenge 2020, both in paper and also in rebuttal comments. The review process is double blind and pointing out to other contributions that are public and reviewers may have already known about the winner teams, may not be fair. I know that papers can be online on arXiv but pointing to another venue as part of the contributions, may reveal the identity of the authors explicitly (if reviewers already know about the challenge)", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rx4l8UyrUmf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "***************************************************************************\nSUMMARY\n\nThe paper presents MoVie (short for ‘modulated convolutional bottlenecks’): a new modulation block which reformulates the FiLM approach (Perez et al. 2018) to fuse the query and the image locally, and perform implicit and holistic visual counting. The proposed approach showcases impressive results, establishing new sota both for open-ended and common object counting benchamrks, and competitive results for visual reasoning beyond counting tasks. \n***************************************************************************\nSTRENGTHS\n\n- The paper is fairly well-written, well-structured and easy to follow.  \n- Obtained results are quite impressive, and establish new sota both for open-ended (Howmany-QA and TallyQA) and common object (COCO) counting benchmarks. The proposed approach also obtains competitive results on VQA (CLEVR and GQA datasets). The authors also say that the proposed approach was a key part of their winning solution of the 2020 VQA challenge. \n- The paper also provides a comprehensive ablative study which empirically validates most of the design choices of the approach. This ablative study is completed with qualitative analysis and visualization of some results.\n- The paper shows that the proposed approach can be easily plugged into generic VQA systems to improve their ‘counting’ capabilities. This is an interesting property, since, in theory, almost all VQA systems could benefit from this approach.\n- The authors plan to make their code public. It could be a valuable resource for researchers working on visual counting (and more generally visual reasoning) and modulated convolutions.\n***************************************************************************\nWEAKNESSES AND REMARKS\n\n- I have some concerns regarding the novelty of the proposed approach. The paper is revisiting an existing idea (i.e. using modulated convolutions for visual reasoning, which has already been proposed in the FiLM paper) with a new formulation, which gives better results, and thus is applied to natural images (in addition to synthetical ones, like in the FiLM paper). \n- The proposed modulation block (a reformulation of the one presented in the FiLM paper) is empirically validated (with a comprehensive and interesting ablative study), but it would have been interesting to give some insights, motivations and theoretical justifications about the key differences compared to the FiLM formulation (i.e. the use of W and the absence of beta - by the way, from Table 1-(a), we can notice that using beta even works slightly better, so I’m not sure to understand the motivations behind this choice-). More globally, the paper contains a lot of empirical choices and lacks theoretical justifications.\n- In Table 5, some major recent works on VQA are missing. One can mention, for example, LXMERT (Tan et al., EMNLP 2019) and LCGN (Hu et al., ICCV 2019), both obtaining better results on GQA than those presented in this paper, without being supervised with scene-graphs (unlike NSM).\n- It would have been interesting to give more details about the query representation in the paper itself, rather than only referring to the appendix. I understand that it’s due to the lack of space, but it is a little bit frustrating to have no information at all in the paper about this point. \n- A similar remark regarding the winning solution of the 2020 VQA challenge. The paper is only mentioning that the proposed approach helped them to secure their first place, without giving any additional information. In my opinion, the paper should either give more details about the winning solution, or keep these information for a dedicated paper without mentioning it in this submission.\n- When describing the results presented in Table 1-(b), the paper says that “performance saturates around 4”. But, one can notice that accuracy continues increasing (respectively decreasing for RMSE) when using 5 modulated bottlenecks. It would have been interesting to go further and try using more bottelnecks.\n- In my opinion, the ‘beyond counting’ sub-section could be considerably improved if the paper explicitly mention that, unlike CLEVR, the GQA dataset does not contain any counting-related questions. In its present form (without knowing this information), we can assume that the gain in accuracy is simply due to better results on this category of questions, when using MoVie. Knowing that there is no such questions in GQA is crucial to understand that the proposed approach can serve as a general mechanism to improve visual reasoning, beyond counting tasks. It would also have been interesting to evaluate the approach on other visual reasoning tasks beyond VQA (e.g. language-driven comparison of images on the NLVR2 dataset – Suhr et al. ACL 2019- which was used in the FiLM paper) to confirm this claim. \n***************************************************************************\nJUSTIFICATION OF RATING\n\nDespite the concerns mentioned above, I think the proposed approach is an interesting addition to the visual counting literature, especially considering its quite impressive results. I also think the paper could be considerably improved by taking into account the comments made above. Overall, I'm leaning to accept (6: Marginally above acceptance threshold).\n\n***************************************************************************\nPOST REBUTTAL UPDATE\n\nThe authors provided a detailed rebuttal which addressed almost all my concerns and answered most of my questions. I wil therefore update my rating from 6 (marginally above acceptance threshold) to 7 (good paper, accept). I believe this paper should be accepted.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a modification of the well-known FILM model for VQA which targets counting problems in particular, which have been a known weakness of existing models. The improvements have also been tested beyond counting. The experimental results are convincing, in particular a scientific competition has been won. The reviewers also appreciated convincing ablation studies.

The idea bas been perceived as interesting enough for publication, and in combination with the experimental results, this compensated several perceived weaknesses (limited novelty w.r.t. the modified FILM model; justifications of some design choices).

All reviewers agreed that this paper is of interest to the community and proposed acceptance. The AC concurs.
Decision: Accept (Poster)