Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper addresses the counting-based VQA scenarios, as well as general object counting problems. On the VQA side, following [1], they rely on grid-based features/predictions, vs region-based annotation/prediction. Then following [2, 3], they employ LSTM with a self attention layer for the question encoding. As part of the contribution of the paper, they modified FiLM (Feature-wise Linear Modulation) formulation to form an alternative modulation for the task of counting, which they call it MoVie. Here is how the paper reformulate FiLM:
$$\bar{v}_{FiLM}=\left [ v\otimes \left ( 1\oplus\Delta \gamma  \right ) \right ]\oplus \beta =v\oplus\left [ (v\otimes\Delta \gamma)\oplus\beta \right ]$$
, where $v$ represents feature vector, $\gamma$ and $\beta$ are both conditioned on query representation, and $\left ( v \otimes \Delta \gamma\right )\oplus \beta$ as the residual in FiLM formulation. 

Differently to FiLM residual function, in MoVie, they added a weight matrix, $W$, to learn $C$ inner products between $v$ and $\delta \gamma$ weighted individually by each column in $W$, to capture more information. In addition, they removed the residual $\beta$. $\bar{v}_{MoVie}=v \oplus W^{T}(v \otimes \Delta \gamma)$. Empirically they showed the presence of $\beta$ is not essential in MoVie formulation. 

By removing $\beta$ from the formula, depending on the dimensions of the $W_{\beta}$, and $W$, MoVie could be more or less efficient than the FiLM original formulation. 

S: strength W: weakness/ room for improvement 

S1: It is intuitive but not trivial that the modification in the FiLM formulation may improve the performance in VQA object counting-related tasks. So the empirical experiments were comprehensive to confirm the assumption, and it is a valuable founding.
 
S2. W1: They showed MoVie’s performance beyond counting tasks and on more general VQA datasets such as GQA and CLEVR. However, I would like to see how replacing FiLM with MoVie changes the performance.
 
W2: It is valuable to see the visualization of some failure and successful cases of the model (Figure 4). However, in order to give a better insight on the MoVie advantages vs FiLM, I suggest presenting some cases that FiLM fails but MoVie is able to respond to the query/count correctly. 
 
W3. In the paper, in multiple places it is stated that the approach is more efficient compared to alternative approaches. However it is not really clear to me how it is considered as more efficient. For example, in Table 2, the number of parameters for MoVie is relatively higher than other approaches. Could you elaborate on the efficiency of the model?
 
W4. For the general counting task, it is presented that MoVie outperforms other methods in the object counting literature on the COCO dataset. However in appendix, it is presented that the model doesn’t have competitive results on the PASCAL VOC dataset not only compared to [4] but also to some other papers in literature. The reason that is mentioned in the paper is related to the scale of the dataset [5]. Object counting datasets are usually compared based on their performance while objects are heavily occluded or in crowd counting tasks. Since the presented failure tasks in Figure 4, are both examples of occlusions, I am wondering how the model performs on popular and challenging crowd counting datasets, such as Penguins, Trancos, or Shanghai datasets. 
 
W5. Image-level object counting, implicit counting, and multimodal reasoning aspects of the tasks were all presented in other papers in the literature (and well-cited within the paper). It is not clear to me besides improving the FiLM formula, and employing it for VQA counting-related and counting tasks, what are the other contributions of the paper? 

References:

[1] Jiang et al, In Defense of Grid Features for Visual Question Answering, CVPR 2020.

[2] Vaswani et al, Attention is all you need, NeurIPS 2017.

[3] Yu et al, Deep modular co-attention networks for visual question answering, CVPR 2019.

[4] Cholakkal et al, Object counting and instance segmentation with image-level supervision, CVPR 2019.

[5] Laradji et al, Where are the blobs: Counting by localization with point supervision, ECCV 2018.

######################################

Reason for the decision: 
Mentioned points marked as S and W. Adding more clarity to the contributions specific to the paper and comparisons as mentioned,  could change my score.

#####################################
After rebuttal: (from 5 to 6)

Thanks for the clarifying the points in your response. I am happy to change my score to Accept the paper as most my concerns are addressed and I believe the paper is a good fit in this conference. 

A comment to the authors that didn't impact my score but raised some concerns:

My concern is about mentioning the first place in VQA challenge 2020, both in paper and also in rebuttal comments. The review process is double blind and pointing out to other contributions that are public and reviewers may have already known about the winner teams, may not be fair. I know that papers can be online on arXiv but pointing to another venue as part of the contributions, may reveal the identity of the authors explicitly (if reviewers already know about the challenge)
review_title: VQA - object counting, room for improvement

final_score: rating: 7: Good paper, accept
review_content: ***************************************************************************
SUMMARY

The paper presents MoVie (short for ‘modulated convolutional bottlenecks’): a new modulation block which reformulates the FiLM approach (Perez et al. 2018) to fuse the query and the image locally, and perform implicit and holistic visual counting. The proposed approach showcases impressive results, establishing new sota both for open-ended and common object counting benchamrks, and competitive results for visual reasoning beyond counting tasks. 
***************************************************************************
STRENGTHS

- The paper is fairly well-written, well-structured and easy to follow.  
- Obtained results are quite impressive, and establish new sota both for open-ended (Howmany-QA and TallyQA) and common object (COCO) counting benchmarks. The proposed approach also obtains competitive results on VQA (CLEVR and GQA datasets). The authors also say that the proposed approach was a key part of their winning solution of the 2020 VQA challenge. 
- The paper also provides a comprehensive ablative study which empirically validates most of the design choices of the approach. This ablative study is completed with qualitative analysis and visualization of some results.
- The paper shows that the proposed approach can be easily plugged into generic VQA systems to improve their ‘counting’ capabilities. This is an interesting property, since, in theory, almost all VQA systems could benefit from this approach.
- The authors plan to make their code public. It could be a valuable resource for researchers working on visual counting (and more generally visual reasoning) and modulated convolutions.
***************************************************************************
WEAKNESSES AND REMARKS

- I have some concerns regarding the novelty of the proposed approach. The paper is revisiting an existing idea (i.e. using modulated convolutions for visual reasoning, which has already been proposed in the FiLM paper) with a new formulation, which gives better results, and thus is applied to natural images (in addition to synthetical ones, like in the FiLM paper). 
- The proposed modulation block (a reformulation of the one presented in the FiLM paper) is empirically validated (with a comprehensive and interesting ablative study), but it would have been interesting to give some insights, motivations and theoretical justifications about the key differences compared to the FiLM formulation (i.e. the use of W and the absence of beta - by the way, from Table 1-(a), we can notice that using beta even works slightly better, so I’m not sure to understand the motivations behind this choice-). More globally, the paper contains a lot of empirical choices and lacks theoretical justifications.
- In Table 5, some major recent works on VQA are missing. One can mention, for example, LXMERT (Tan et al., EMNLP 2019) and LCGN (Hu et al., ICCV 2019), both obtaining better results on GQA than those presented in this paper, without being supervised with scene-graphs (unlike NSM).
- It would have been interesting to give more details about the query representation in the paper itself, rather than only referring to the appendix. I understand that it’s due to the lack of space, but it is a little bit frustrating to have no information at all in the paper about this point. 
- A similar remark regarding the winning solution of the 2020 VQA challenge. The paper is only mentioning that the proposed approach helped them to secure their first place, without giving any additional information. In my opinion, the paper should either give more details about the winning solution, or keep these information for a dedicated paper without mentioning it in this submission.
- When describing the results presented in Table 1-(b), the paper says that “performance saturates around 4”. But, one can notice that accuracy continues increasing (respectively decreasing for RMSE) when using 5 modulated bottlenecks. It would have been interesting to go further and try using more bottelnecks.
- In my opinion, the ‘beyond counting’ sub-section could be considerably improved if the paper explicitly mention that, unlike CLEVR, the GQA dataset does not contain any counting-related questions. In its present form (without knowing this information), we can assume that the gain in accuracy is simply due to better results on this category of questions, when using MoVie. Knowing that there is no such questions in GQA is crucial to understand that the proposed approach can serve as a general mechanism to improve visual reasoning, beyond counting tasks. It would also have been interesting to evaluate the approach on other visual reasoning tasks beyond VQA (e.g. language-driven comparison of images on the NLVR2 dataset – Suhr et al. ACL 2019- which was used in the FiLM paper) to confirm this claim. 
***************************************************************************
JUSTIFICATION OF RATING

Despite the concerns mentioned above, I think the proposed approach is an interesting addition to the visual counting literature, especially considering its quite impressive results. I also think the paper could be considerably improved by taking into account the comments made above. Overall, I'm leaning to accept (6: Marginally above acceptance threshold).

***************************************************************************
POST REBUTTAL UPDATE

The authors provided a detailed rebuttal which addressed almost all my concerns and answered most of my questions. I wil therefore update my rating from 6 (marginally above acceptance threshold) to 7 (good paper, accept). I believe this paper should be accepted.

review_title: Official Blind Review #4 - post-rebuttal update

final_score: rating: 7: Good paper, accept
review_content: Description:

This work presents Modulated conVolutional bottlenecks (MoVie) to focus on visual counting where multi-modal information is fused locally. By providing motivation of modulation for visual counting, this paper presents the MoVie module which consists of four modulated convolutional bottlenecks. This paper generalizes FiLM (Perez et al 2018) as a residual function for modulation and introduces a modulation block for MoVie. MoVie can be easily incorporated in any generic VQA on top of any convolutional feature map (eg shown for two models Pythia and MCAN). Extensive ablation studies are presented with SOTA results reported for multiple tasks: open-ended counting with question queries (Howmany-QA and TallyQA) and counting common objects with class queries, apart from the VQA challenge 2.0. 

Strength:
- This paper is well-motivated with well-justified compelling results. 
- Generalization of FiLM as a residual function for modulation is really interesting and using a learnable weight matrix is intuitive. 
- Extensive interesting ablation studies have been provided - number of modulated bottlenecks, modulation design, scale robustness, etc. 
- SOTA on multiple counting tasks plus VQA challenge 2.0. 
- Extensive analysis has been provided with interesting visualization maps. 

Weakness:
- One might argue that anonymity is not properly maintained by mentioning “we won first place in the VQA challenge 2020”. In the future, it is expected to set a precedence by appropriately delexicalizing and stating “we achieve X position in the challenge” (similar to anonymizing Github links)
- In Section 3.2, three branch training is not motivated clearly. The ablation study without three-branch training would help identify the main signal in the training and further strengthen the claims about the generalization of MoVie. 
- Implementation details related to the resources, framework, training days, etc. would help in reproducibility apart from the promised code release.  

Questions:
- In ML retrospectives Perez et al (2019) noted that the FiLM model could easily overfit the data. Was a similar situation observed with MoVie module? Could the authors further specify how much the results varied for different regularization? 
https://ml-retrospectives.github.io/neurips2019/accepted_retrospectives/2019/film/
- In footnote 1, the authors note that the addition of MoVie bottleneck only at the top improved the performance. However, Perez et al. (2019) noted that “Without any neural layers following a FiLM layer, FiLM would have a very limited capacity”. Could the authors comment on this observation?

Suggestions/Comments:
- Please take care of using citep compared to citet (natbib style) appropriately. In the current version, it is really confusing in the main text. 
- It would help to bold the best results in Table 1, 4 and 5
- Please be consistent with the usage of “Tab” and “Tables”. 
- Please mention the metric used in Table 4 and 5. 
- The backbone used for MoVie in Table 3,4 and 5 should be clearly stated. 
- Section 4.2 We use choose -> We choose

This is a good paper and it would be interesting to see the discussions to further improve the paper. 

--------------------------------------------------------------------------------------------------------------------------------------------------------
Post Rebuttal update:

Thanks to the authors for providing relevant details and fellow reviewers for nice discussions. Original rating is maintained. 
review_title: Good paper; FiLM to MoVie

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method to count general objects for a query. Inspired by Perez et al. (2018), the authors design a convolutional neural network consisting of modulated bottlenecks. While the other methods utilize an explicit approach with an object detector, the proposed method is a data-driven and implicit one. Additionally, the authors highlight that MoVie can be integrated with other reasoning tasks. The authors report the experimental results for object counting, visual question answering (VQA), and GQA.

**Pros**
- The proposed approach using modulated convolutions seems reasonable. The motivation is clarified in Sec. 3.1.
- The generalization beyond counting is a good insight for visual reasoning tasks.
- Experimental results show the state-of-the-art performance on counting, VQA, and GQA. Perhaps the authors can compare computational times between the proposed method and the existing methods with symbolic models because MoVie seems faster than them. Table 7 shows the additional computational cost of MoVie, but the reviewer is also interested in comparisons to symbolic models.

**Cons**
- One of the highlighted result is the generalization beyond counting. However, Table 5 only shows the performance with the performance of MoVie in comparison to the other methods. An additional ablation study is necessary to show the contribution of MoVie itself. Table 4 shows Movie boosts the performance not only for the number questions but also for Yes/No questions and the other types for validation data; however, the performance gain is moderate for test-dev data. 
- Table 2 shows that the combination of X-101 and MoVie achieves the best performance; as shown, the backbone network is different from the existing methods. The authors should use the same backbone network for a fair comparison.
- The reason why the authors omit beta from Eq. (1) is unclear as far as observed from Table 2. If both beta and residual connections are employed, ACC is slightly better, while RMS is worse than that of MoVie. The significance is also unclear on this table.

**Minor comments**
- Use of "ConvNet" should be avoided since it is confusing whether "ConvNet" is a general convolutional neural network or a particular method or a library.

**Overall rating**
The reviewer is leaning toward acceptance because the motivation is clear and because the proposed method is effective in several experiments. The reviewer would like the authors to solve the cons above to improve the rating.

**Additional comment after rebuttal**
Thanks to to the sufficient answers and results, the first rating is maintained.
review_title: A promising approach for counting objects

Meta-review: The paper proposes a modification of the well-known FILM model for VQA which targets counting problems in particular, which have been a known weakness of existing models. The improvements have also been tested beyond counting. The experimental results are convincing, in particular a scientific competition has been won. The reviewers also appreciated convincing ablation studies.

The idea bas been perceived as interesting enough for publication, and in combination with the experimental results, this compensated several perceived weaknesses (limited novelty w.r.t. the modified FILM model; justifications of some design choices).

All reviewers agreed that this paper is of interest to the community and proposed acceptance. The AC concurs.
Decision: Accept (Poster)