Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
------------------
Summary:
------------------
This work introduces a discrete and interpretable model for answering visually grounded counting questions. The proposed model executes a sequential decision process in which it 1) selects an image region to "add to the count" and then 2) updates the likelihood of selecting other regions based on their relationships (defined broadly) to the selected region. After substantial module pre-trianing, the model is trained end-to-end with the REINFORCE policy gradient method (with the recently proposed self-critical sequence training baseline). Compared to existing approaches for counting (or VQA in general), this approach not only produces lower error but also provides a more human-intuitive discrete, instance-pointing representation of counting. 

-----------------------
Preliminary Evaluation:
-----------------------
The paper presents an interesting approach that seems to outperform existing methods. More importantly in my view, the model treats counting as a discrete human-intuitive process. The presentation and experiments are okay overall but I have a few questions and requests below that I feel would strengthen the submission.

------------------
Strengths:
------------------
- I generally agree with the authors that approaching counting as a region-set selection problem provides an interpretable and human-intuitive methodology that seems more appropriate than attentional or monolithic approaches. 

- To the best of my knowledge, the writing does a good job of placing the work in the context of existing literature.

- The dataset construction is given appropriate attention to restrict its instances to counting questions and will be made available to the public.

- The model outperforms existing approaches given the same visual and linguistic inputs / encodings. While I find improvements in RMSE a bit underwhelming, I'm still generally positive about the results given the improved accuracy and human-intuitiveness of the grounded outputs.

- I appreciated the analysis of the effect of "commonness" and think it provides interesting insight into the generalization of the proposed model.

- Qualitative examples are interesting.

------------------
Weaknesses:
------------------
- There is a lot going on in this paper as far as model construction and training procedures go. In its current state, many of the details are pushed to the supplement such that the main paper would be insufficient for replication. The authors also do not promise code release. 

- Maybe it is just my unfamiliarity with it, but the caption grounding auxiliary-task feels insufficiently introduced in the main paper.  I also find it a bit discouraging that the details of joint training is regulated to the supplementary material, especially given that the UpDown is not using it! I would like to see an ablation of the proposed model without joint training.

- Both the IRLC and SoftCount models are trained with objectives that are aware of the ordinal nature of the output space (such that predicting 2 when the answer is 20 is worse than predicting 19). Unfortunately, the UpDown model is trained with cross-entropy and lacks access to this notion. I believe that this difference results in the large gap in RMSE between IRLC/SoftCount and UpDown. Ideally an updated version of UpDown trained under an order-aware loss would be presented during the rebuttal period. Barring that due to time constraints, I would otherwise like to see some analysis to explore this difference, maybe checking to see if UpDown is putting mass in smooth blobs around the predicted answer (though there may be better ways to see if UpDown has captured similar notions of output order as the other models).

- I would like to see a couple of simple baselines evaluated on HowMany-QA. Specifically, I think the paper would be stronger if results were put in context with a question only model and a model which just outputs the mean training count. Inter-human agreement would also be interesting to discuss (especially for high counts).

- The IRLC model has a significantly large (4x) capacity scoring function than the baseline methods. If this is restricted, do we see significant changes to the results?

- This is a relatively mild complaint. This model is more human-intuitive than existing approaches, but when it does make an error by selecting incorrect objects or terminating early, it is no more transparent about the cause of these errors than any other approach. As such, claims about interpretability should be made cautiously.  

------------------
Curiosities:
------------------
- In my experience, Visual Genome annotations are often noisy, with many different labels being applied to the same object in different images. For per-image counts, I don't imagine this will be too troubling but was curious if you ran into any challenges.

- It looks like both IRLC and UpDown consistently either get the correct count (for small counts) or underestimate. This is not the Gaussian sort of regression error that we might expect from a counting problem. 

- Could you speak to the sensitivity of the proposed model with respect to different loss weightings? I saw the values used in Section B of the supplement and they seem somewhat specific. 

------------------
Minor errors:
------------------
[5.1 end of paragraph 2] 'that accuracy and RSME and not' -> 'that accuracy and RSME are not'
[Fig 9 caption] 'The initial scores are lack' -> 'The initial scores lack'
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper presents a novel method for answering “How many …?” questions in the VQA datasets. Unlike previously proposed approaches, the proposed method uses an iterative sequential decision process for counting the relevant entity. The proposed model makes discrete choices about what to count at each time step. Another qualitative difference compared to existing approaches is that the proposed method returns bounding boxes for the counted object. The training and evaluation of the proposed model and baselines is done on a subset of the existing VQA dataset that consists of “How many …?” questions. The experimental results show that the proposed model outperforms the baselines discussed in the paper.

Strengths:
1.	The idea of sequential counting is novel and interesting.
2.	The analysis of model performance by grouping the questions as per frequency with which the counting object appeared in the training data is insightful. 
 
Weaknesses:
1.	The proposed dataset consists of 17,714 QA pairs in the dev set, whereas only 5,000 QA pairs in the test set. Such a 3.5:1 split of dev and test seems unconventional. Also, the size of the test set seems pretty small given the diversity of the questions in the VQA dataset.
2.	The paper lacks quantitative comparison with existing models for counting such as with Chattopadhyay et al. This would require the authors to report the accuracies of existing models by training and evaluating on the same subset as that used for the proposed model. Absence of such a comparison makes it difficult to judge how well the proposed model is performing compared to existing models.
3.	The paper lacks analysis on how much of performance improvement is due to visual genome data augmentation and pre-training? When comparing with existing models (as suggested in above), this analysis should be done, so as to identify the improvements coming from the proposed model alone.
4.	The paper does not report the variation in model performance when changing the weights of the various terms involved in the loss function (equations 15 and 16).
5.	Regarding Chattopadhyay et al. the paper says that “However, their analysis was limited to the specific subset of examples where their approach was applicable.” It would be good it authors could elaborate on this a bit more.
6.	The relation prediction part of the vision module in the proposed model seems quite similar to the Relation Networks, but the paper does not mention Relation Networks. It would be good to cite the Relation Networks paper and state clearly if the motivation is drawn from Relation Networks.
7.	It is not clear what are the 6 common relationships that are being considered in equation 1. Could authors please specify these?
8.	In equation 1, if only 6 relationships are being considered, then why does f^R map to R^7 instead of R^6?
9.	In equations 4 and 5, it is not clarified what each symbol represents, making it difficult to understand.
10.	What is R in equation 15? Is it reward?

Overall:
The paper proposes a novel and interesting idea for solving counting questions in the Visual Question Answering tasks. However, the writing of the paper needs to be improved to make is easier to follow. The experimental set-up – the size of the test dataset seems too small. And lastly, the paper needs to add comparisons with existing models on the same datasets as used for the proposed model. So, the paper seems to be not ready for the publication yet.
review_title: Size of test set too small, lacks comparison with existing models, paper writing needs to be improved.

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a new approach for counting in VQA called Interpretable Counting in Visual Question Answering.  The authors create a new dataset (HowMany-QA) by processing the VQA 2.0 and Visual Genome dataset. In the paper, the authors use object detection framework (R-FCN) to extract bounding boxes information as well as visual features and propose three different strategies for counting. 1: SoftCount; 2: UpDown; 3: IRLC.  The authors show results on HowMany-QA dataset for the proposed methods, and the proposed IRLC method achieves the best performance among all the baselines.   

[Strenghts]

This paper first introduced a cleaned visual counting dataset by processing existing VQA 2.0 and Visual Genome dataset, which can filter out partial non-counting questions. The proposed split is a good testbed for counting in VQA. 

The authors proposed 3 different methods for counting, which both use object detection feature trained on visual genome dataset.  The object detector is trained with multiple objectives including object detection, relation detection, attribute classification and caption grounding to produce rich object representation. The author first proposed 2 baselines: SoftCount uses a Huber loss, UpDown uses a cross entropy loss. And further proposed interpretable RL counter which enumerates the object as a sequential decision process. The proposed IRLC more intuitive and outperform the previous VQA method  (UpDown) on both accuracy and RMSE. 

[Weaknesses]

This paper proposed an interesting and intuitive counting model for VQA. However, there are several weaknesses existed:

1: The object detector is pre-trained with multiple objectives. However, there is no ablation study to show the differences. Since the model only uses the object and relationship feature as input, the authors could show results on counting with different  objects detector. For example, object detector trained using object + relation v.s. object + relation + attribute v.s. object + relation + attribute + caption. 

2: Figure 9 shows an impressive result of the proposed method. Given the detection result, there are a lot of repetitive candidates detection bounding boxes. Without any strong supervision, IRLC could select the correct bounding boxes associated with the different objects. This is interesting, however, the authors didn't show any quantitative results on this. One experiment could verify the performance on IRLC is to compute the IOU between the GT COCO bounding box annotation on a small validation set. The validation set could be obtained by comparing the number of the bounding box and VQA answer with respect to similar COCO categories and VQA entities. 

3: The proposed IRLC is not significantly outperform baseline method (SoftCount) with respect to RMSE (0.1). However, it would be interesting to see how the counting performance can change the result of object detection. As Chattopadhyay's CVPR2017 paper Sec 5.3 on the same subset as in point 2. 

[Summary]

This paper proposed an interesting and interpretable model for counting in VQA. It formulated the counting as a sequential decision process that enumerated the subset of target objects. The authors introduce several new techniques in the IRLC counter. However, there is a lack of ablation study on the proposed model. Taking all these into account, I suggest accepting this paper if the authors could provide more ablation study on the proposed methods. 

review_title: Review from AnonReviewer2

Meta-review: Important problem and all reviewers recommend acceptance. I agree.
Decision: Accept (Poster)