Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper tackles the object counting problem in visual question answering. It is based on the two-stage method that object proposals are generated from the first stage with attention. It proposes many heuristics to use the object feature and attention weights to find the correct count.  In general, it treats all object proposals as nodes on the graph. With various agreement measures, it removes or merges edges and count the final nodes. The method is evaluated on one synthetic toy dataset and one VQA v2 benchmark dataset. The experimental results on counting are promising.  Although counting is important in VQA, the method is solving a very specific problem which cannot be generalized to other representation learning problems.  Additionally, this method is built on a series of heuristics without sound theoretically justification, and these heuristics cannot be easily adapted to other machine learning applications. I thus believe the overall contribution is not sufficient for ICLR.

Pros:
1. Well written paper with clear presentation of the method. 
2. Useful for object counting problem.
3. Experimental performance is convincing. 

Cons:
1. The application range of the method is very limited. 
2. The technique is built on a lot of heuristics without theoretical consideration. 

Other comments and questions:

1. The determinantal point processes [1] should be able to help with the correct counting the objects with proper construction of the similarity kernel.  It may also lead to simpler solutions. For example, it can be used for deduplication using A (eq 1) as the similarity matrix. 

2. Can the author provide analysis on scalability the proposed method? When the number of objects is very large, the graph could be huge. What are the memory requirements and computational complexity of the proposed method?  
In the end of section 3, it mentioned that "without normalization," the method will not scale to an arbitrary number of objects. I think that it will only be a problem for extremely large numbers. I wonder whether the proposed method scales. 

3. Could the authors provide more insights on why the structured attention (etc) did not significantly improve the result? Theoritically, it solves the soft attention problems. 

4. The definition of output confidence (section 4.3.1) needs more motivation and theoretical justification. 

[1] Kulesza, Alex, and Ben Taskar. "Determinantal point processes for machine learning." Foundations and Trends® in Machine Learning 5.2–3 (2012): 123-286.
review_title: Improve object counting with a lot of heuristics 

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Summary: 
- This paper proposes a hand-designed network architecture on a graph of object proposals to perform soft non-maximum suppression to get object count.

Contribution:
- This paper proposes a new object counting module which operates on a graph of object proposals.

Clarity:
- The paper is well written and clarity is good. Figure 2 & 3 helps the readers understand the core algorithm.

Pros:
- De-duplication modules of inter and intra object edges are interesting.
- The proposed method improves the baseline by 5% on counting questions.

Cons:
- The proposed model is pretty hand-crafted. I would recommend the authors to use something more general, like graph convolutional neural networks (Kipf & Welling, 2017) or graph gated neural networks (Li et al., 2016).
- One major bottleneck of the model is that the proposals are not jointly finetuned. So if the proposals are missing a single object, this cannot really be counted. In short, if the proposals don’t have 100% recall, then the model is then trained with a biased loss function which asks it to count all the objects even if some are already missing from the proposals. The paper didn’t study what is the recall of the proposals and how sensitive the threshold is.
- The paper doesn’t study a simple baseline that just does NMS on the proposal domain.
- The paper doesn’t compare experiment numbers with (Chattopadhyay et al., 2017).
- The proposed algorithm doesn’t handle symmetry breaking when two edges are equally confident (in 4.2.2 it basically scales down both edges). This is similar to a density map approach and the problem is that the model doesn’t develop a notion of instance.
- Compared to (Zhou et al., 2017), the proposed model does not improve much on the counting questions.
- Since the authors have mentioned in the related work, it would also be more convincing if they show experimental results on CL

Conclusion:
- I feel that the motivation is good, but the proposed model is too hand-crafted. Also, key experiments are missing: 1) NMS baseline 2) Comparison with VQA counting work  (Chattopadhyay et al., 2017). Therefore I recommend reject.

References:
- Kipf, T.N., Welling, M., Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.
- Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R. Gated Graph Sequence Neural Networks. ICLR 2016.

Update:
Thank you for the rebuttal. The paper is revised and I saw NMS baseline is added. I understood the reason not to compare with certain related work. The rebuttal is convincing and I decided to increase my rating, because adding the proposed counting module achieve 5% increase in counting accuracy. However, I am a little worried that the proposed model may be hard to reproduce due to its complexity and therefore choose to give a 6.
review_title: Model is too hand-crafted and key experiments missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
 - This paper mainly focuses on a counting problem in visual question answering (VQA) using attention mechanism. The authors propose a differentiable counting component, which explicitly counts the number of objects. Given attention weights and corresponding proposals, the model deduplicates overlapping proposals by eliminating intra-object edges and inter-object edges using graph representation for proposals. In experiments, the effectiveness of proposed model is clearly shown in counting questions on both a synthetic toy dataset and the widely used VQA v2 dataset.

Strengths
 - The proposed model begins with reasonable motivation and shows its effectiveness in experiments clearly. 
 - The architecture of the proposed model looks natural and all components seem to have clear contribution to the model.
 - The proposed model can be easily applied to any VQA model using soft attention. 
 - The paper is well written and the contribution is clear.

Weaknesses
 - Although the proposed model is helpful to model counting information in VQA, it fails to show improvement with respect to a couple of important baselines: prediction from image representation only and from the combination of image representation and attention weights. 
 - Qualitative examples of intermediate values in counting component--adjacency matrix (A), distance matrix (D) and count matrix (C)--need to be presented to show the contribution of each part, especially in the real examples that are not compatible with the strong assumptions in modeling counting component.

Comments
 - It is not clear if the value of count "c" is same with the final answer in counting questions. 


review_title: Counting in VQA

Meta-review: Initially this paper received mixed reviews. After reading the author response, R1 and and R3 recommend acceptance.

R2, who recommended rejecting the paper, did not participate in discussions, did not respond to author explanations, did not respond to AC emails, and did not submit a final recommendation. This AC does not agree with the concerns raised by R2 (e.g. I don't find this model to be unprincipled).

The concerns raised by R1 and R3 were important (especially e.g. comparisons to NMS) and the authors have done a good job adding the required experiments and providing explanations.

Please update the manuscript incorporating all feedback received here, including comparisons reported to the concurrent ICLR submission on counting. 
Decision: Accept (Poster)