Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper does a psychological-computational-combined experiments for context reasoning. The experiment is done by "lift-the-flap" -- masking out a region of interest in the image and let either the human or a convNet based model to guess what it is by checking the context. Both of them are first shown with a blurred image with masked region, and then start to guess by clicking on surrounding regions and unblur them. A lot of baselines are compared and it is shown that the computational model is working well, and the behavior is highly correlated with humans.

+ The paper reads very well, the illustrations and tables are very well done.
+ The experiment itself is interesting that it delves into the context problem directly. It would be interesting to see how it works for objects that are out of context though:
http://people.csail.mit.edu/myungjin/outOfContext.html
+ A like it that a great set of baselines and analysis are done for the paper. It strengthens the paper a lot.

- I think the paper needs to have a baseline for the upper bound as well: what is the accuracy if the region is seen? In other words, what is the performance if no context is needed. It would be interesting to see the gap over there. A baseline could be a region-classification model, e.g., from:
Chen, Xinlei, et al. "Iterative visual reasoning beyond convolutions." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
Just from the results (e.g. Fig 3) it seems our current models are already doing pretty well! (and it is VGG, not the best model yet), but on the other hand it is not clear how such models can help recognize visible objects better -- maybe a lot of the things that the context can offer has already been incorporated in the object pixels itself. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I agree that context is important in some visual recognition tasks. I think this is an ambitious study and find the employed experimental methods interesting. 

However, it is not clear to me what has been actually revealed by this study. 

In the last section, there is a statement “the model adequately predicts human sampling behavior and reaches almost human-level performance in this contextual reasoning task.” I think that at least the first half is overstatement; it is not well validated by the experimental results.

For instance, contrary to the authors’ claim, I do not think the effectiveness of the recurrent connections in ClickNet is sufficiently validated. I think a yet another baseline is missing in the experiments, which is a strategy of clicking points in the periphery of the black box while avoiding (or minimizing) overlaps of the regions deblurred by previous clicks. 

Although ClickNet-RandPrior appears to be close to this strategy, it does not seem to use any constraint of avoiding such region overlap. On the other hand, in the training of ClickNet, a constraint \sum_t\alpha_{ti}=1 is used, which seems to play this very role, i.e., making it possible for ClickNet to avoid the region overlap. Isn’t the good performance of ClickNet fully attributable to this constraint?

The proposed method is to make subjects (or ClickNet) click a series of points in the input image and then deblur the local regions around the points. I suppose this procedure is accumulative, that is, once a local region is deblurred, it will be kept deblurred in the subsequent clicks. Then, I'm not sure if the order of clicking points really matters, whether it is a human subject or the ClickNet. For instance, is there any evidence that a click is dependent on the previous clicks, other than the above behavior of avoiding overlaps?

Additionally, I am somewhat skeptical if a pretrained VGG can really extract useful features from (partially) blurred images, even though it is not trained on blurred images. It is widely known that CNNs for visual recognition tasks are vulnerable to image blur, noise, etc. when they do not exist in the images used for training, which is a kind of domain shift. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper is about context reasoning in the visual recognition. They designed a task called 'lift-the-flap', where the human or the model are given an image with one of the object blacked out. The image is provided in a relative low resolution, and the subjects could choose to click on area to reveal a local window of high-res image. The subject then need to answer what is the object in the blacked out region after a few clicks (ranging from 1 to 8 clicks). The authors has collected human data using mTurk, and also trained a model (named ClickNet) to perform the task. ClickNet has a LSTM memory which can carry the information of previous clicks and use spatial attention mechanism to decide the next click location (just use the argmax of the attention weight.) I find the paper is clearly-written and quite easy to follow. I think this is an interesting paper comparing human attentional performance to a trained computation model (without any human-imposed prior), showing that they behave similarly. However, I am not sure it would be of general interest to the ICLR audience. This may be more suitable for cognitive science conference/journal in my opinion. I have a few suggestions below:

1. In formula 2, the attention weight (before softmax) is  " e_ti = A_h * h_{t−1} + A_a * a_ti " I am surprised there is no term that is h_{t-1} * a_ti, which do the content-based reading that is common in memory model like (DNC) or in language processing (transformer.) Can the author comment on this decision?

2. In result 5.1, the authors titled: "WHAT: IMPORTANCE SAMPLING VIA ATTENTION AND PRIOR INFORMATION". I am not sure I get where is the importance sampling appear in this paragraph. Importance sampling is a specific term, which refers to estimating properties of a particular distribution, while only having samples generated from a different distribution than the distribution of interest. I did not see how that is related to this paragraph. Or, I may have missed it, and would like the authors to elaborate.

3. I could not follow well the discussion in section 5.5 WHEN: ROLE OF RECURRENT CONNECTIONS. Based on the title of the section, I would expect to discuss why the recurrence of LSTM is important. Though, the paragraph is mainly talking about other control models like SVM and HMM, so I don't quite follow it. Also, it is not clear to me the word 'When' is a good choice. I am convinced that the LSTM helps the model carrying information across click, but it is not clearly shown the sequential order is critical (say click location 1, then location 2, then location 3 compared to location 2, location 3, then location 1 matters) Maybe it does? But, it is now shown in the results.

4. The comparison of where the human versus the model click (in Figure 6) is impressive. I wonder if the authors could go even one step further that is to see if the temporal order of the clicks are similar in human vs. model. It may not be the case. However, if it is true, that could make an even stronger case that the model has a similar way to decide attentional location compared to human subjects.

Minor:
1. It is impressive that the authors have considered a wide range of control models. Though, for a conference paper with limited page limit, I don't think it is necessary to describe all these models in the main text ( for the SVM, HMM, and CRF models.)

review_title: Official Blind Review #2

Meta-review: The authors present the task lift-the-flap where an agent (artificial or human) is presented with a blurred image and a hidden item. The agent can de-blur the parts of the image by clicking on it. The authors introduce a model for this task (ClickNet) and they compare this against others.
As reviewers point, this paper presents an interesting set of experiments and analyses. Overall, this type of work can be quite influential as it gives an alternative way to improve our models by unveiling human strategies and using those as inductive biases for our models. That being said, I find the conclusions of this paper quite narrow for the general audience of ICLR (as R2 and R3 also point), as authors look into an artificial task and show ClickNet performs well. But what have we learned beyond that? How do we use these results to improve either our models or our understanding of these models? I believe these are the type of questions that  are missing from the current version of the paper and that if answered would greatly increase its impact and relevance to the ICLR community. At the moment though, I cannot recommend this paper for acceptance.

Decision: Reject