Lift-the-flap: what, where and when for context reasoning
Authors: Mengmi Zhang
Claire Tseng
Karla Montejo
Joseph Kwon
Gabriel Kreiman
Venue:  (2020)
Keywords: contextual reasoning
visual recognition
human behavior
intelligent sampling

Abstract: Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge. Both spatial and temporal contextual information play a critical role in the domain of visual recognition. Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition. The task is to reason about the scene context and infer what a target object hidden behind a flap is in a natural image. To tackle this problem, we first describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy. As a proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-art recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object over multiple clicks. The proposed model achieves human-level contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.

Under review as a conference paper at ICLR 2020
LIFT-THE-FLAP: WHAT, WHERE AND WHEN FOR CONTEXT REASONING
Anonymous authors Paper under double-blind review
ABSTRACT
Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge. Both spatial and temporal contextual information play a critical role in the domain of visual recognition. Here we investigate spatial constraints (what image features provide contextual information and where they are located), and temporal constraints (when different contextual cues matter) for visual recognition. The task is to reason about the scene context and infer what target object hidden behind a flap is in a natural image. To tackle this problem, we first describe online human psychophysies experiment recording active sampling via mouse clicks lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy. As proof of the usefulness of these clicking patterns and visual features, we extend a state-of-the-ar recurrent model capable of attending to salient context regions, dynamically integrating useful information, making inferences, and predicting class label for the target object multiple clicks. The proposed model achieves human- contextual reasoning accuracy, shares human-like sampling behavior and learns interpretable features for contextual reasoning.
INTRODUCTION
The tiny object on the table is probably spoon, not an elephant. Objects do not appear in isolation. Instead, they co-vary with other objects, their sizes and colors usually respect regularities with respect to nearby elements, and objects tend to appear at specific locations within a scene. Humans exploit these contextual associations. Contextual analyses based on the statistical summary object relationships, provide an effective source of information for perceptual inference tasks, such as object detection (Torralba (2003); Park et al. (2010): Hoiem et al. (2005): Torralba et al. (2010); Liu
ett al. (2018b)), scene classification (Gonfaus et al. (2010): Torralba et al. (2005); Yao et al. (2012)
semantic segmentation (Yao et al. (2012)), and visual question answering (Teney et al. (017)
An example of how contextual information is incorporated during object recognition is lift-the-flap books, where a flap covers part of the page. Children make guesses about what is behind the flap based on the context and check their answers by lifting the flap (Figure [a]. Here we investigate what image features matter for contextual reasoning and where those features are with respect to the target object of interest. Furthermore, scene interpretation humans involves sequence of eye movements (Zhang et al. (2018)), each one of these image samples providing additional context to inform interpretation of the contents of the next location. Therefore, we also investigate whether integration of scene information over time ("when" information) matters for context reasoning.
To tackle the problem of contextual reasoning, we introduce the lift-the-flap task and conduct online psychophysics experiments where subjects make mouse clicks while they explore important contextual cues to identify a hidden target (Figure [b). We investigate the contextual reasoning strategies observed from human active clicking patterns. As a proof of concept, propose recurrent attention model (ClickNet), to automatically learn these contextual reasoning strategies. The model guides attention towards regions with informative context, decides where to click on the image, and makes inferences about the target behind the flap. The learnt clicking patterns and predicted class labels by ClickNet share remarkable similarities with human behavior.

---

Under review as a conference paper at ICLR 2020
Required 1/2/4/8 clicks
What is black box:
"trafficlight"
time
(a) Lift-the-flay game
(h) Schematics of Behavioral Experiment
Figure 1: Schematic of the lift-the-flap task and human behavioral experiment. (a) The task requires subjects to capitalize on the scene context a natural image to infer what is behind the black box (the hidden target). The original image (bottom left) reveals the target object ("traffic light"); this image was not shown in the actual experiments. (b) A blurred image with the hidden target presented to the subject. To identify contextual areas of importance surrounding the hidden target, subjects used the computer mouse to click on image a pre-specified number of times (red dots). Upon clicking certain location, circle of fixed radius was revealed high resolution. After the required number of clicks, subjects typed a noun describing the hidden target object. The experiments were conducted online using Amazon Mechanical Turk Turk (2012) on 100 subjects, 50 trials per subject. Figure 3 also shows results for variation of the experiment conducted in the lab while tracking eye movements.
2
RELATED WORKS
2.1 ROLE OF CONTEXT IN HUMAN VISION
Contextual information affects the efficiency of several visual processes (Auckland et al. 2007 Biederman al. 1982, Hollingworth 1998] Bar & Aminofi 2003 Goh et al. [2004]|mmnoot Amiinoff al. 2006), such as object recognition (A (Auckland et 2l.7), object detection (Biederman et al. 1982 Hollingworth| 1998), visual working memory (Friedman 1979 Aminoff et al.] (2066) and visual search (Henderson et al. 1999). Objects appearing in a familiar background can be detected more accurately and processed more quickly than objects appearing in an incongruent scene. Here we focus on what visual features contribute to contextual reasoning, which parts of image regions attract human attention make inferences, and the dynamic sequence of directed sampling needed to idenfity a hidden target.
2.2 ROLE CONTEXT IN COMPUTER VISION
Contextual reasoning about objects and relations is critical to machine vision. In fact, many object recognition studies using natural image datasets such as ImageNet, rely implicitly but strongly on contextual feature regularities Geirhos et al. 20188; Brendel & Bethge (2019). Some works also show that these models can fail when objects are placed in an incongruent context (Beery et al. 2018: Dvornik et al. (2018) Choi et al. (2012)) These studies motivate examining the role of pure context without any object information in situations of complete occlusion lift-the-flap); here provide human and computational benchmarks for object inference using exclusively context
information. Several studies use context to improve object detection Park et al.) 2010 Holem et al,
2005 Torralba et al.
2010 Liu al.] 2018b Chen et all
take the form of 2018b Context
global scene context (oorralba et al. 2010), ground plane estimation (Park et al.) 2010). geometric
context (Hoiem al. 2005), relative location Desai et al. (2011), 3D layout (Lin al. 2013). and
spatial support and geographic information (Divvala et al. 2009). Researchers proposed Conditional
Random Field (CRF) models that reason jointly across multiple computer vision tasks in image labeling and scene classification Gonfaus et al. (2010): Yao et al.] (2012): Ladicky et al. (2010). Context can lead to improved performance in both object detection and semantic segmentation tasks Mottaghi et al. (2014). Neural network architectures incorporating contextual information have been used in object priming (Torralba) 2003), place and object recognition (Wu et al., 2018 Torralba ct al. 2005), object detection (Liu al. 2018b Chen et al. 2018b), and visual question answering (Teney et al. 2017). Here we focus on developing a biologically inspired computational model for contextual reasoning that can automatically and dynamically sample image regions of interest, integrating information in space and time to make inferences about a hidden object. Additionally, compare the model's performance against human behavior in the same task.
2

---

Under review as a conference paper at ICLR 2020
Several interesting approaches have combined graphical models with deep neural networks for structural inference, primarily in structured prediction tasks
Marino et al.] 2016 6 Choi & Savarese
2012 Chen al. 20 Teney et al. 2017 Hu et al. 2016 Battagla et al. [201 Xu et al. 2017
Chen et al. 201 Choi et al. 2012 Hu et al. (2016) designed a structured model to improve
classification performance by leveraging relations among scenes, objects, and their attributes. A
structured inference model is also used in (Choi & Savarese] 2012, Deng et al., (0166) to analyze relations in group activity recognition. Several works, like Structural- Jain al. (2016) and Interaction Net Battaalal et al. (2016), combine spatiotemporal graphs and sequence learning. These works assume full contextual information is available, while our experiment we consider only partial contextual information that is sequentially revealed after each mouse click. DeepLab sends the response at the final layer of a CNN to a CRF model for semantic image segmentation Chen al. (2018)). Subsequently, (Schwing & Urtasun (2015): /Zheng et al. (2015) transformed the CRF model into Recurrent Neural Network. Breaking away from these studies where graph optimization is performed globally, our proposed model selects important visual features using attention mechanism and integrates partial information over multiple steps, which is computationally more efficient and accurate in the current task (Section 5)
3 LIFT-THE-FLAP TASK
3.1 HUMAN BEHAVIORAL EXPERIMENTS
Subjects were presented with a natural image where one object was hidden by rectangular black box and everything else was blurred. They were allowed a fixed number of mouse clicks between and 8, each click revealed part of the image in high resolution. To minimize overlap between clicks, we enforced subjects to click at places at least 10 pixels apart from all previous clicks. After the target number of clicks, they had provide a single word to describe the object hidden behind the black box (Figure[b) The clicking experiments were run on Amazon Mechanical Turk Tuuk (2012) The stimulus set consisted of 573 images from the test set of the MSCOCO Dataset (Lin et al. 2014), spanning 80 object categories. This dataset has been widely used for object recognition and detection studies (Lin et 2014). We constrained the stimulus set to have a uniform distribution of 6 target objects per category. To avoid any potential memory effects, subjects were only exposed to each image once. The trial presentation order was randomized.
3.2 GROUND TRUTH RESPONSES
In contrast to experiments where subjects are forced to perform N-way categorization (c.g.. Tang jet al.] (2018)) here there were no constraints on how subjects describe the hidden object. This probing mechanism
implemented for two reasons: (i) it is difficult for humans memorize 80
object classes and there could be non-uniform memory effects impacting the results; (ii) presenting humans with an 80-choice question could introduce biases in their inference processes.
We could not simply use the 80 category labels to evaluate performance because subjects could use other words or synonyms and we interested in context reasoning rather than language abilities. Therefore, to evaluate humans' performance, we separately collected distribution of ground truth answers for each hidden target by presenting to 10 other subjects, who did not participate in the main task, the same set of images with the target objects highlighted by a bounding box but not hidden. During the lift-the-flay flap task, a response was considered be correct if matched any of the ground truth labels, allowing for plurals and misspellings.
3.3 EVALUATION METRICS
We introduce several evaluation metrics to measure contextual reasoning accuracy and to compare the consistency of mouse clicking patterns between humans and computational models. We evaluated ClickNet on the MSCOCO Dataset using the typical classification accuracy measure. In Fig 5b we report top-1 classification accuracy a function of context-object ratio. The context-object ratio is defined as the total area of the image excluding the hidden target divided by the hidden target object size. For example, context-object ratio of implies that the size of the black box and the size of the contextual information is the same (see Figure for example images with different context-object ratios).
3

---

Under review as a conference paper at ICLR 2020
Prediction
Click Location
Feature Extraction VGG16
Attention Prediction
Click Location
Predicted
Class Labels
Feature Extraction using VGG26
"trafficlight"
Predicted Class Labels "bird"
LSTM
11
Figure 2: Architecture overview of the ClickNet model. The diagram depicts the iterative modular steps carried out by ClickNet for contextual reasoning over multiple clicks in the lift-the-flap task (Figure lb). ClickNet consists of 3 main modules: feature extraction, attention, and recurrent memory. For illustrative purposes, only the first and second clicks in a trial are shown here. ClickNet performs feature extraction using the VGGL6 network pre-trained on ImageNet and produces feature maps ag- Conditioned on the hidden state fgg and feature maps ag. ClickNet produces an attention map Cgg which is used to select the next click location (red dots) and to modulate the feature maps for contextual reasoning (Figure(ST). The recurrent network the LSTM module (Figure S2) integrates over time the attentionally modulated feature maps Zo. and outputs predicted class label after each click (here "bird" and "trafficlight" before the 1st and 2nd clicks). After the first click, the input image gets updated with parts at clicked locations revealed in high resolution. These three modular steps repeat until the specified number of clicks have been made. See supplementary figures SI and S2 for implementation details on the attention and LSTM modules, respectively.
To measure the degree of spatiotemporal consistency between two mouse clicking patterns (human versus human; human versus computational models), we introduced two measures. First, we computed the minimum Euclidean distance between the sequence of clicks each trial, regardless of order. The smaller the median in the distribution of distances, the more similar the two mouse clicking patterns are. Second, we introduced a click sequence score, as originally defined in evaluating eye fixation sequence consistency Madsen et al.) 2012 Zhang et al. (018). Here, the same metrics to compare the spatial-tempont consistency between click sequences. The larger the click sequence score, the more similar the sequences are.
4 CLICKNET ARCHITECTURE
We propose recurrent neural network for context reasoning (ClickNet), extending previous work on image captioning (Xu et al. 2015). ClickNet integrates altention-modulated context information multiple clicks, makes decision about the next click location based on the attention map, and infers the class label of the hidden target after every click (Figure 2
As in the human psychophysics experiment (Figure lb), ClickNet is first presented with blurred image Ios which the original image I with uniform gaussian blur and where the target object covered by black box. ClickNet makes the first attempt predict class label yo out of pre-defined set C' object classes and decides its first click location mj- In every trial, over a series of T' clicks, the input image 1e ClickNet gets updated with circular regions of constant radius R centered at all previous click locations M = {m.,...TTT, J. revealed in its original resolution in I. The black box is constant and none of its content is ever revealed, even if the model opts to click within the box or if the circle centered on the click encompasses part of the box.
4.1 CONVOLUTIONAL FEATURE EXTRACTION
At each time where & E {0, ..., T). ClickNet takes I, as input and uses a feed-forward convolutional neural network to extract feature maps or, We use the VGG16 network (Simonyan & Zisserman 2014], pre-trained on ImageNet (Deng et al., 2009). Consistent with previous works showing that these feed-forward convolutional neural networks for visual recognition tasks are vulnerable to domain shift such as image blur and noise when those transformations do not exist in the images
used for training (Liu et al. 2018a Dodge & Karam 2016)), observe that it is necessary to
4

---

Under review as a conference paper at ICLR 2020
fine-tune the pre-trained VGG16 network for lift-the-flap task (see implementation details in 4.4 and focus on specific parts of the image and select features at those locations, we have to preserve the spatial organization of features; thus, ClickNet uses the output feature maps at the last convolution layer of VGG16.
Section A
A feature vector au of dimension D represents the part of the image I at location i, where i 1,.,, L and W x and W and are the width and height, respectively, of the feature map:
= a aui E IR
4.2 ATTENTIONAL MODULATION
We use a "soft-attention" mechanism as introduced by Ba et al. (2014) to compute "the context gist" Z Tt (Figure STD. For each location in a,, the attention mechanism generates positive scalar ati representing the relative importance of the feature vector au for context reasoning. This relative importance at depends on the feature vectors ati. combined with the hidden state the previous step h 1 of recurrent network described below.
exp(ess)
Cti A + Agati Otgi E exp(ee)
weight matrices initialized randomly and to be learnt. Instead of addition, an alternative is to use element-wise multiplication: he o Ag Empirically, did not observe any performance difference with such multiplicative term (see Section A Because not all attended regions might be useful for context reasoning, the soft attention module also predicts a gating vector 3, from the previous hidden state 14-1, such that 3. determines how much the current observation contributes to the context vector at each location: 3, = (W where Wa Lxx is weight matrix and each element B1s in 3r is gating scalar at location i. As also noted by Xu et al. (2015), 3e helps put more emphasis on the salient objects in the images. Once the attention map a and the gating scale 3, are computed, the model applies the "soft-attention" mechanism to compute 2, by summing over all the I regions in the image:
where An 1x and Aa RIXD
2t
Buogan
(3)
The next click location m441 corresponded to the maximum on the attention map:
m++1 arg max Cres
(4)
The attentional module is smooth and differentiable and ClickNet can learn all the weight matrices in an end-to-end fashion via back- poockpropagation.
4.3 RECURRENT CONNECTIONS USING LONG SHORT-TERM MEMORY (LSTM)
We use long short-term memory (LSTM) network to output predicted class label yt based on the previous hidden state h and the context gist vector 2, for I, (Figure S2). Our implementation of LSTM closely follows Zaremba et al. 220 where T., IR + R. defines a linear transformation with learnable parameters. The variables es It, fe, Ct. Ot. h, represent the input, forget, memory, output and hidden state the LSTM respectively:
i
fr Og
TD+En (Ze, lt-1)
(5)
gt
tanh
= f. O Ct +
gt. h, = O tanh(c))
(6)
where n is the dimensionality LSTM, a is the logistic sigmoid activation, and O indicates element-wise multiplication.
5

---

Under review as a conference paper at ICLR 2020
To cue ClickNet about the location of the hidden target, initialize the memory state co and hidden state ho of the LSTM based on a binary mask that contains zeros everywhere and ones in the hidden target location. Specifically, co and ho predicted by average of all feature vectors ag all I. locations with two separate linear transformations Wo E RaxD and Who E RuxD:
Σ aoi): ho Who 1
co Woo(
aoi)
To predict the class label y/ of the hidden target. the LSTM computes classification vector where each entry denotes a class probability given the hidden state:
J/+ = arg max p(y.), p(ye) x L
(8)
where Ln E RCxn is a matrix of learnt parameters initialized randomly.
4.4 TRAINING AND IMPLEMENTATION DETAILS
We trained ClickNet end-to-end by minimizing the cross entropy loss between the predicted label Ye at each time step t and the ground truth label r. In contrast with previous work where there regularization term E, Qei 1 in the loss function
encourage the model acquire as much information all over the image by exploration (Xu et al. (0155). we did not find significant performance increase in the lift-the-flap task when adding such term (see Section A Hence, let ClickNet freely explore the image without any such constraints. Since we do not impose any constraints on the next click locations, ClickNet might make decisions to click at the previously clicked locations.. The loss function is definef by:
LOSS = Σ (- log( (((nl)))))
We used all images from the MSCOCO training set for training and validating all models. On every training image, each object can be blocked out as the hidden target. The input image size to ClickNNtt was 400 x 400 pixels. We used Gaussian filter of size 51 x 51 with variance 64 pixels to blur the images. The radius A of the circular region revealed by each click was 55 pixels. As in the human psychophysics experiments (Fig 1b). in each trial, we set the total number of time steps T 8 for training ClickNet (ClickNet predicts the label after the 1st click at T = 1). The dimension of the LSTM module was 512. The feature maps extracted from the last convolution layer was of size 2048 x 28 x 28, and the total number of locations was L = 28 x 28 = 784. The Adam optimizer (Kingma & Ba 2014) was used with a learning rate of 10-1 to fine-tune the VGG16 network, and learning rate of 4 x 10 to train the attentional module and the LSTM module. The network was developed in Pytorch, based on (Xu et al. 2015). All source code for our proposed architecture, and the data from the psychophysics experiments will be released publicly upon publication.
4.5 VARIATIONS OF PROPOSED NETWORK ARCHITECTURE AND COMPARATIVE METHODS
Previous work has shown that possible to augment vision systems with human perceptual supervision many difficult computer vision tasks, such Vondrick al. (2015); Kovashka et al. (2016). One central goal in our study is to investigate what, where, and when matter for human contextual reasoning in the lift-the- flap game and whether these factors could help improve current machine learning algorithms. We now introduce two variations of ClickNet with human inputs at the testing stage:
Instead of clicking at the location with highest activation value the attention map predicted by ClickNet, we substitute the input with human clicking images.
ClickNett- -end-edr.....rr.. We observe strong spatial bias in the human clicking patterns where most of the clicks tend to be nearby the hidden target (see Sec 5.4. To test whether this is sufficient to account for human behavior, we generated random clicks surrounding the black bounding box and used the resulting images with strong spatial clicking prior as inputs to ClickNet. We set the same criterion to consider two clicks as overlapping as in the human behavioral experiments; the probability that two clicks overlap in this model is 0.0065.

---

Under review as a conference paper at ICLR 2020
(a) 80
(b) 80
HumanClick-1
HumanClick-B8
Human Fixationn2
HumanFikation- Fixation-6600
60
60
Accuracy (%)
Accuracy (%)
40
40
20
ClfckNet- CliokNeet-hmanaaii humanolick
CliokNet
OlicNNe RanoPrior
VGG Attention humandllck
VGG Attention
VGG humanalick
VGG Random
ClickNet CliekNet-human humanclick
CliokNel
ClickNet RandPror
VGG Attention humanalick
VGG Attention
VGG humanclick
VGG Random
Figure 3: Contextual reasoning accuracy of humans and models. Performance for humans (horizontal lines) and models (bars) for (a) I click (gray), and (b) 8 clicks (black) (Fig (S4 shows results for and 4 clicks, and additional comparison models). Section 3.3 defines the evaluation metric and Section 4.5 describe each model. Error bars denote SEM across images.
Example 1
Example 2
Humans
ClickNet
Humans
ClickNet
bird
Kite
trafticlignt
traffic light
Click=1
bird
bird
llghht
traffic light
Click= 8
Figure 4: Example visualization for humans and ClickNet. Two example trials (first four columns example last four columns is example 2), either with 1 click (rows 1) or 8 clicks (rows 2), for one human (columns 1, 2, 5, 6) or ClickNet (column 3, 4, 7, 8) (Fig S3 shows results for and clicks). Red dots denote clicked locations. Top-left corner shows output labels after the required number of clicks. Column 2 and 6 show the mouse click maps aggregated over subjects. Brighter regions denote more mouse clicks (see scale bar on right). Column and 8 show the attentional map predicted by ClickNet. Brighter regions denote higher attentional values.
To gauge how much contextual information helps object recognition, it is interesting to isolate the effect of context and object, analyze them separately and then study both combined. The lift-the-flap task provides benchmark study the contextual reasoning problem alone. In addition, we evaluate ClickNet when both the object region and the context are revealed to ClickNet (an upper bound) and when only the object region (the tightest bounding box encompassing the object) revealed to ClickNet (Section
D
To study the role of attention and recurrent connections, we introduced two ablated models.
Variations of VGG16. One intuitive way of solving the context reasoning problem is use feed-forware object recognition network pre-trained on ImageNet, Zisserman, 2014), and fine-tune to classify the hidden target on MSCOCO dataset. During training, the input to the network was image where one object the image was randomly covered with a black bounding box. We tested the performance of this alternative model on the 573 images selected for human psychophysics experiments with different input variations: human clicking images (VGG-lumanclick) the blurred images (VGG-Blur), the full-resolution images (VGG-Fullren) and images with random clicks (VGG-Random).
VGG16 (Simonyan &

---

Under review as a conference paper at ICLR 2020
baseball bat
Ratio 3
baseball glove
03
skateboard
suriboard
ennis racket
Ratio 20
Actual
buttle
Accuracy
wine glass
cup
fork
knile
Ratio
basebyel
surthaad
tarms rackat
buttle
glass cilt call
gode
Predicted Labels
Cantexe-abject ratio (log scale)
(a) Confusion matrix from ClickNet
(b) Accuracy versus context-object ratio
Figure 5: Improvement in contextual reasoning accuracy with context-object ratio and patterns mistakes. (a) Partial view for illustrative purposes of confusion matrix showing the mistakes made by ClickNet among 10 of the 80 object categories in MSCOCO (Fig [S5 shows the complete confusion matrix with all 80 categories). The element in row i, column denotes the probability that ClickNet predicted label j while the ground truth label was i (see scale bar right). The sum of probabilities Human (circle) and model (triangle) accuracy for 1 click (gray) and 8 clicks (black) as function of context-object ratio, shown in logarithmic scale (Sec. 3.3). Right: example images with different context-object ratios. Only the images on the first column were shown (the second column is shown here for illustrative purposes only). Error bars indicate SEM across images.
row in the full confusion matrix (but not here) equals 1. (b)
PGG-Attention. Previous work has demonstrated the efficiency of attention in computer vision tasks (Nguyen et al., 2018), such as question answering and image captioning Xu et al. (2015). To study the effect of attention in contextual reasoning. added attention module the end of VGG16. To make the complexity of the architecture comparable with ClickNet, we added the same number of fully connected layers in the LSTM module. As in ClickNet, we used the location with the highest activation value on the attention map to predict the next click. VGG-Attention takes the updated image as input and iteratively predicts the hidden target label. In contrast to ClickNet, the network is feed-forwart and there is no incorporation of past information integrated over clicks. We also test VGG- Attentiton with human clicks VGGGAAtention--i wattenttion-ummanlieeeee and randomly generated click locations with strong spatial priors
Cel--Attention-Raadie
We also considered several competitive baselines and existing methods of modeling temporal dynamics. These include Human-fixations SVM-category, SVMM-ategory-inntance Hidden Markoy Model (HMM) and DDepLab-Conditionn Random Field (CRF). See Section for these comparison methods.
RESULTS
5.1 WHAT: REGION SELECTION VIA ATTENTION AND PRIOR INFORMATION
Subjects inferred the identity of the hidden target object with 36.7 + 0.9% accuracy after click (Fig 3 gray horizontal line). Performance showed small, but significant improvement when allowing subjects 8 clicks, reaching 48.4 0.9% (Fig 3 black horizontal line, p 10-9 9
two-tailed t-test, t=-6, df=2593). In-lab experiments corroborated these results showing accuracies of 48.1 0.9% after 200 ms exposure and 56.3 0.9% after 1,600 ms exposure to the images.
The same images that subjects saw were used to evaluate ClickNet (Figure 3. The ClickNet model showed a close approximation human performance in the mturk experiments, reaching a top-1 classification performance of 34.9 + 0.9% for 1 click and 48.87 + 0.9% for 8 clicks. In both cases, performance was only slightly lower than human performance in click (p 0.47, two-tailed t-test, t=0.73, df=1828) and clicks (p 0.86, two-tailed t-test, t=0.17 17, 1909). Performance for intermediate numbers of clicks shown in (Figure S4). For all the computational models, random guessing would yield accuracy 1.25%.
The worst performing model, VGG-Random, yielded performance above chance levels, emphasizing that small amounts of high-resolution contextual data at arbitrary locations can help solve the problem. Yet. VGG-Random was well below ClickNet's performance (p < 10-15 two-tailed t-test, t=21, df=3028). Adding attention the model (VGG-Attention) yielded only
8

---

Under review as a conference paper at ICLR 2020
minimal improvement. An important ingredient missing in VGG-Random and VGG-Attention is the informed location of the clicks. Humans and ClickNet do not sample the image randomly, but rather explore informative locations. Figure 4 shows qualitative examples of clicking patterss from humans and ClickNet. Both humans and ClickNet attend to salient regions on the images. For example, clicks often occur near birds in the first example and near traffic lights in the second example. Accordingly, substituting the random clicks for the human clicks into the VGG models (VGG-humanclick and substituting the ClickNet clicks with random clicks leads to large drop in performance when there only 1 click, even when we artificially try to boost performance by constraining the clicks to be near the hidden target object Ceiicket-RannPrionn
yielded large performance boost. Conversely,
This effect is also evident with 2 clicks and clicks (Fig. (5). but disappears with 8 clicks because there already a lot of high resolution information in the image surrounding the hidden target, and ClickNet can integrate information over time to capitalize on it. Interestingly, the ClickNet sampling clicks are sufficiently close to human clicks that substituting the ClickNet clicks with human clicks does not improve performance
We considered several other comparative models (Fig. S4 Interestingly, using just few clicks, ClickNet reaches performance that is essentially equivalent to that of VGG using a full resolution version of the entire image. Other comparative models (VGG-Blur, SVM-category, SYM-categoyy-instaness HMM, Dbeplab-CRFF) showed above chance performance but their accuracies were well below ClickNet.
5.2 WHAT: THE MORE, THE MERRIER
To investigate how much context information is needed to enhance recognition, we evaluated accuracy as a function of context-object ratio (Fig 55 Sec 3.3). Images with higher context-object ratio contained more context information for inference, and yielded higher accuracy both for humans and models. Similarly, accuracy improved with increasing numbers of clicks (Fig and Fig 5b).
It is not just the quantity of context, but also the specific quality of contextual information that matters. In the real world, objects do not tend to appear in isolation but rather they co-vary with other objects. As ClickNet explores more regions on the image, it integrates information at previous clicked locations and learns associations of objects. The pattern of mistakes made by ClickNet is indicative of those associations (Fig 5a and Fig (5)). ClickNet often makes "reasonable" wrong guesses when there is ambiguity in context reasoning, as humans do. For example, knife tends to be associated and therefore confused with spoon, fork, and wine glass, but knife seldom co-occurs with baseball bat or skateboard in these images.
5.3
W WHERE: CONSISTRNCY OF HUMAN AND MODEL CLICKS
We hinted at the similarity in the clicking patterns between humans and ClickNet based on the accuracy of the ClickNet-humanclie
and RainkNet-RanPPrior models in Fig 3 To more directly assess whether ClickNet learned to sample the image to gather information about areas of contextual relevance, we directly quantified the similarity in clicking patterns (Figurelßand S6). To interpret the distance between human and model clicks, we computed the degree of human-human consistency in the clicking patterns. The clicking patterns of ClickNet were overall similar to those made by humans. The model clicks were still different from the consistency between two humans (8 clicks:
< 10 15 two-tailed t-test, t=36, df=29542); yet, the model clicks were more similar to human clicks than random clicks (8 clicks: p < 10-15 two-tailed t-test, t=44, df=35310).
5.4 WHERE: TENDENCY OF CLICKING NEARBY THE TARGET
There was a strong spatial bias towards clicking near the target for both humans and ClickNet (e.g.. Figure 4. To quantify this spatial bias, we computed the Euclidean distance between the clicked locations and the center of the bounding box. normalized by the diagonal of the bounding box (Figure (ic) Humans tended to click within one diagonal distance of the target box. Interestingly, although ClickNet does not take any human supervision during training, ClickNet still learned to capture the tendency of clicking near the target. We asked whether this spatial bias in sampling behavior is sufficient to explain performance in this task in a modified version of ClickNet. We removed the clicks dictated by the attention module and instead forced the clicks to be randomly Both distributed while still respecting the spatial distribution in Fig [cc| by 28.3% and 7% over all click ClickNet and BacckNet-humanllic surpassed conditions respectively (Figure 3) Similar results were obtained when using only the VGG

---

Under review as a conference paper at ICLR 2020
Normaloed Mormol Euclidion Distncee
Normalzed MormolozedEcllee Euclidkan Distance
0.0
10
0.7
Normalizee Distance
0.6
0.6
0.5
0.5
0.4
0.4
0.8
0.5
0.2
0.2
0.1
0.1
Homan-
Human- Moddl
Hummnn Kandom
Model Randoi
Humen- Human
Human- Modda
Human Mandom
Madal Randoin
Checsee
Wandrrm
(a) Click-1 Consistency
(b) Click-8 Consistency
(c) Spatial distribution
Figure 6: Model click locations were similar to human sampling, (a-b) Consistency of click patterns between human subjects (human-human), consistency between humans and model (human-model), consistency between humans and random (human-random) and consistency between models and random (model-random) for click (a) and 8 clicks (b) measured by the distribution of normalized Euclidean distances with respect to the diagonal of the image between any pairs of clicks by humans and ClickNet or random clicks. In each trial, we permute the sequence of mouse clicks between pairs of human and model clicks such that their sum of Euclidean distance minimized across all clicks. The white circles denote the median of the distribution and the light grey bar denote the 1st and 3rd quartiles. (h) Euclidean Distance between click locations and center of the hidden target bounding box normalized by the diagonal of the hidden target bounding box.
architecture: VGG-Attention- -tt--ttentioo--hmmnliikkk was 16% better than Cec-AAteeniionRanndPee (Fig (S4) Therefore, the spatial bias in clicking behavior cannot account for performance: sampling for context reasoning involves more than clicking near the target.
5.5 WHEN: TEMPORAL INTEGRATION HELPS RECOGNITION
Several lines of evidence support the importance of the recurrent network in the LSTM module in ClickNet. ClickNet outperformed the competitive baselines and state-of-the-a comparative methods to make inferences (Figure 3 and Fig S4). We tested whether the co-occurrence of object categories the number of objects per category present in the image would be sufficient for context reasoning (SVM-category and fl-caaegory-instaneeee
Even though these alternative models were exposed to full contextual information the image and assumed perfect labeling of all objects in the image (except for the hidden target object), there was still a large overall performance drop in performance with respect ClickNet. Moreover, graphical models for inference, such as Hidden Markov Model and Deepl_ab ab with Conditional Random Field (Sec (45) failed to reach ClickNet's accuracy in this task (Fig (4). The ablation studies eliminating the LSTM module further support the role of integrating information over multiple clicks in this task, as evidenced by the observation that ClickNet outperformed the VGG-Attention model.
DISCUSSION
Here we quantitatively studied the role contextual information in visual recognition in human observers and computational models in task that involved inferring the identity of a hidden target object. Context influenced recognition based on the amount of context, the specific location of contextual cues, and the dynamic integration of salient visual features. We introduced a recurrent neural network model that combines a feed-forward visual stream module that extracts image features in dynamic fashion, combined with an attention module prioritize different image locations and select the next click location, and recurrent LSTM module that integrates information over time and produces a label for the hidden object. Surprisingly, even though the model lacks the expertise that humans have in interacting with objects in their context, the model approximates human sampling behavior (Click consistency in spatial domains (Euclidean distance distribution and spatial bias in Figure 6 and temporal domains (click sequence similarity score in Figure 56) and reaches almost human-Teve performance in this contextual reasoning task (contextual reasoning accuracy in Figure g and Figure 5b): and reaches almost human-level performance in this contextual reasoning task. The model opens the doors to examine more complex form of reasoning about scenes and how integrate sequential sampling with prior knowledge.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Elissa Aminoff, Nurit Gronau, and Moshe Bar. The parahippocampal cortex mediates spatial and nonspatial associations. Cerebral Cortex, 17(7):14933113 2006.
Mark E Auckland, Kyle R Cave, and Nick Donnelly. Nontarget objects can influence perceptual processes during object recognition. Psychonomic bulletin & review, 14(2):332-337, 2007.
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint arXiv: 1412.775,, 2014.
Moshe Bar and Elissa Aminoff. Cortical analysis of visual context. Neuron, 38(2):347-358, 2003.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp.
2016.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 456-473, 2018.
Irving Biederman, Robert Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology 14(2):143-177, 1982.
Phil Blunsom. Hidden markov models. Lecture notes, August, 15(18-19):48, 2004.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-tocal-feature models works surprisingly well on imagenet. arXiv preprint arXiv:/904.00760, 2019.
Blang-Chiet Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IFEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2018a.
Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinav Gupta. Iterative visual reasoning beyond convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7239,7248, 2018b.
Myung Jin Choi, Antonio Torralba, and Alan S Willsky. Context models and out-of-context objects. Pattern Recognition Letters, 33(7):853-862, 2012.
Wongun Choi and Silvio Savarese. A unified framework for multi-target tracking and collective activity recognition. In European Conference Computer Vision, pp. 215-230. Springer, 2012.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Zhiwei Deng, Arash Vahdat, Hexiang Hu, and Greg Mori. Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4772-4781, 2016.
Chaitanya Desai, Deva Ramanan, and Charless C Fowlkes. Discriminative models for multi-class object layout. International journal of computer vision, 95(1):1-12, 2011.
Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study of context in object detection. In Computer Vision and Pattern Recognition, 2009, CVPR 2009. IEEE Conference on, pp. 1271-1278. IEEE, 2009.

---

Under review as a conference paper at ICLR 2020
Samuel Dodge and Lina Karam. Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2016.
Nikita Dvorikk, Julien Mairal, and Cordelia Schmid. Modeling visual context is key to augmenting object detection datasets. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 364-380, 2018.
Alinda Friedman. Framing pictures: The role of knowledge automatized encoding and memory for gist. Journal of experimental psychology: General, 108(3):316, 1979.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture: increasing shape bias improves accuracy and robustness. arXiv preprint IrXiv:/811.12255. 2018.
Joshua OS Goh, Soon Chun Siong, Denise Park, Angela Gutchess, Andy Hebrank, and Michael WI. Chee. Cortical areas involved in object, background, and object-background processing revealed with functional magnetic resonance adaptation. Journal of Neuroscience, 24(45):10223-102238 2004.
Josep M Gonfaus, Xavier Boix, Joost Van de Weijer, Andrew D Bagdanov, Joan Serrat, and Jordi Gonzalez. Harmony potentials for joint classification and segmentation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp- 3280-3287. IREE, 2010.
John M Henderson, Phillip A Weeks Jr, and Andrew Hollingworth. The effects of semantic consistency on eye movements during complex scene viewing. psychology: Human perception and performance, 25(1):210 1999.
Journal of experimental
Derek Hoiem, Alexei A Efros, and Martial Hebert. Geometric context from a single image. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, volume 1, pp- 654-661. IEEE, 2005.
Andrew Hollingworth. Does consistent scene context facilitate object perception? Journal of Experimental Psychology: General, 127(4):398, 1998.
Hexiang Hu, Guang-Tong Zhou, Zhiwei Deng, Zicheng Liao, and Greg Mori. Learning structured inference neural networks with label relations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2960-2968, 2016.
Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5308-5317, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69988 2014.
Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman, et al. Crowdsourcing in computer vision. Foundations and Trends in computer graphics and Vision, 10(3):177-243, 2016.
Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip HS Torr Graph cut based inference with co-occurrence statistics. In European Conference on Computer Vision, pp. 239-253. Springer, 2010.
Dahua Lin, Sanja Fidler, and Raquel Urtasun. Holistic scene understanding for 3d object detection with rgbd cameras. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1417-1424, 2013.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Mengchen Liu, Shixia Liu, Hang Su, Kelei Cao, and Jun Zhu. Analyzing the noise robustness of deep neural networks. arXiv preprint 1810.0381 2018a.
12

---

Under review as a conference paper at ICLR 2020
Yong Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Structure inference net: Object detection using scene-level context and instance-leve relationships. In Proceedings of the IEEE Conference Computer Viston and Pattern Recognition, pp. 6985 0985-6994, 2018b.
Adrian Madsen, Adam Larson, Lester Loschky, and N Sanjay Rebello. Using scanmatch scores to understand differences in eye movements between correct and incorrect solvers on physics problems. In Proceedings of the Symposium Eye Tracking Research and Applications, pp- 193-196. ACM, 2012.
Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. arXiv preprint arXiv:/612.04888 2016.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whor Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 891-898, 2014.
Tam V Nguyen, Qi Zhao, and Shuicheng Yan. Attentive systems: A survey. International Journal of Computer Vision,
2018.
Dennis Park, Deva Ramanan, and Charless Fowlkes. Multiresolution models for object detection. In European conference on computer vision, pp. 241-254. Springer, 2010.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: 02767, 2018.
Alexander G Schwing and Raquel Urtasun. Fully connected deep structured networks. arXiv preprint arXiv: 1eXivv:503..235/ 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Hanlin Tang, Martin Schrimpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 155035)8835-88400 2018.
Damien Teney, Lingqiao Liu, and Anton den Hengel. Graph-structured representations for visual question answering. arXiv preprint, 2017.
Torralba, Murphy, and WT Freeman. Using the forest to see the trees: Ob-ject recognition in contex. Comm. the ACM, 2010.
Antonio Torralba. Contextual priming for object detection. International jourraationaauuuralll of computer vision, 53(2):169-191, 2003.
Antonio Torralba, Kevin P Murphy, and William Freeman. Contextual models for object detection using boosted random fields. In Advances in neural information processing systems, pp. 1401-1408, 2005.
Amazon Mechanical Turk. Amazon mechanical turk. Retrieved August, 17:2012, 2012.
Carl Vondrick, Hamed Pirsiavash, Aude Oliva, and Antonio Torralba. Learning visual biases from human imagination. In Advances in neural information processing systems, pp. 289-297, 2015.
Kevin Wu, Eric Wu, and Gabriel Kreiman. Learning scene gist with convolutional neural networks to improve object recognition. In Information Sciences and Systems (CISS), 2018 52nd Annual Conference on, pp. 1-6. IEEE, 2018.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-20511 2015.
13

---

Under review as a conference paper at ICLR 2020
Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the a whole: Joint object detection, scene classification and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 702-709. IEEE, 2012.
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv: (40992399, 2014.
Mengmi Zhang. Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman, Finding any waldo with zero-shot invariant and efficient visual search. Nature communications, 9(1):373,, 2018.
Shuai Zheng, Sadeep Jayasumana, Bernardino Nomera-Paredess Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference computer vision, pp- 1529-1537, 2015.
14

---

Under review as a conference paper at ICLR 2020
APPENDIX
We provide supplementary figures and materials here. All labels in supplementary figures and tables pre-fixed with letter S in front.
A INTRODUCTION TO ADDITIONAL ABLATED MODELS AND CONTROLS
ClickNet-noFineTune As explained in Section there is domain shift problem in foed-forward convolutional neural networks for visual recognition tasks when trained using one type of images (e.g., without blurring) and tested using other types of images (blurred). To quantify this effect, here we introduce ClickNet-noFinee Tune where the weights of the feature extractor module were loaded from the VGG16 network pre-trained ImageNet and were fixed during training. The results reported in Figure S4 There was performance drop of around 13% with 1 click and 15.1% % with 8 clicks, compared with ClickNet.
ClickNet-WithAlpphee WithAlphaaLoss. Previous work included an additional alpha loss term in equation 9 ct al. 2015)). To study the effect of this alpha loss term as explained in Section we introduced ClickNet- WichNee-WitsAlpaal.oo... The updated loss function becomes:
Xu
T
LOSS = (- Log(P(v))))) +
(1 C1- Σ an))
(10)
We chose the hyperparameter X same. The result is reported in Figure S4 There was performance drop of around 2% with click
in this equation. The other training specifications remain the
and 4% with 8 clicks compared with ClicNNe. Results presented throughout the main text do not include this alpha term.
CicckNetAAtteMMulti In Section we additively combined ht-1 and asj. An alternative way of computing attention is to combine h and asj via multiplication. As mentioned in Section 44 we
introduced MeickNet-AttenMen and empirically test this possibility. The attention computation in Equation was updated follows:
EN = AAh.- + Aga0 + An o Agati
(11)
where denotes element wise multiplication. The results are reported in Figure S4 There was a performance drop of around 1% with 1 click and 4% with 8 clicks compared with ClickNet.
B INTRODUCTION TO COMPETITIVE BASELINES AND EXISTING METHODS
Human-fixations. We were concerned about the variable viewing conditions in the MTTurk experiments. Therefore, we conducted in-lab psychophysics measurements as a benchmark. In the in-lab experiment, after 500 ms fixation, a bounding box with a fixation cross in the center presented for 1,000 ms indicated the target position and cued subjects to attend to the hidden target location. To ensure that in- subjects paid attention to the hidden target location, we recorded their eye movements using an EyeLink D1000 system (SR Research, Canada). The image with the black box was shown for 200, 400, 800, or 1600 ms. Subjects freely moved their eyes; after stimulus offset, subjects said a single noun describing what the hidden target was. We recruited 4 naive subjects (22 to 24 years old, 2 female), each one participating in 573 trials.
SVM-category. To study the effect of object co-occurrences, we used a binary vector of size 1 CxC as input to classifier, where the ith entry is if there was an object from category ith in the image and 0 otherwise. We assume that the model had perfect information about all the object labels (all objects were visible except for the hidden target). A multi-class support vector machine (SVM) classifier was used to predict the hidden target based on this vector.
Extending SVM-category, we constructed a vector of size 1 x C where the ith entry contained the number n of instances of the ith category in the image. A multi-class SVM classifier was used to predict the hidden target based on this vector.
U'M-category-instanc
Hidden Markoy Model (HMM). To study the temporal dynamics over multiple clicks, considered a Hidden Markov Model where we used all the training images in MSCOCO dataset to calculate the co-occurence matrix of size C x C the transition probability matrix. We use
15

---

Under review as a conference paper at ICLR 2020
normalized uniform vector of size 1 x C' as the initial probability. We fine-tuned VGG16 on the MSCOCO dataset and used it for classifying the cropped region at the human clicked locations where the classification vector contributes to emission matrix. The Viterbi dccoding algorithm Blunsom (004) was used for making inferences about the hidden target.
Random Field (CRF) One interesting solution to reason about the hidden target is run sfate-of-the-arr semantic segmentation algorithms and use majority voting on the predicted labels over all pixels in the bounding box. We used the instantiation in DeepLab-CRF (Chen et 2017).
CrepLab-Condition
C UPPER BOUND: PERFORMANCE INCLUDING OBJECT INFORMATION
Our paper focuses on contextual reasoning with the target object completely occluded. It is interesting provide upper bound for performance where the object regions are revealed to ClickNet using the same images (CickNet-OhjRRvealed. The setup is as in the lift-the-flap problem but the initial condition is different: ClickNet-ObjRevealed is first presented with the tightest bounding box of the target object with the target revealed in high resolution and context blurred, and then the model has click on the image in sequential manner and all the previous clicked regions are deblurred. We fine-tuned the original ClickNet on this task and tested it on the same set of images but with the target objects revealed. The results are reported in Figure S4 With click, ClickNet-ObjRevene (8 clicks), the accuracy of Objjevee-d increased to 71.7 + 0.84 %.
showed an accuracy of 66.1 + 0.88 %. As more context IS revealed
In addition, we include another baseline ClickNet-ObjOnly where only the object is presented to CliekNet-ObjOnly in high resolution from the beginning and there is no context at all. In this
case, the clicks have no relevance. We report the performance in Figure S4 With one click, Rejeeeled showed an accuracy of 47.3 + 0.93 % and with 8 clicks the accuracy was 47.2 0.93 %
As an additional upper bound comparison point we considered sfate-of-the-art object detection algorithms. In the literature, there are many works relying context for object detection Chen
et al. 2018b and Redmon & Farhadi 2018 Here, we tested YOLO3 Redmon & Farhadi (018) TOLO3Reedmon on these two cases (object revealed and object only). Since there no recurrent connections in
YOLO3, the reported results below assume YOLO3 can see full-resolution, i.e. no-blurring. In
the object-revealed case, the accuracy was 65 + 2 % and in the object-only case, the accuracy was
44 + 2%.
There are several interesting observations from these upper bound measurements.
(i) As expected, revealing the object leads to better performance than the lift-the-flap condition with hidden object. Yet, remarkably, contextual information above was essentially equivalent object information alone (compare Figure 3 with the numbers above). Furthermore, revealing the object only increased performance by about 10-15 % with respect the context only condition.
(ii) The accuracy of ClicKNet-OjjctRevvae important role of context in object recognition even when the object is shown.
increases over clicks, which further confirms the
(iii) BhicKNet-OjjecReevalle and ClickNet-ObjeetOnly slightly outperform YOLO3 in the object-revealed and object-only cases.
(iv) The summation of accuracies of ClickNet:ObjectOnly and ClickNet is not equal to Objeceelbbeedeeeeaeeed which shows that the combination of context and object recognition is not linear.
(v) ClickNet- Obijettt--by is robust to time changes: although more clicks do not reveal any more information in this case, the recurrent connections did a good in maintaining its accuracy (instead of forgetting what ClickNet has seen at the zero click).
D INCONGRUENT CONTEXT
The observation that contextual information can help infer what a completely occluded object suggests that placing objects in "wrong" context could impair recognition. Indeed, several
16

---

Under review as a conference paper at ICLR 2020
behavioral and computational tests have shown that objects are harder to recognize when they out of context [Beery et al. 2018 : Dvornik et al. 2018 Choi et al. 2012 We verified these previous observations in separate experiments where constructed images where the objects were placed in incongruent contexts. To consistent with the lift-the-flap case, we used the same test set from
MSCOCO, cropped the objects and pasted them in either congruent context (Figure S7 a)) or in incongruent context (Figure b)),,, as done in previous studies Choi et al. 2012) We tested these congruent and incongruent images both on ClickNet and humans, As expected, and consistent with previous work, we observed that performance in congruent images was higher than performance in incongruent images both for ClickNet (13.8% difference) and humans (18.4% difference).
17

---

Under review as a conference paper at ICLR 2020
Attention Map
Click Location
1x1
m
i E 21, ... 1 Gating value at each location
W 1xn
sigmoid
1 x
a argmax.c. 2
Attention modulated feature vector at each location
hr.
1nn
softmax
m, Click location
1x1
exp
1xD
Rx i (1,2, 4
multiply
Aaxx
Cg i E (1,2, L3
Attention value each location
Bu E (1, L)
Feature vector each location
Figure S1: Schematic illustration of the attention module implementation. Expanding on the overall ClickNet architecture shown in Fig 22] here we zoom into the attention module. The attention module takes as inputs the features each location aw and the output of the LSTM module he and selects the next click location and a map that modulates the features at each location (see Section for description of all the variables).
1
D
he
Gating map
Paaga (1,2, L.
LxD
LSTM
Attention map
Attention modulated feature maps
locations
Context gist vector
Lncxn
L.x D
Predicted
x
Class Labels
Classification vector
Or
"bird"
Feature maps
Figure S2: Schematic illustration of the LSTM module implementation. Expanding on the overall ClickNet archieteture shown in Fig 2 here zoom into the LSTM module. The LSTM module takes as input context gist vector Z, and integrates the information with the previous state inform the attention module in the next time step via h and predict class label (see Section 4 for description of all the variables).
18

---

Under review as a conference paper at ICLR 2020
Example
Example 2
Click
Click=2 2 Click=4
Click = 8
Click=1
Click=2 = 2 Click 44
Click 8
bird
bird
trafficlihtt
lights
Humans
seagull
bind
bird
bird
stoplight
wires
trafficlight
lights
00
ClickNer
kite
kite
bird
bird
Traffic
traffic light
traffic light
traffic light
Figure S3: Example visualziation for humans and ClickNet. This figure shows click locations and attention maps using the same format as Fig 4 here adding results for 2 clicks and clicks.
19

---

Under review as a conference paper at ICLR 2020
80-
Mouseclok-1
Acoumay (N
60
40
HumanClich
HumonFisation
Clicdher- humm-rruilll
Gilae
Cilcc RandPPor
Glosteet HoFine Tuna
Cloclee- Altantiondiall
Clichiet WihAppatt Lov
Clisheer -bjony
Chowlet- Revesied
VGG Attenttten----- -ttenttnntuuaanse tumanstio
vaG- Attenton
Vala Aftertonn RandPhor
VGG Fulloo
VGG -nummardd
vaaa Rardom
VG3 Blir
SVM wal--notooyy Instarens
SVM cat-catpoyy
Histen
Deseet ab CRF
(a) Click-1
Mouseciik-2 2
Accurscy (N)
60-
HomanGick
HumanFiratin
ClicWer- humbnalic
Clicktel
Clishlat- AurdPyy
CissWer- NaFinna Tune
CaskWer- Artentionduil
ClisMter- wanAAppnao.oo oos
ClicAet- -lotAtt--ojOnyt ObjCnty
ClesWrr- CilsWer-CCiraeee Objaavaaled
VGG Attentttor----- humereiic
VGG Attention
VGG Altantion RandiPior
VoG- Fulkre
VaB humandckc
VGG Rondom
voG Bir
SVM cato-ayttrryyy-ee -nstanonn
SVM oategory
Hiddaa
Decpl al CRF
(b) Click-2
80-
Mouseciick.4
Accunny (N)
60
40-
20
HumanCllik
0
Humtan Fitation
Clickte Clorket-tumsren Humenolii
Cilakhal
Cilcttee- RandfPrit
Cleah NoFing Tune
Ciict Attetionbuut
Clisktet- Clicktet- wihhtlpha..o
Ciisdtet- ObyOnty
Clechiet- Ooylaveatedd
VaG- Attertioot homas taak
VaG Attemmio
vaG Attortion---nnn RandiPior flr
VGG Fulree
vaG humaresie
vaG Random
vaia. -Biu
SVM call--anngrryy iiatannes
SVM cat-catapoyy
Hitssenktakovlldd
Deggl ab ORF
(c) Click-4
80-
Mousecick
Accuracy (%)
60-
40
20
HumenClich
Humaniizaion
Clic humenolot
OliWWer
Cilcklet- RardPpor
GlaaNeet Norina Tund
Cassttet- Casktee-AAtattion Attantiondull
ClokNer- WinAlppat.co ons
CiiaAist ObyOnly
ClisdNed Roy
vaa Attentonn-nni Attenton-fumarciis humarciick
VaG VaG-Arannio Artntion
vaG. Attentonn RardPPaor
vag- Fulles
VGG- humandlic
vGG Randem
VGG Olie
SVM wall--ottgoyy--yy iattago----ittareeee instaned
SVM CMM-cctegory
HiddenMarkowtedd
Despd ab ORF
(d) Click-8
Figure S4: Contextual reasoning accuracy of humans and models. Expanding on the results in Fig. 3a here we add the results for 2 clicks and 4 clicks, as well as additional comparative models
(Section
Section and Section Section D describe each model).
20

---

Under review as a conference paper at ICLR 2020
pank
Actual Labels
potted din na
Predicted Labels
Figure S5: Confusion matrix for ClickNet with all click conditions. The format is the same as in Figure 5a except showing all 80 categories here. The element in row i, column denotes the probability that ClickNet predicted label while the ground truth label was (see scale bar on right). The sum of all probabilities in row equals
21

---

Under review as a conference paper at ICLR 2020
Click Sequence Similarity Score
2 2-click
0.5
4-click
8-click
0.4
chance
0.3
0.2
0.1
Between- Hewween-HumanClck HumanClick
-uniintiok-Clickkeet ClickNet
Between- HumanClick
HumanClick Cimannliok-CliccNett ClickNet
Between Hoaveen-Huminn HumanClick
Clliciiiikkkli ClickNet
HumanClick-
HumanClick
Figure S6: image-by-image consistency in the spatiotemporal pattern of click sequences across different number of clicks. CLick sequence score is originally defined in evaluating eye fixation sequence consistency Madsen et al. 2012 Zhang et al. 2018 Here, we use the same metrics to compare the click sequences between-subjects and between ClickNet and subjects for 2 (light gray), (dark gray), and 8 (black) clicks. The larger the click sequence score, the more similar the sequences are. The dashed line indicates chance performance, obtained by randomly permuting the clicks among images. Results shown here are averaged over subject pairs.
(a) Congruent context
(b) Incongruent context
Figure S7: Example images from congruent and incongruent context. (a) congruent example: fork (from another image) pasted in a dining scene; (b) incongruent example: knife (from another image) pasted a ski field. Red bounding boxes indicate target object locations.
22


Reviews:
[{"review_id": "rkx502giYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I agree that context is important in some visual recognition tasks. I think this is an ambitious study and find the employed experimental methods interesting. \n\nHowever, it is not clear to me what has been actually revealed by this study. \n\nIn the last section, there is a statement “the model adequately predicts human sampling behavior and reaches almost human-level performance in this contextual reasoning task.” I think that at least the first half is overstatement; it is not well validated by the experimental results.\n\nFor instance, contrary to the authors’ claim, I do not think the effectiveness of the recurrent connections in ClickNet is sufficiently validated. I think a yet another baseline is missing in the experiments, which is a strategy of clicking points in the periphery of the black box while avoiding (or minimizing) overlaps of the regions deblurred by previous clicks. \n\nAlthough ClickNet-RandPrior appears to be close to this strategy, it does not seem to use any constraint of avoiding such region overlap. On the other hand, in the training of ClickNet, a constraint \\sum_t\\alpha_{ti}=1 is used, which seems to play this very role, i.e., making it possible for ClickNet to avoid the region overlap. Isn’t the good performance of ClickNet fully attributable to this constraint?\n\nThe proposed method is to make subjects (or ClickNet) click a series of points in the input image and then deblur the local regions around the points. I suppose this procedure is accumulative, that is, once a local region is deblurred, it will be kept deblurred in the subsequent clicks. Then, I'm not sure if the order of clicking points really matters, whether it is a human subject or the ClickNet. For instance, is there any evidence that a click is dependent on the previous clicks, other than the above behavior of avoiding overlaps?\n\nAdditionally, I am somewhat skeptical if a pretrained VGG can really extract useful features from (partially) blurred images, even though it is not trained on blurred images. It is widely known that CNNs for visual recognition tasks are vulnerable to image blur, noise, etc. when they do not exist in the images used for training, which is a kind of domain shift. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gLWa9UcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper does a psychological-computational-combined experiments for context reasoning. The experiment is done by \"lift-the-flap\" -- masking out a region of interest in the image and let either the human or a convNet based model to guess what it is by checking the context. Both of them are first shown with a blurred image with masked region, and then start to guess by clicking on surrounding regions and unblur them. A lot of baselines are compared and it is shown that the computational model is working well, and the behavior is highly correlated with humans.\n\n+ The paper reads very well, the illustrations and tables are very well done.\n+ The experiment itself is interesting that it delves into the context problem directly. It would be interesting to see how it works for objects that are out of context though:\nhttp://people.csail.mit.edu/myungjin/outOfContext.html\n+ A like it that a great set of baselines and analysis are done for the paper. It strengthens the paper a lot.\n\n- I think the paper needs to have a baseline for the upper bound as well: what is the accuracy if the region is seen? In other words, what is the performance if no context is needed. It would be interesting to see the gap over there. A baseline could be a region-classification model, e.g., from:\nChen, Xinlei, et al. \"Iterative visual reasoning beyond convolutions.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\nJust from the results (e.g. Fig 3) it seems our current models are already doing pretty well! (and it is VGG, not the best model yet), but on the other hand it is not clear how such models can help recognize visible objects better -- maybe a lot of the things that the context can offer has already been incorporated in the object pixels itself. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hyg_7QhvFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is about context reasoning in the visual recognition. They designed a task called 'lift-the-flap', where the human or the model are given an image with one of the object blacked out. The image is provided in a relative low resolution, and the subjects could choose to click on area to reveal a local window of high-res image. The subject then need to answer what is the object in the blacked out region after a few clicks (ranging from 1 to 8 clicks). The authors has collected human data using mTurk, and also trained a model (named ClickNet) to perform the task. ClickNet has a LSTM memory which can carry the information of previous clicks and use spatial attention mechanism to decide the next click location (just use the argmax of the attention weight.) I find the paper is clearly-written and quite easy to follow. I think this is an interesting paper comparing human attentional performance to a trained computation model (without any human-imposed prior), showing that they behave similarly. However, I am not sure it would be of general interest to the ICLR audience. This may be more suitable for cognitive science conference/journal in my opinion. I have a few suggestions below:\n\n1. In formula 2, the attention weight (before softmax) is  \" e_ti = A_h * h_{t−1} + A_a * a_ti \" I am surprised there is no term that is h_{t-1} * a_ti, which do the content-based reading that is common in memory model like (DNC) or in language processing (transformer.) Can the author comment on this decision?\n\n2. In result 5.1, the authors titled: \"WHAT: IMPORTANCE SAMPLING VIA ATTENTION AND PRIOR INFORMATION\". I am not sure I get where is the importance sampling appear in this paragraph. Importance sampling is a specific term, which refers to estimating properties of a particular distribution, while only having samples generated from a different distribution than the distribution of interest. I did not see how that is related to this paragraph. Or, I may have missed it, and would like the authors to elaborate.\n\n3. I could not follow well the discussion in section 5.5 WHEN: ROLE OF RECURRENT CONNECTIONS. Based on the title of the section, I would expect to discuss why the recurrence of LSTM is important. Though, the paragraph is mainly talking about other control models like SVM and HMM, so I don't quite follow it. Also, it is not clear to me the word 'When' is a good choice. I am convinced that the LSTM helps the model carrying information across click, but it is not clearly shown the sequential order is critical (say click location 1, then location 2, then location 3 compared to location 2, location 3, then location 1 matters) Maybe it does? But, it is now shown in the results.\n\n4. The comparison of where the human versus the model click (in Figure 6) is impressive. I wonder if the authors could go even one step further that is to see if the temporal order of the clicks are similar in human vs. model. It may not be the case. However, if it is true, that could make an even stronger case that the model has a similar way to decide attentional location compared to human subjects.\n\nMinor:\n1. It is impressive that the authors have considered a wide range of control models. Though, for a conference paper with limited page limit, I don't think it is necessary to describe all these models in the main text ( for the SVM, HMM, and CRF models.)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present the task lift-the-flap where an agent (artificial or human) is presented with a blurred image and a hidden item. The agent can de-blur the parts of the image by clicking on it. The authors introduce a model for this task (ClickNet) and they compare this against others.
As reviewers point, this paper presents an interesting set of experiments and analyses. Overall, this type of work can be quite influential as it gives an alternative way to improve our models by unveiling human strategies and using those as inductive biases for our models. That being said, I find the conclusions of this paper quite narrow for the general audience of ICLR (as R2 and R3 also point), as authors look into an artificial task and show ClickNet performs well. But what have we learned beyond that? How do we use these results to improve either our models or our understanding of these models? I believe these are the type of questions that  are missing from the current version of the paper and that if answered would greatly increase its impact and relevance to the ICLR community. At the moment though, I cannot recommend this paper for acceptance.

Decision: Reject