Lift-the-flap: what, where and when for context reasoning

Context reasoning is critical in a wide variety of applications where current inputs need to be interpreted in the light of previous experience and knowledge . Both spatial and temporal contextual information play a critical role in the domain of visual recognition . Here we investigate spatial constraints ( what image features provide contextual information and where they are located ) , and temporal constraints ( when different contextual cues matter ) for visual recognition . The task is to reason about the scene context and infer what a target object hidden behind a flap is in a natural image . To tackle this problem , we first describe an online human psychophysics experiment recording active sampling via mouse clicks in lift-the-flap games and identify clicking patterns and features which are diagnostic for high contextual reasoning accuracy . As a proof of the usefulness of these clicking patterns and visual features , we extend a state-of-the-art recurrent model capable of attending to salient context regions , dynamically integrating useful information , making inferences , and predicting class label for the target object over multiple clicks . The proposed model achieves human-level contextual reasoning accuracy , shares human-like sampling behavior and learns interpretable features for contextual reasoning . 1 INTRODUCTION . The tiny object on the table is probably a spoon , not an elephant . Objects do not appear in isolation . Instead , they co-vary with other objects , their sizes and colors usually respect regularities with respect to nearby elements , and objects tend to appear at specific locations within a scene . Humans exploit these contextual associations . Contextual analyses based on the statistical summary of object relationships , provide an effective source of information for perceptual inference tasks , such as object detection ( Torralba ( 2003 ) ; Park et al . ( 2010 ) ; Hoiem et al . ( 2005 ) ; Torralba et al . ( 2010 ) ; Liu et al . ( 2018b ) ) , scene classification ( Gonfaus et al . ( 2010 ) ; Torralba et al . ( 2005 ) ; Yao et al . ( 2012 ) ) , semantic segmentation ( Yao et al . ( 2012 ) ) , and visual question answering ( Teney et al . ( 2017 ) ) . An example of how contextual information is incorporated during object recognition is lift-the-flap books , where a flap covers part of the page . Children make guesses about what is behind the flap based on the context and check their answers by lifting the flap ( Figure 1a ) . Here we investigate what image features matter for contextual reasoning and where those features are with respect to the target object of interest . Furthermore , scene interpretation in humans involves a sequence of eye movements ( Zhang et al . ( 2018 ) ) , each one of these image samples providing additional context to inform interpretation of the contents of the next location . Therefore , we also investigate whether integration of scene information over time ( ” when ” information ) matters for context reasoning . To tackle the problem of contextual reasoning , we introduce the lift-the-flap task and conduct online psychophysics experiments where subjects make mouse clicks while they explore important contextual cues to identify a hidden target ( Figure 1b ) . We investigate the contextual reasoning strategies observed from human active clicking patterns . As a proof of concept , we propose a recurrent attention model ( ClickNet ) , to automatically learn these contextual reasoning strategies . The model guides attention towards regions with informative context , decides where to click on the image , and makes inferences about the target behind the flap . The learnt clicking patterns and predicted class labels by ClickNet share remarkable similarities with human behavior . 2 RELATED WORKS . 2.1 ROLE OF CONTEXT IN HUMAN VISION . Contextual information affects the efficiency of several visual processes ( Auckland et al. , 2007 ; Biederman et al. , 1982 ; Hollingworth , 1998 ; Bar & Aminoff , 2003 ; Goh et al. , 2004 ; Aminoff et al. , 2006 ) , such as object recognition ( Auckland et al. , 2007 ) , object detection ( Biederman et al. , 1982 ; Hollingworth , 1998 ) , visual working memory ( Friedman , 1979 ; Aminoff et al. , 2006 ) and visual search ( Henderson et al. , 1999 ) . Objects appearing in a familiar background can be detected more accurately and processed more quickly than objects appearing in an incongruent scene . Here we focus on what visual features contribute to contextual reasoning , which parts of image regions attract human attention to make inferences , and the dynamic sequence of directed sampling needed to idenfity a hidden target . 2.2 ROLE OF CONTEXT IN COMPUTER VISION . Contextual reasoning about objects and relations is critical to machine vision . In fact , many object recognition studies using natural image datasets such as ImageNet , rely implicitly but strongly on contextual feature regularities Geirhos et al . ( 2018 ) ; Brendel & Bethge ( 2019 ) . Some works also show that these models can fail when objects are placed in an incongruent context ( Beery et al . ( 2018 ) ; Dvornik et al . ( 2018 ) ; Choi et al . ( 2012 ) ) . These studies motivate examining the role of pure context without any object information in situations of complete occlusion ( lift-the-flap ) ; here we provide human and computational benchmarks for object inference using exclusively context information . Several studies use context to improve object detection ( Park et al. , 2010 ; Hoiem et al. , 2005 ; Torralba et al. , 2010 ; Liu et al. , 2018b ; Chen et al. , 2018b ) . Context can take the form of global scene context ( Torralba et al. , 2010 ) , ground plane estimation ( Park et al. , 2010 ) , geometric context ( Hoiem et al. , 2005 ) , relative location Desai et al . ( 2011 ) , 3D layout ( Lin et al. , 2013 ) , and spatial support and geographic information ( Divvala et al. , 2009 ) . Researchers proposed Conditional Random Field ( CRF ) models that reason jointly across multiple computer vision tasks in image labeling and scene classification Gonfaus et al . ( 2010 ) ; Yao et al . ( 2012 ) ; Ladicky et al . ( 2010 ) . Context can lead to improved performance in both object detection and semantic segmentation tasks Mottaghi et al . ( 2014 ) . Neural network architectures incorporating contextual information have been used in object priming ( Torralba , 2003 ) , place and object recognition ( Wu et al. , 2018 ; Torralba et al. , 2005 ) , object detection ( Liu et al. , 2018b ; Chen et al. , 2018b ) , and visual question answering ( Teney et al. , 2017 ) . Here we focus on developing a biologically inspired computational model for contextual reasoning that can automatically and dynamically sample image regions of interest , integrating information in space and time to make inferences about a hidden object . Additionally , we compare the model ’ s performance against human behavior in the same task . Several interesting approaches have combined graphical models with deep neural networks for structural inference , primarily in structured prediction tasks ( Marino et al. , 2016 ; Choi & Savarese , 2012 ; Chen et al. , 2018a ; Teney et al. , 2017 ; Hu et al. , 2016 ; Battaglia et al. , 2016 ; Xu et al. , 2017 ; Chen et al. , 2018b ; Choi et al. , 2012 ) . Hu et al . ( 2016 ) designed a structured model to improve classification performance by leveraging relations among scenes , objects , and their attributes . A structured inference model is also used in ( Choi & Savarese , 2012 ; Deng et al. , 2016 ) to analyze relations in group activity recognition . Several works , like Structural-RNN Jain et al . ( 2016 ) and Interaction Net Battaglia et al . ( 2016 ) , combine spatiotemporal graphs and sequence learning . These works assume full contextual information is available , while in our experiment we consider only partial contextual information that is sequentially revealed after each mouse click . DeepLab sends the response at the final layer of a CNN to a CRF model for semantic image segmentation Chen et al . ( 2018a ) . Subsequently , Schwing & Urtasun ( 2015 ) ; Zheng et al . ( 2015 ) transformed the CRF model into a Recurrent Neural Network . Breaking away from these studies where graph optimization is performed globally , our proposed model selects important visual features using an attention mechanism and integrates partial information over multiple steps , which is computationally more efficient and accurate in the current task ( Section 5 ) . 3 LIFT-THE-FLAP TASK . 3.1 HUMAN BEHAVIORAL EXPERIMENTS . Subjects were presented with a natural image where one object was hidden by a rectangular black box and everything else was blurred . They were allowed a fixed number of mouse clicks between 1 and 8 , each click revealed part of the image in high resolution . To minimize overlap between clicks , we enforced subjects to click at places at least 10 pixels apart from all previous clicks . After the target number of clicks , they had to provide a single word to describe the object hidden behind the black box ( Figure 1b ) . The clicking experiments were run on Amazon Mechanical Turk Turk ( 2012 ) . The stimulus set consisted of 573 images from the test set of the MSCOCO Dataset ( Lin et al. , 2014 ) , spanning 80 object categories . This dataset has been widely used for object recognition and detection studies ( Lin et al. , 2014 ) . We constrained the stimulus set to have a uniform distribution of 6 - 8 target objects per category . To avoid any potential memory effects , subjects were only exposed to each image once . The trial presentation order was randomized . 3.2 GROUND TRUTH RESPONSES . In contrast to experiments where subjects are forced to perform N-way categorization ( e.g. , Tang et al . ( 2018 ) ) , here there were no constraints on how subjects describe the hidden object . This probing mechanism was implemented for two reasons : ( i ) it is difficult for humans to memorize 80 object classes and there could be non-uniform memory effects impacting the results ; ( ii ) presenting humans with an 80-choice question could introduce biases in their inference processes . We could not simply use the 80 category labels to evaluate performance because subjects could use other words or synonyms and we are interested in context reasoning rather than language abilities . Therefore , to evaluate humans ’ performance , we separately collected a distribution of ground truth answers for each hidden target by presenting to 10 other subjects , who did not participate in the main task , the same set of images with the target objects highlighted by a bounding box but not hidden . During the lift-the-flap task , a response was considered to be correct if it matched any of the ground truth labels , allowing for plurals and misspellings . 3.3 EVALUATION METRICS . We introduce several evaluation metrics to measure contextual reasoning accuracy and to compare the consistency of mouse clicking patterns between humans and computational models . We evaluated ClickNet on the MSCOCO Dataset using the typical classification accuracy measure . In Fig 5b , we report top-1 classification accuracy as a function of context-object ratio . The context-object ratio is defined as the total area of the image excluding the hidden target divided by the hidden target object size . For example , a context-object ratio of 1 implies that the size of the black box and the size of the contextual information is the same ( see Figure 5b for example images with different context-object ratios ) . To measure the degree of spatiotemporal consistency between two mouse clicking patterns ( human versus human ; or human versus computational models ) , we introduced two measures . First , we computed the minimum Euclidean distance between the sequence of clicks in each trial , regardless of order . The smaller the median in the distribution of distances , the more similar the two mouse clicking patterns are . Second , we introduced a click sequence score , as originally defined in evaluating eye fixation sequence consistency Madsen et al . ( 2012 ) ; Zhang et al . ( 2018 ) . Here , we use the same metrics to compare the spatial-temporal consistency between click sequences . The larger the click sequence score , the more similar the sequences are .

Summary: The paper does a psychological-computational-combined experiments for context reasoning. The experiment is done by "lift-the-flap" -- masking out a region of interest in the image and let either the human or a convNet based model to guess what it is by checking the context. Both of them are first shown with a blurred image with masked region, and then start to guess by clicking on surrounding regions and unblur them. A lot of baselines are compared and it is shown that the computational model is working well, and the behavior is highly correlated with humans.