Learning a metacognition for object detection

1 INTRODUCTION . Learning accurate representations of the world is critical for prediction , inference , and planning in complex environments ( Lake et al. , 2017 ) . In human vision , these representations are generated by a perceptual system that transforms light entering the retina into representations of the physical space and the objects in it ( Kar & DiCarlo , 2021 ; Güçlü & van Gerven , 2017 ) . While human perception is generally robust and reliable , it nonetheless suffers from errors , such as when we fail to detect an object in a crowded visual scene , when we confuse one object for another , or when we experience a visual illusion . Critically , in these situations , people have a metacognition that helps them recognize that their visual system has failed them and adjust their reasoning accordingly . End-to-end object recognition models have revolutionized computer vision in the past decade ( Krizhevsky et al. , 2012 ; Redmon et al. , 2016 ; Zou et al. , 2019 ) , but they face a similar challenge : how can we identify when a non-existent object is incorrectly detected ( hallucinations ) , or when a real object is not reliably recognized ( misses ) ? Traditional approaches to this problem have focused on increasing the training data ( Lin et al. , 2014 ; Deng et al. , 2009 ) or improving the model ’ s architecture ( Carion et al. , 2020 ; Liu et al. , 2021 ) . While these methods are generally successful , the possibility of misses and hallucinations remains—particularly when confronted with out-of-sample data—and a foolproof visual model must be able to identify when this occurs . We propose that augmenting object recognition models with a metacognition—an external meta-representation of their own computational processes—can allow these systems to monitor their percepts and flexibly decide when to trust or doubt proposed representations , much like humans do . In this paper , we propose a formulation of metacognition that can be learned online , requiring no human annotation or feedback . Given a pre-trained object detection model and a data stream from videos or scene viewpoints , our approach is to perform a joint inference over the objects generating the percepts , and a meta-representation of the system ’ s performance . Our model , METAGEN , achieves this by representing its metacognition as a generative model that captures the joint distribution between objects in a scene and the model ’ s pattern of detections . Critically , we make this problem tractable by drawing on an insight from cognitive science . Infants come into the world equipped with a basic form of ‘ core knowledge ’ or ‘ start-up sofware ’ ( Lake et al. , 2017 ; Carey , 2009 ) . These priors constrain the representations of world states that humans consider possible and support drawing richer inferences with less data . Specifically , we focus on early object constraints known as Spelke principles ( Spelke & Kinzler , 2007 ; Smith et al. , 2019 ) : objects persist in time , they can not occupy the same position as another object , and they move continuously in space . Conditioning inference on these basic principles enables METAGEN to learn a metacognitive representation of the object detection model by analyzing and resolving patterns of percepts that violate the Spelke principles of objects . We evaluate METAGEN in a variety of ways and against several comparison models , focused on performance on out-of-sample data . Using scenes rendered in in the ThreeDWorld ( TDW ) virtual environment ( Gan et al. , 2020 ) , we first examine METAGEN ’ s capacity to learn accurate metacognition online for a variety of state-of-the-art object detection networks that span different dominant paradigms ( single-stage , two-stage , and transformer models ) . We find that , during learning , METAGEN already produces marked improvements over key comparison models : the object detection model without a metacognition and the object detection models with a confidence threshold fit to the data . We then evaluate METAGEN after learning by conditioning on its learned metacognition and testing its accuracy on a new set of data . We find that METAGEN also outperforms comparison models on a set of novel scenes . Overall , our work makes two main contributions . First , we propose a novel approach to improving object detection models through a metacognition that helps monitor a model ’ s performance . This distinction between perceptual processing systems and high-level cognitive systems mirrors the architecture of the human mind ( Firestone & Scholl , 2016 ) where both processes work jointly to build accurate representations of the world . Second , we present METAGEN , a model that learns a metacognition for object detection models . We show that METAGEN can learn an accurate representation of object detection models without feedback and leads to rapid improvements in the system ’ s accuracy . 2 RELATED WORK . Metacognition in AI . Previous work has argued for the importance of metacognition for machine learning and AI ( Cox , 2005 ) . Models that use metacognition during learning have shown promise for improving classification accuracy ( Babu & Suresh , 2012 ; Subramanian et al. , 2013 ) . This work has focused on engineering an inflexible metacognition to guide learning . In this paper , we focus on a complimentary problem : learning a metacognition without feedback that helps monitor a neural network ’ s detections . Computational cognitive science . Our core idea—learning a metacognitive representation of a perceptual system—is inspired by research in cognitive science showing that human reasoning is structured around mental models of the physical world ( Ullman et al. , 2017 ; Battaglia et al. , 2013 ) , of the social world ( Jara-Ettinger et al. , 2019 ; Baker et al. , 2017 ) , and of ourselves ( Gopnik , 1993 ; Nisbett & Wilson , 1977 ) . Our work is also related to computational models of human core knowledge ( Smith et al. , 2019 ; Kemp & Xu , 2009 ) . The difference is that our work uses object principles to learn a metacognition , whereas past work has worked on modeling object principles themselves . Uncertainty-aware AI . The spirit of our work relates more closely to uncertainty-aware AI . This work focuses on building end-to-end systems that express uncertainty in their inferences ( Sensoy et al. , 2018 ; Kaplan et al. , 2018 ; Ivanovska et al. , 2015 ) . Our work focuses instead on how to learn a model of uncertainty over a pre-trained model . These two approaches complement each other . In humans , metacognitive uncertainty supplements the intrinsic uncertainty in visual perception . Meta-Learning . A major challenge for machine learning algorithms is generalizing to new datasets and tasks that they were not trained on . Recent and exciting work in machine learning has explored the possibility of meta-learning ( Hospedales et al. , 2020 ) , which involves applying a model trained on one task to a new setting ( Finn et al. , 2017 ) . At first glance , our work may appear to fall within the field of meta-learning . However , in contrast to meta-learning , METAGEN does not improve the firstlevel learning algorithm ( the neural network for object detection ) . Instead , METAGEN functions at a level above learning algorithm , taking the outputs of the learning algorithm as unchangeable inputs . 3 METAGEN . Figure 1 : Schematic depicting the forward generative model . Gt is the prior over metacognition ( Vt ) at time t ; Wt is the world state ; and Ot are the observations that are generated . Wt , Ot , and Gt collectively influenceGt+1 , the prior over Vt1 . In this section we present our METAGEN model , applied to the problem of inferring what objects are in a 3D scene , given a video from a moving camera . METAGEN captures the relationship between the objects in a scene and the detections obtained by its perceptual module ( i.e. , the output from an object recognition model ) through a hierarchical generative model with two levels ( Figure 1 ) . The lower level captures the joint distribution between world states and object detections , as determined by a metacognitive representation V that summarizes its perception module ’ s performance . The upper level describes the dynamics of the prior over V , capturing the idea that a metacognitive representation can—and should—change with experience . Given a collection of observations ~o obtained from a scene , our approach is to perform a joint inference over the unobservable world state W and the metacognitive representation V , conditioned on Spelke principles of objects . We begin by presenting the generative model ( 3.1 ) and then turn to how we perform joint inference over the world state and the system ’ s metacognition ( 3.2 ) . 3.1 GENERATIVE MODEL . 3.1.1 WORLD-STATE REPRESENTATION AND SPELKE PRINCIPLES . Let W be the space of possible world states , where W ∈W consists of a collection of objects , each associated with a category and with a 3D position in space . Throughout , we use Wt to refer the world state from the t-th scene encountered by METAGEN , and ~W = { W0 , . . . , WT } to refer to the complete sequence of world states . In our approach , Spelke principles constrain possible world states . Specifically , the assumption of object permanence implies that each world state has a fixed set of objects that does not change . Next , the assumption that two objects can not occupy the same physical position is implemented as a prior over the locations of the objects in a world state . The prior over the location of an object is a uniform over 3-D space , except near another object , where the probability of placement decreases to 0 . Finally , the assumption of spatio-temporal continuity implies that any moving object must exhibit a continuous trajectory . Our Experimental task focuses on inferring permanent fixtures in a room , so we therefore do not implement this constraint , although doing so is a solved problem ( Smith et al. , 2019 ; Kemp & Xu , 2009 ) . Implementation details are available in the Appendix . 3.1.2 PERCEPTUAL MODULE . The goal of METAGEN is to build a metacognitive representation of any object detection model that can transform images into object detections , each associated with a category and a position in the 2D image . Although METAGEN does not have access to ground truth ( W ) , we assume access to the camera ’ s state cs , which determines its position in space and orientation . This helps with inferring and tracking the location of different objects ( as this is not the main focus of our work ) . We additionally assume access to video boundaries , enabling METAGEN to reset object representations across scenes . Critically , the perceptual models functionally serves as a black box , and METAGEN does not have any access to the internal state of the model , knowledge about its performance , or any feedback about the final inferences . Formally , let Ot = ~o be the collection of observations associated with the t-th scene . Each observation ocs ∈ Ot is a percept associated with camera state cs ( from the complete camera trajectory cj ) , and consists of a set of point detections obtained from the object recognition model , each containing the object ’ s category and 2-D location on an image : d = ( c , x , y ) .

Summary: The paper proposes a new method to incorporate Spelke's principals of object perception as constraints to improve the performance of an out-of-the-box object detector. This is done via defining a hierarchical generative model which defines "metacognitive" priors over the a set of observations. Through joint inference over these metacognitive priors and new unobserved states, the method outputs better object detections. The authors show improved performance on a synthetic dataset which contains scenes rendered in a virtual environment.