Learning a metacognition for object detection

1 INTRODUCTION . Learning accurate representations of the world is critical for prediction , inference , and planning in complex environments ( Lake et al. , 2017 ) . In human vision , these representations are generated by a perceptual system that transforms light entering the retina into representations of the physical space and the objects in it ( Kar & DiCarlo , 2021 ; Güçlü & van Gerven , 2017 ) . While human perception is generally robust and reliable , it nonetheless suffers from errors , such as when we fail to detect an object in a crowded visual scene , when we confuse one object for another , or when we experience a visual illusion . Critically , in these situations , people have a metacognition that helps them recognize that their visual system has failed them and adjust their reasoning accordingly . End-to-end object recognition models have revolutionized computer vision in the past decade ( Krizhevsky et al. , 2012 ; Redmon et al. , 2016 ; Zou et al. , 2019 ) , but they face a similar challenge : how can we identify when a non-existent object is incorrectly detected ( hallucinations ) , or when a real object is not reliably recognized ( misses ) ? Traditional approaches to this problem have focused on increasing the training data ( Lin et al. , 2014 ; Deng et al. , 2009 ) or improving the model ’ s architecture ( Carion et al. , 2020 ; Liu et al. , 2021 ) . While these methods are generally successful , the possibility of misses and hallucinations remains—particularly when confronted with out-of-sample data—and a foolproof visual model must be able to identify when this occurs . We propose that augmenting object recognition models with a metacognition—an external meta-representation of their own computational processes—can allow these systems to monitor their percepts and flexibly decide when to trust or doubt proposed representations , much like humans do . In this paper , we propose a formulation of metacognition that can be learned online , requiring no human annotation or feedback . Given a pre-trained object detection model and a data stream from videos or scene viewpoints , our approach is to perform a joint inference over the objects generating the percepts , and a meta-representation of the system ’ s performance . Our model , METAGEN , achieves this by representing its metacognition as a generative model that captures the joint distribution between objects in a scene and the model ’ s pattern of detections . Critically , we make this problem tractable by drawing on an insight from cognitive science . Infants come into the world equipped with a basic form of ‘ core knowledge ’ or ‘ start-up sofware ’ ( Lake et al. , 2017 ; Carey , 2009 ) . These priors constrain the representations of world states that humans consider possible and support drawing richer inferences with less data . Specifically , we focus on early object constraints known as Spelke principles ( Spelke & Kinzler , 2007 ; Smith et al. , 2019 ) : objects persist in time , they can not occupy the same position as another object , and they move continuously in space . Conditioning inference on these basic principles enables METAGEN to learn a metacognitive representation of the object detection model by analyzing and resolving patterns of percepts that violate the Spelke principles of objects . We evaluate METAGEN in a variety of ways and against several comparison models , focused on performance on out-of-sample data . Using scenes rendered in in the ThreeDWorld ( TDW ) virtual environment ( Gan et al. , 2020 ) , we first examine METAGEN ’ s capacity to learn accurate metacognition online for a variety of state-of-the-art object detection networks that span different dominant paradigms ( single-stage , two-stage , and transformer models ) . We find that , during learning , METAGEN already produces marked improvements over key comparison models : the object detection model without a metacognition and the object detection models with a confidence threshold fit to the data . We then evaluate METAGEN after learning by conditioning on its learned metacognition and testing its accuracy on a new set of data . We find that METAGEN also outperforms comparison models on a set of novel scenes . Overall , our work makes two main contributions . First , we propose a novel approach to improving object detection models through a metacognition that helps monitor a model ’ s performance . This distinction between perceptual processing systems and high-level cognitive systems mirrors the architecture of the human mind ( Firestone & Scholl , 2016 ) where both processes work jointly to build accurate representations of the world . Second , we present METAGEN , a model that learns a metacognition for object detection models . We show that METAGEN can learn an accurate representation of object detection models without feedback and leads to rapid improvements in the system ’ s accuracy . 2 RELATED WORK . Metacognition in AI . Previous work has argued for the importance of metacognition for machine learning and AI ( Cox , 2005 ) . Models that use metacognition during learning have shown promise for improving classification accuracy ( Babu & Suresh , 2012 ; Subramanian et al. , 2013 ) . This work has focused on engineering an inflexible metacognition to guide learning . In this paper , we focus on a complimentary problem : learning a metacognition without feedback that helps monitor a neural network ’ s detections . Computational cognitive science . Our core idea—learning a metacognitive representation of a perceptual system—is inspired by research in cognitive science showing that human reasoning is structured around mental models of the physical world ( Ullman et al. , 2017 ; Battaglia et al. , 2013 ) , of the social world ( Jara-Ettinger et al. , 2019 ; Baker et al. , 2017 ) , and of ourselves ( Gopnik , 1993 ; Nisbett & Wilson , 1977 ) . Our work is also related to computational models of human core knowledge ( Smith et al. , 2019 ; Kemp & Xu , 2009 ) . The difference is that our work uses object principles to learn a metacognition , whereas past work has worked on modeling object principles themselves . Uncertainty-aware AI . The spirit of our work relates more closely to uncertainty-aware AI . This work focuses on building end-to-end systems that express uncertainty in their inferences ( Sensoy et al. , 2018 ; Kaplan et al. , 2018 ; Ivanovska et al. , 2015 ) . Our work focuses instead on how to learn a model of uncertainty over a pre-trained model . These two approaches complement each other . In humans , metacognitive uncertainty supplements the intrinsic uncertainty in visual perception . Meta-Learning . A major challenge for machine learning algorithms is generalizing to new datasets and tasks that they were not trained on . Recent and exciting work in machine learning has explored the possibility of meta-learning ( Hospedales et al. , 2020 ) , which involves applying a model trained on one task to a new setting ( Finn et al. , 2017 ) . At first glance , our work may appear to fall within the field of meta-learning . However , in contrast to meta-learning , METAGEN does not improve the firstlevel learning algorithm ( the neural network for object detection ) . Instead , METAGEN functions at a level above learning algorithm , taking the outputs of the learning algorithm as unchangeable inputs . 3 METAGEN . Figure 1 : Schematic depicting the forward generative model . Gt is the prior over metacognition ( Vt ) at time t ; Wt is the world state ; and Ot are the observations that are generated . Wt , Ot , and Gt collectively influenceGt+1 , the prior over Vt1 . In this section we present our METAGEN model , applied to the problem of inferring what objects are in a 3D scene , given a video from a moving camera . METAGEN captures the relationship between the objects in a scene and the detections obtained by its perceptual module ( i.e. , the output from an object recognition model ) through a hierarchical generative model with two levels ( Figure 1 ) . The lower level captures the joint distribution between world states and object detections , as determined by a metacognitive representation V that summarizes its perception module ’ s performance . The upper level describes the dynamics of the prior over V , capturing the idea that a metacognitive representation can—and should—change with experience . Given a collection of observations ~o obtained from a scene , our approach is to perform a joint inference over the unobservable world state W and the metacognitive representation V , conditioned on Spelke principles of objects . We begin by presenting the generative model ( 3.1 ) and then turn to how we perform joint inference over the world state and the system ’ s metacognition ( 3.2 ) . 3.1 GENERATIVE MODEL . 3.1.1 WORLD-STATE REPRESENTATION AND SPELKE PRINCIPLES . Let W be the space of possible world states , where W ∈W consists of a collection of objects , each associated with a category and with a 3D position in space . Throughout , we use Wt to refer the world state from the t-th scene encountered by METAGEN , and ~W = { W0 , . . . , WT } to refer to the complete sequence of world states . In our approach , Spelke principles constrain possible world states . Specifically , the assumption of object permanence implies that each world state has a fixed set of objects that does not change . Next , the assumption that two objects can not occupy the same physical position is implemented as a prior over the locations of the objects in a world state . The prior over the location of an object is a uniform over 3-D space , except near another object , where the probability of placement decreases to 0 . Finally , the assumption of spatio-temporal continuity implies that any moving object must exhibit a continuous trajectory . Our Experimental task focuses on inferring permanent fixtures in a room , so we therefore do not implement this constraint , although doing so is a solved problem ( Smith et al. , 2019 ; Kemp & Xu , 2009 ) . Implementation details are available in the Appendix . 3.1.2 PERCEPTUAL MODULE . The goal of METAGEN is to build a metacognitive representation of any object detection model that can transform images into object detections , each associated with a category and a position in the 2D image . Although METAGEN does not have access to ground truth ( W ) , we assume access to the camera ’ s state cs , which determines its position in space and orientation . This helps with inferring and tracking the location of different objects ( as this is not the main focus of our work ) . We additionally assume access to video boundaries , enabling METAGEN to reset object representations across scenes . Critically , the perceptual models functionally serves as a black box , and METAGEN does not have any access to the internal state of the model , knowledge about its performance , or any feedback about the final inferences . Formally , let Ot = ~o be the collection of observations associated with the t-th scene . Each observation ocs ∈ Ot is a percept associated with camera state cs ( from the complete camera trajectory cj ) , and consists of a set of point detections obtained from the object recognition model , each containing the object ’ s category and 2-D location on an image : d = ( c , x , y ) .

Summary: This paper proposes a metacognitive framework to augment object detection models to help improve them without human input. The framework is based on learning to detect and fix a model’s output if it violates early object constraints (Spelke principles), namely object permanence, cohesion, and spatiotemporal continuity. The framework, called METAGEN, does not require access to the internal state of the detection model but operates only on its outputs, maintaining a metacognitive representation which summarizes the model’s performance (propensity to hallucinate or incorrectly miss objects of different classes). The representation is initialized for a given scene from a learned and evolving prior, and is updated along with the estimated world state using a particle filter with various rejuvenation steps over objects, locations and metacognitive parameters. METAGEN is tested using single-stage (RetineNet), two-stage (FasterR-CNN), and transformer (DETR) models of object detection within a dataset gathered using the synthetic ThreeDWorld environment, and is shown to improve test performance both during inference and after stopping inference on both training and new test scenes.