Learning a metacognition for object detection
Authors: Marlene Berke
Mario Belledonne
Zhangir Azerbayev
Julian Jara-Ettinger
Venue: ICLR 2022 Submitted (2022)
Keywords: metacognition
object detection
unsupervised learning

Abstract: In contrast to object recognition models, humans do not blindly trust their perception when building representations of the world, instead recruiting metacognition to detect percepts that are unreliable or false, such as when we realize that we mistook one object for another. We propose METAGEN, an unsupervised model that enhances object recognition models through a metacognition. Given noisy output from an object-detection model, METAGEN learns a meta-representation of how its perceptual system works and uses it to infer the objects in the world responsible for the detections. METAGEN achieves this by conditioning its inference on basic principles of objects that even human infants understand (known as Spelke principles: object permanence, cohesion, and spatiotemporal continuity). We test METAGEN on a variety of state-of-the-art object detection neural networks. We find that METAGEN quickly learns an accurate metacognitive representation of the neural network, and that this improves detection accuracy by filling in objects that the detection model missed and removing hallucinated objects. This approach enables generalization to out-of-sample data and outperforms comparison models that lack a metacognition.

Under review as a conference paper at ICLR 2021
LEARNING A METACOGNITION FOR OBJECT DETEC- TION
Anonymous authors Paper under double-bling review
ABSTRACT
In contrast to object recognition models, humans do not blindly trust their percep- tion when building representations of the world, instead recruiting metacognition to detect percepts that are unreliable or false, such as when we realize that we mis- took one object for another. We propose METAGEN, an unsupervised model that enhances object recognition models through metacognition. Given noisy output from an object-detection model, METAGEN learns a meta-representation of how its perceptual system works and uses it to infer the objects in the world responsi- ble for the detections. METAGEN achieves this by conditioning its inference on basic principles of objects that even human infants understand (known Spelke principles: object permanence, cohesion, and spatiotemporal continuity). We test METAGEN variety of otate-of-the-art object detection neural networks. We find that METAGEN quickly learns an accurate metacognitive representation the neural network, and that this improves detection accuracy by filling objects that the detection model missed and removing hallucinated objects. This approach enables generalization to out-of-sample data and outperforms comparison models that lack metacognition.
INTRODUCTION
Learning accurate representations of the world is critical for prediction, inference, and planning in complex environments (Lake al., 2017). In human vision, these representations are generated by perceptual system that transforms light entering the retina into representations of the physical space and the objects in (Kar & DiCarlo 2021 Güçlü & van Gerven 2017). While human perception is generally robust and reliable, it nonetheless suffers from errors, such as when we fail to detect an object in a crowded visual scene, when we confuse one object for another, or when we experience a visual illusion. Critically, in these situations, people have metacognition that helps them recognize that their visual system has failed them and adjust their reasoning accordingly.
End-to-end object recognition models have revolutionized computer vision in the past decade (Krizhevsky et al.) 2012, Redmon et al.) 2016 Zou et al.) 2019). but they face a similar challenge: how can identify when non-existent object is incorrectly detected (hallucinaatins), real object is not reliably recognized (misses)? Traditional approaches to this problem have focused on increasing the training data (Lin al. 2014 Deng al.] (2009) improving the model's ar- chitecture (Carion et al., 2020 Liu et al. 2021). While these methods are generally successful, the mamains-particulan
when
when confronted with out-of-sample data and a foolproof visual model must be able to identify when this occurs. We propose that aug- menting object recognition models with a metacognition an external meta-representattin of their own computational processes can allow these systems to monitor their percepts and flexibly decide when trust or doubt proposed representations, much like humans do.
possibility of misses and hallucinations
In this paper, we propose formulation of metacognition that can be learned online, requiring no human annotation or feedback. Given a pre-trained object detection model and a data stream from videos scene viewpoints, our approach is to perform joint inference over the objects generat- ing the percepts, and meta-repressntattinn of the system's performance. Our model, METAGEN, generative model that captures the joint distri-
achieves this by representing its metacognition bution between objects in a scene and the model's pattern of detections. Critically, we make this problem tractable by drawing on insight from cognitive science. Infants come into the world

---

Under review as a conference paper at ICLR 2021
equipped with basic form of 'core knowledge' "start-up sofware' (Lake et al. 2009). These priors constrain the representations of world states that humans consider possible and support drawing richer inferences with less data. Specifically, known as Spelke principles (Spelke & Kinzler 2007] [Smith et al. (219)): objects persist in time, they cannot occupy the same position as another object, and they move continuously in space. Condition- ing inference on these basic principles enables METAGEN to learn metacognitive representation of the object detection model by analyzing and resolving patterns of percepts that violate the Spelke principles of objects.
2017 Carey
focus early object constraints
We evaluate METAGEN in a variety of ways and against several comparison models, focused on performance on out sample data. Using scenes rendered in in the ThreeDWorld (TDW) virtual environment (Gan et al. 2020), we first examine METAGEN'S s capacity to learn accurate metacog- nition online for a variety of state-of-the-art object detection networks that span different dominant paradigms (single-stage, two-stage, and transformer models). We find that, during learning. META- GEN already produces marked improvements over key comparison models: the object detection model without a metacognition and the object detection models with confidence threshold to the data. We then evaluate METAGEN after learning by conditioning on its learned metacognition and testing its accuracy on a new set of data. We find that METAGEN also outperforms comparison models on set of novel scenes.
Overall, our work makes two main contributions. First, we propose a novel approach to improv- ing object detection models through metacognition that helps monitor a model's performance. This distinction between perceptual processing systems and high-level cognitive systems mirrors the architecture of the human mind (Firestone & Scholl 2016) where both processes work jointly to build accurate representations of the world. Second, we present METAGEN, a model that learns a metacognition for object detection models. We show that METAGEN can learn an accurate rep- resentation of object detection models without feedback and leads to rapid improvements in the system's s accuracy.
RELATED WORK
Metacognition in AL Previous work has argued for the importance of metacognition for machine learning and AI (Cox) 2005). Models that use metacognition during learning have shown promise for improving classification accuracy (Babu & Suresh 2012] Subramanian et al., 2013). This work has focused on engincering inflexible metacognition to guide learning. In this paper, we focus on a complimentary problem: learning a metacognition without feedback that helps monitor a neural network's 's detections.
Computational cognitive science. Our core idea idea-learning a metacognitive representation of a perceptual system is inspired by research in cognitive science showing that human reasoning is structured around mental models of the physical world (Ullman et al. 2017]] Battaglia et al.] 2013
of the social world (Jara-Ettinger al. 2019 Baker al.] (017]. and of ourselves (Gopnik 1993
Nisbett & Wilson
1977). Our work is also related to computational models of human core knowl- edge (Smith et 2019 Kemp & Xu 2009). The difference that our work uses object principles to learn a metacogmition, whereas past work has worked on modeling object principles themselves.
Uncortainty-awart AI. The spirit of our work relates more closely to uncertainty-aware AI. This work focuses on building end-to-end systems that express uncertainty in their inferences (Sensoy al. 2018 Kaplan et al., 2018 Ivanovska et al. 2015). Our work focuses instead on how to learn model of uncertainty over pre-trained model. These two approaches complement each other. In humans, metacognitive uncertainty supplements the intrinsic uncertainty in visual perception.
Meta-Learning. A major challenge for machine learning algorithms is generalizing to new datasets and tasks that they were not trained on. Recent and exciting work in machine learning has explored the possibility of meta-learning (Hospedales et al. 2020), which involves applying a model trained on one task to a new setting (Finn et al. 2017). At first glance, our work may appear to fall within the field of meta-leanaaee learning. However, in contrast to meta- learann. METAGEN does not improve the first- level learning algorithm (the neural network for object detection). Instead, METAGEN functions at a level above learning algorithm, taking the outputs of the learning algorithm as unchangeable inputs.
2

---

Under review as a conference paper at ICLR 2021
3
METAGEN
In this section we present our METAGEN model, applied to the problem of inferring what objects are in a 3D scene, given a video from moving camera. METAGEN captures the lationship between the objects in scene and the detections obtained by its perceptual mod- ule (i.e., the output from an object recognition model) through a hierarchical generative model with two levels (Figure []. The lower level captures the joint distribution between world states and object detections, as determined by metacognitive representation V that summa- rizes its perception module's ' performance. The upper level describes the dynamics of the prior over V, capturing the idea that metacognitive representation can can-and should should-changee change with experience.
GE
G++1
VL
Wt
Vi+1
++1
Ot
Ot+1
Figure I: Schematic depicting the forward gener- ative model. G2 is the prior over metacognition (V₂) at time t: W is the world state: and O are the observations that are generated. We, Os, and G2 collectively influence G.+1, the prior over Ve,-
Given a collection of observations obtained
from scene, our approach is perform joint inference over the unobservable world state W and the metacognitive representation V, conditioned on Spelke principles of objects. We begin by presenting the generative model (3.1) and then turn how we perform state and the system's 's metacognition (3.2).
joint joint inference over the world
3.1 GENERATIVE MODEL.
3.1.1 WORLD-STATE REPRESENTATION AND SPELKE PRINCIPLES
Let W be the space of possible world states, where W E W consists of collection of objects, each associated with a category and with 3D position in space. Throughout, we use We to refer the world state from the t-th scene encountered by METAGEN, and W = (Wo..., Wr} to refer to the complete sequence of world states.
In our approach, Spelke principles constrain possible world states. Specifically, the assumption of object permanence implies that each world state has a fixed set of objects that does not change. Next, the assumption that two objects cannot occupy the same physical position is implemented as prior over the locations of the objects world state. The prior over the location of an object is uniform over 3-D space, except near another object, where the probability of placement decreases to Finally, the assumption of spatio-temporal continuity implies that any moving object must exhibit continuous trajectory. Our Experimental task focuses on inferring permanent fixtures in room, so we therefore do not implement this constraint, although doing so a solved problem (Smith et al.) Kemp & Xu 2009). Implementation details available in the Appendix.
2019
3.1.2
PERCEPTUAL MODULE
The goal of METAGEN is to build metacognitive representation of any object detection model that can transform images into object detections, each associated with a category and a position in the 2D image. Although METAGEN does not have access to ground truth (W), we assume access to the camera's ' state es, which determines its position in space and orientation. This helps with inferring and tracking the location of different objects (as this is not the main focus of our work). We additionally assume access to video boundaries, enabling METAGEN to reset object representations across scenes. Critically, the perceptual models functionally serves black box, and METAGEN does not have any access to the internal state of the model, knowledge about its performance, or any feedback about the final inferences.
Formally, let O = be the collection of observations associated with the t-th scene. Each observa- tion Ocs O, is percept associated with camera state (from the complete camera trajectory ej),
3

---

Under review as a conference paper at ICLR 2021
and consists of set of point detections obtained from the object recognition model, each containing the object's category and 2-D location on an image: = (c, F. y).
3.1.3 METACOGNITIVE REPRESENTATION
The metacognitive representation of the perceptual model, V, consists of two probability distri- butions per object category. The first distribution captures the model's propensity to hallucinate objects, represented through Poisson distribution with rate X for object category The second distribution captures the model's propensity to correctly detect an object that's in view. We represent this through a Geometric distribution with rate Pe, which helps account for the fact that object models can sometimes detect the same object multiple times. Under this formulation, the model's miss rate for an object in category is 1 PP--
Because each distribution is captured by a single parameter, the perceptual module's metacognition V can then be represented as a -yy2 matrix storing each category's hallucation rate de and miss rate 1 Pc- This defines a generative model such that V(O, W, cs) describes the probability that the visual system would produce percept O when processing world state W E W with the camera state cs. This is calculated by processing every object in W that is visible from camera state cs, through the metacognitive representation captured in V.
3.1.4 METACOGNITIV DYNAMICS
The generative model described so far captures how METAGEN learns metarepresentation of its perceptual module's performance. However, an object detection model's propensity to hallucinate or miss objects can vary across scenes, and leaving flexibility in V to be adjusted is desirable property. At the same time, experience in a previous scene includes critical information about the perceptual module that should inform expectations about its performance in therefore includes an evolving kernel Ge (Figure D]. capturing changing priors over V.
scene. Our generative model
When t 0, each category's 's initial hallucination rate Ae is initialized from Gamma distribution with parameters a = and 3 = 1. After inference in a scene, the prior over X evolves by computing the inferred number of hallucinations at time / (based the difference between world state W and observations O,), updating a and 3 as the conjugate prior over the Poisson distribution in the metacognition.
Beliefs about the miss rates evolve in an analogous manner. When & = 0, the detection rate Pa for each category is initialized from a Beta distribution with parameters a 1 and | (equivalent to a uniform distribution). The parameters from the Beta distribution are then updated as the conjugate prior over the Geometric distribution, based on the inferred missed detections (by comparing W. against Oz).
3.2 INFERENCE
Given collection of observations Ō - {0.)²= (i.e., multiple sets of percepts from multiple world states) and the corresponding camera trajectories oj. our goal is to infer Pr(V, ((,,))) cj), given by
T Pr Pr (O,W,, V,, ejt) Pr(V.)
Pr Pr (V, x II
(1)
The posterior, Eq. [ approximated via Sequential Monte- Carlo using particle filter. An estimate of the joint posterior can be sequentially approximated via:
T
Pr Pr(V, (10 = Pr (v.
II
Pr (o.li', Pr (W:') Pr (V
4

---

Under review as a conference paper at ICLR 2021
We
P(M)
umbrels
P(H)
Joint inference over W
and V
AOEA
T
PUM
Or
NN(W,)
W
Figure 2: Conceptual figure depicting inference.
where W1 ..... WT is the estimate Wi...., Wr after T observations, and V23 ...., estimate Vi,... VT after T observations. Here the transition kernel, Pr(V, [ O ') is gov- erned by the belief dynamics described in section 3.1.4
...,, Vrr the
3.2.1 INFERENCE PROCEDURE DETAILS
We sequentially approximate the joint posterior given in eq. 2 Jusing particle filter with 100 parti- cles. The solution space to the inference problem that we consider is sparse, and Sequential Monte- Carlo methods can suffer from degeneracy and loss of diversity in these situations. Our inference approach solves these problems by implementing particle rejuvenation over objects, locations, and beliefs about V. Rejuvenation conducted using series of Metropolis- -asopoiis--ssinngss moves with data- driven proposals designed to obtain samples from Pr(V, cO))).
During object rejuvenation, new state W/ is proposed by either adding removing object from the current state W,. Objects in W, have an equal chance of being removed to produce W7, and new objects added based data-driven distribution. This distribution samples object categories from categorical distribution biased towards categories observed in the current scene (see Appendix). With probability 0.5, location is sampled from a 3-D uniform distribution, and otherwise sampled from a data-driven function with location biased toward 3D points likely to have caused the 2D detections (see Appendix). The proposal for adding new object removing and existing object from the world state is accepted or rejected according to the MH algorithm.
After a proposed change to W, second rejuvenation step performed on locations, wherein an object in W is randomly selected (with equal probability) to have new location proposed. With probability 0.5, the new location is drawn from multivariate normal distribution centered on the previous location, and it is otherwise sampled from data-driven proposal (see Appendix). The proposed world-state W" with perturbed location then accepted or rejected according to MH algorithm.
Finally, new metacognition V' is proposed by perturbing V. Each parameter the -Cl-by-2 matrix perturbed, with new values generated for (i, and for (i..)) Vi,j. The new value for each (ii.j), is sampled from the appropriate distribution (Beta for misses, Gamma for hallucinations) with 04 and Br, while the new value for V (i is sampled with a:-1 and 3 These new values are accepted or rejected according to the MH algorithm.
This three-step rejuvenation process for world states, locations, and metacognition is done for each particle for 500 iterations and the last state reached the chain used as the new rejuvenated particle.
5

---

Under review as a conference paper at ICLR 2021
3.2.2 ESTIMATING V
After all T time steps in the particle filter, we estimate V1 by taking the expectation of the marginal distribution by averaging across particles weighted by their likelihood VJ, = =
M EM==((T lief about the true V after all observations have been observed. Given Vi T.e the posterior predictive distribution is defined as:
Lm). where m indexes the particles. This VI is the final estimate of the be-
T x
Pr
II Pr (jr Vú = -VT Pr (Ws) t=1
(3)
This posterior predictive distribution can then be used make better inferences about world states for novel scenes, Wr+1..... or for reassessing previous world states Wi,..., Wr, that were origi- nally inferring using less informed V.
EXPERIMENTS
To evaluate METAGEN, we created a dataset of trajectories navigating simulated indoor scenes, sampling several viewpoints per trajectory. We ran several object detection models on these images and tested METAGEN's ability to infer the underlying world states and properties of the object detection models causing the detections.
4.1 DATASET
We evaluate METAGEN on a dataset simulated indoor scenes observed from multiple perspec- tives by moving agent (Figure 2}. We generate scenes and render frames in the ThreeDWorld physical simulation platform (Gan et al. 2020). Each scene is initialized as an empty 13-unit-by- 13-unit room. The number of objects in the scene is uniformly drawn from the counting numbers up to 4 and each object is uniformly assigned one of object categories: potted plant, chair, bowl, Iv, or umbrella. We chose a canonical object model for each category. We sequentially place objects at locations drawn from uniform distribution, resampling if two objects are within 3 units of each other or the walls so as to avoid collisions.
The camera trajectory consists of camera positions and focal points. Roughly, the camera circles the room looking toward the center, with noise from Gaussian processes added to the camera position and focal points. The details and parameters are discussed in the Appendix. We generate frames by querying the camera for an image at linearly-spaced times.
We generated 100 videos in total, randomly split into two sets of 50 videos. The first set was used to test METAGEN's performance during online learning (Training), and the second set was used to test METAGEN's performance
novel set of scenes after halting inference over V (Test).
METAGEN's performance during online learning could be affected by the order in which the videos processed (for instance, METAGEN might benefit if the most challenging videos appear later in the sequence). Our results therefore average METAGEN's performance across four different order- ings of the videos. Videos were first randomly labeled from 50, and the four counterbalancee
orders were: (1 [1... 50). {50... 1), (26 50, 25). and {25 1, 26 50).
4.2 OBJECT DETECTION MODELS
To test METAGEN's capacity to learn and use a metacognition, we tested its performance using three modern neural networks for object detection as its perception module. The three networks were selected to represent a variety of state-of-the-arr architectures. In particular, we use RetinaNet. a one-stage detector (Lin et al. 2017); Faster R-CNN, a two-stage detector (Ren et al. 2013). and DETR, vision transformer (Carion et al. 2020). We process the outputs of the object detectors by filtering for the object categories present in the scenes and performed Man-Mmmm Suppression (NMS) with loU (Intersection over Union) threshold of 0.4 (applied only for RetinaNet and

---

Under review as a conference paper at ICLR 2021
Faster R-CNN, it is not typically used for DETR). As imput to METAGEN, took the top five highest-confidence detections per frame.
4.3 METRICS USED
performance in two ways: by testing whether can learn an accurate metacognition of the object detection module and 2) by testing whether the learned metacognition leads to improved accuracy.
We evaluate METAGEN's
To measure how well METAGEN learned a metacognition, we calculated the mean squared error (MSE) between the inferred visual system V and the true V. The true V was calculated for each object detection system. The MSE is given by
1 MSE
(oo - -o02 + (ee Dco)))
(4)
2|C1
aEC
where C' is the the set of object classes.
To measure METAGEN'S object detection accuracy, could use the standard mean Average Precision (mAP) metric because the outputs of METAGEN are points in 3D space rather than 2D bounding boxes. But we still needed a metric by which to compare METAGEN's ' object detection accuracy to those of the neural networks without metacognition.
To solve this problem, we projected METAGEN's inferences about the 3D location of objects to point on each 2D image, and for the neural networks, we took the centroid of the bounding boxes. This way, METAGEN and the neural networks' detections have the same format: an object label and point on the image.
To assess object detection accuracy on given frame, we took the Jaccard similarity coefficient (the Intersection over Union for sets) of the object labels that were detected and the objects that were actually present in give frame. The Jaccard similarity is given by:
ID n G
J(D, G)
ITD
U G
where D is the set of object labels that were detected (i.e. D (chair, chair, bowl)) and G is the set of objects in the ground-truth world state (i.e. G = {chair. bowl)). (In this example, [[D, C) = )
A limitation of this metric is that it does directly assess the spatial accuracy of the detections: only the categorical accuracy per frame. That said, highly inaccurate inferences about a 3D location would result in 2D projections onto the wrong frames, this metrie indirectly captures spatial accuracy. Averaging across frames yields an accuracy per video, and averaging across videos yields an overall accuracy.
COMPARISON MODELS
Throughout, METAGEN's videos (METAGEN Online Learning), and its performance when learning is halted and all inferences about world states are conditioned on the learned metacognition (METAGEN Learned V\. We test METAGEN Online Learning our Training set, and then test METAGEN Learned V on both the Training set (assessing its accuracy changes when can retrospectively re-evaluate its inferences after learning metacognition). and the Test set.
performance while it is learning a metacognition as it processes
To test if the learned metacognition confers benefit beyond what can be obtained from the neural networks alone, used two comparison models. The first comparison model (NN Output / META- GEN Input) consists the processed neural network output, which served as an input to METAGEN (Section 4.2
Our second comparison model (NN Output with Fitted Threshold) served as a more stringent test that compared METAGEN against the Neural Network output, fit to maximize accuracy. For each neural network, we fit the confidence threshold so as to maximize accuracy on the first 50 videos using grid-search. This required access the ground-truth object labels, giving advantage to
7

---

Under review as a conference paper at ICLR 2021
METAGEN Online Learning
Output Fitted Threshold
NN Output METAGEN Input
1.00
0.50
Avg. Diff in Accuracy
Avg. Accuracy
0.75
0.25
0.50
0.00
0.25
-.25
0.00
-0.50
10 20
30
40
60
Training Set Test Set
Training Set Test Set
Num videos observed
METAGEN Leamndd
1/
Figure 3: Results for METAGEN and comparison models with DETR. (A) METAGEN Online Learning MSE as a function of scenes observed. (B) Average accuracy for all models in Training Set and Test Set. (C) Accuracy difference between METAGEN Learned V and the two baseline models. Positive values indicated METAGEN outperforms the comparison models.
this comparison model over METAGEN. The values of the fitted threshold were 0.20 for DETR, 0.14 for RetinaNet, and 0.00 for Faster R-CNN.
6 RESULTS
To assess METAGEN's ability to learn accurate metacognitive representation V, we examined the MSE as function of the number of observed videos. Figure shows the dynamics of the MSE for V for the DETR model. 95% CIs over the MSE were calculated for each of the four runs by bootstrapping over the 100 particles. After observing just 10 videos, the MSE dropped from 0.035 0.014, and after training videos, the MSE at 0.009. Results MSE at 0.023) and Faster R-CNN (final MSE at 0.010). This demonstrates the speed with which METAGEN is able learn an accurate metacognition V, without access to ground-truth world states.
similar for RetinaNet (final
Does the learned metacognition enable METAGEN make better inferences about what's in scene? Figure 3B shows barplot depicting the accuracies of all four models (Section 5 using DETR. We find that METAGEN outperforms the neural networks even with the fitted confidence threshold. The error bars represent 95% confidence intervals calculated by bootstrapping over the videos. METAGEN outperformed the two comparison models for all three object detection models (Table
To evaluate the magnitude of the difference between METAGEN with Learned V and the two Neural Network comparison models (NN Output METAGEN input and NN Output with Fitted Thresh- old), we calculated average accuracy difference in accuracy and boostrapped the mean difference to obtain a 95% confidence intervals over performance difference. METAGEN showed a substantial improvement against both comparison models (Figure 3C for DETR; Table 2 for full results).
Finally, to confirm that METAGEN's success can be attributed to its learned metacognition, rather than purely to the Spelke object principles, we tested METAGEN with an erroncous metacognition consisting of low 12) miss rates and high (1.0) hallucination rates. Under these parameters, METAGEN only distinguish real detections from hallucinations based whether the pattern of detections across frames adheres to Spelke principles. This lesioned METAGEN led to a degenerate solution where all detections were always treated hallucinations, for all object detection models, with an accuracy of 0.2288, confirming that Spelke principles alone cannot account for METAGEN's success.
7 DISCUSSION
We proposed that metacognition metarepresentattion of one's computational processes can be powerful tool for increasing a system's 's robustness, enabling it to decide when to accept or reject proposed representations obtained from pre-trained systems. We presented an implementation
8

---

Under review as a conference paper at ICLR 2021
Table I: Main results
Perceptual Module
Model
Version
Avg. Accuracy Training
Avg. Accuracy Test
DETR
NN Output
Fitted threshold
0.361 (0.300, 0.422)
0.361 (0.313, 0.412)
METAGEN input
0.315 (0.257, 0.373)
0.292 (0.241, 0.346)
METAGEN
Online learning
0.609 (0.552, 0.667)
Learned V
0.562 (0.494, 0.627)
0.631 (0.569, 0.695)
RetinaNet
NN Output
Fitted threshold
0.582 (0.512, 0.650)
0.543 (0.462, 0.568)
METAGEN input
0.510 (0.452, 0.568)
0.539 (0.468, 0.609)
METAGEN
Online learning
0.671 (0.595,0.744)
Learned V
0.666 (0.594, 0.735)
0.647 (0.562, 0.730)
FasterR-CNN
NN Output
Fitted threshold
0.656 (0.589, 0.724)
0.630 (0.554, 0.705)
METAGEN input
0.636 (0.589, 0.724)
0.630 (0.554, 0.705)
METAGEN
Online learning
0.711 (0.639.0.775)
Learned V
0.697 (0.628, 0.762)
0.650 (0.563, 0.732)
Table 2: Accuracy difference between METAGEN learned V and NN output with fitted threshold
Perceputal Module
Avg. Difference on Test Set % Increase in Accuracy
DETR
0.270 (0.195, 0.350)
74.8%
RetinaNet
0.104 (0.0782, 0.130) 0.020 (0.000, 0.004)
19.2%
Faster R-CNN
3.1%
of this idea, METAGEN, applied to the context of inferring what objects were in a scene. Given set of observations by an object detection model with unknown performance, METAGEN performed joint inference over metacognitive representation of the system and over the objects causing the detections. We tested METAGEN using variety of modern object detection models, and found that METAGEN can quickly learn accurate metacognition and use it to correct errors from the classification system, improving the system's 's overall accuracy.
That improvement in accuracy was observed on custom dataset of synthetic images, in some ways quite different from those the object detection systems were trained on. Thus, our results show that METAGEN can be particularly helpful in improving the ability of an object detection system to generalize to out-of-sample data, especially when considering that METAGEN does not require any internal access the model knowledge of its performance metries. By learning metacognition for how the detection system performs on new dataset, METAGEN is able filter the outputs of the detection system to be tailored to new, out-of-sample data.
While our focus was on object detection, we believe that similar ideas can be applied to related domains. For instance, face-detection models could augmented with a metacognition that learns to represent its own accuracy, conditioned on an expectation that all faces should be, in principle, equally discriminable. This could allow METAGEN model learn its biases (e.g., such learn- ing that it has poor performance for faces from people of color). From this standpoint, METAGEN can also support explainable AI by generating simplified metarepresentations of a model's 's perfor- mance, which can be easily analyzed. Similarly, because METAGEN can help infer the presence of missed objects, or flag hallucinations an image, this approach could be fruitful as way of generating additional training data for self-supervised learning.
Finally, our work highlights how computational solutions that appear in humans can fruitful for approaching related problems in machine learning and AI. In recent years, models that capture human reasoning have received substantial attention
et al. 2019 Baker et al. 2017 Jara- Ettinger et al. 2019 Battaglia et al. 2013). By understanding how humans leverage representations of their cognitive processes to create more nuanced and accurate representations of the world, we may also be better able to design human-like artificial cognition.

---

Under review as a conference paper at ICLR 2021
REPRODUCIBILITY
To ensure reproducibility, we provide all of the code for the METAGEN model, producing the dataset, and the analyses conducted. Please see the anonymous git repo linked below for the code and for demo:
REFERENCES
Giduthuri Sateesh Babu and Sundaram Suresh. Sequential projection-based metacognitive learn- ing in a radial basis function network for classification problems. IEEE transactions on neural networks and learning systems, 24(2):1942066, 2012.
Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 144): 1-10, 2017.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 183745) 2013.
Susan Carey. The origin of concepts. Oxford university press, 2009.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michaes Frahm (eds.), Computer Vision ECCV 2020, pp. 213- 229, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58452-8.
Michael T Cox. Metacognition in computation: A selected research review. Artificial intelligence, 169(2))104-1411 2005.
Marco F. Cusumano- Towner, Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. Gen: A general-purpor purposse probabilistic programming system with programmable inference. Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDD 2019, pp. 221-236, New York, NY, USA, 2019. ACM. ISBN 978-144503-6712-77 doi: 10. (145/3314221.331444
URL http: doi. acm. org/10 10. 21....1155/331222..3333333
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks, 2017.
Chaz Firestone and Brian J Scholl. Cognition does not affect perception: Evaluating the evidence for "top-down" effects. Behavioral and brain sciences, 39, 2016.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint +xXiv:2007.04954 2020.
Alison Gopnik. How we know our minds: The illusion first-person knowledge intentionality.
Behavioral and Brain sciences, 16(1):1-14, 1993.
Umut Güçlü and Marcel AJ van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neurolmage, 145:329-336, 2017.
Timothy Hospedales, Antreas Antoniou, Paul Micaclli, and Amos Storkey. Meta-lcarning in neural networks: A survey, 2020.
Magdalena Ivanovska. Audun Jøsang, Lance Kaplan, and Francesco Sambo. Subjective networks: Perspectives and challenges. International Workshop on Graph Structures for Knowledge Rep- resentation and Reasoning, pp. 107-124. Springer, 2015.
10

---

Under review as a conference paper at ICLR 2021
Julian Jara- Eatanettineer, Laura Schulz, and Josh Tenenbaum. The naive utility calculus unified, quantitative framework for action understanding. PsyArXiv, 2019.
Lance Kaplan, Federico Cerutti, Murat Sensoy, Alun Preece, and Paul Sullivan. Uncertainty aware ai ml: why and how. arXiv preprint arXiv: 0tt/://80907822. 07882, 2018.
Kohitij Kar and James J DiCarlo. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition. Neuron, 109(1):
164-176, 2021.
Charles Kemp and Fei Xu. An ideal observer model of infant object perception. In Advances in Neural Information Processing Systems, pp. 925-832, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-11055, 2012.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference Computer Vision (ICCV), pp. 2999- 3007, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
Richard E Nisbett and Timothy D Wilson. Telling more than can know: verbal reports mental processes. Psychological review, 84(3):231, 1977.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference Computer Vision and Pat- tern Recognition (CVPR), June 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL http:: httpp:::ppoocceiinggg...... 14bfa6bb 88 4 www.bbbbbaaoeeeeeee 00t.:/6676655500222222224------0
pdf
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. In Advances Neural Information Processing Systems, pp. 3179-3188 3189, 2018
Kevin Smith, Lingjie Mei, Shunyu Yao, Jiajun Wu, Elizabeth Spelke, Josh Tenenbaum, and Tomer Ullman. Modeling expectation violation in intuitive physics with coarse probabilistic object rep- resentations. Advances in Neural Information Processing Systems, pp. 8983-8993, 2019.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96. 2007.
Kartick Subramanian, Sundaram Suresh, and Narasimhan Sundararajan. A metacognitive neuro- fuzzy inference system (mefis) for sequential classification problems. IEEE Transactions on Fuzzy Systems, 1006) 2013.
Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9)644-6665, 2017.
Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey, 2019.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
The METAGEN model and inference were in implemented in the Julia-based probablistic program- ming language Gen Sicuuanno-Townee et 2019
A.1 ADDITIONAL GENERATIVE MODEL PARAMETERS
Implementation of Spelke principles was as follows. World states without object persistence were not included in W, which is equivalent to implicitly setting their prior probability O. The sumption that two objects cannot occupy the same region in space was implemented through prior over W where the probability of having one object near another decreased according Gaussian distribution with x²
To account for noise in network's location detection O,, each detection modeled as having 2D) spatial noise, following a Gaussian with œz.y 40 pixels.
Finally, the full generative model requires specifying prior distribution over camera positions and focal points (although these are observable), set as uniform over 3D space, and a prior distribution over expected number of objects in scene, sampled from geometric distribution with parameter 0.9 (with uniform prior over category type).
A.2 DETAILS ABOUT THE INFERENCE PROCEDURE
During rejuvenation, new objects are proposed to be added to W: using data-driven distribution. The category of the object is comes from categorical distribution, where 10% of the weight is divided evenly among the categories, and the other 90% is divided proportionally to the number of times that object was observed the scene With probability 0.5, the location is sampled from a 3-D uniform distribution, and otherwise sampled from data-driven function. Using the data-driven function, the point is sampled based on proximity to the line-segment that, when projected onto the 2D image, would result in the point where the detection was observed. The probability of proposing a particular point decreases with the distance from this line segment, following a Gaussian with _² 0.01.
In the second rejuvenation step a new location is proposed for an object. With probability 0.5, the new location drawn from multivariate normal distribution centered the previous location, with Zz...z 0.01. Otherwise, it is sampled as described in the previous paragraph (based on proximity the line segment that would results in the detection's 2D location).
A.3 DETAILS ABOUT THE CAMERA TR TRAJECTORY IN THE DATASET
In our dataset of simulated indoor scenes, the agent's trajectory consists of a series camera po- sitions and focal points. The camera trajectory is a circular path around the periphery of the room with noise generated by Gaussian process with an RBF kernel with o = 0.7 and & = 2.5. The height of the camera is held constant y 2. The focal point trajectory is Gaussian process with a mean above the center of the floor and component-wis parameters o = 0.7 and 2. Figure shows some example trajectories.
12

---

Under review as a conference paper at ICLR 2021
0
4
Figure 4: Three sampled agent trajectories. The wide circular patterns at the top are camera posi- tions, and the smaller patterns near the bottom are camera focal points.
13


Reviews:
[{"review_id": "xptab57VVMn", "reviewer": "Reviewer_Rg2j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of this article is a very relevant one: How to account for an AI inaccuracy for decision processes? The proposed idea of a metacognitive component is a well inspired one. The proposed model is sound and well described, but simple: object detection algorithms do not typically have flat probabilities of hallucination or miss, but tend to be more likely to miss objects that are further away and viewed under specific angles. In a realistic scenario, specific instances of an object class may be more prone to misses. The proposed model does not account for that. \n\nAnother issue with the article is the very synthetic nature of the experimental setup. The history of computer vision tells us that results showed on artificial data rarely transfer to real-world situations. In particular, as the authors note, the chosen scenario is quite far from the data the detectors were trained on and the performance reported seems quite low for such a simple problem. Some form of domain adaptation or fine-tuning may be required here to compensate for this and make the baselines more realistic. \n\nAlthough it is a good idea to use a simplified scenario for experimental purposes, I have some concern that the performance improvement may only be due to the artificially low performance of the baseline algorithms on this task. The fact that the error metric is non-standard and does not account for localisation makes it more difficult to really assess how meaningful are the results. \n\nAdditionally, it is unclear from the experimental results that the metacognitive model could be generalised easily to other scenarios due to the low variety in conditions between the training and test data. \n", "summary_of_the_paper": "This article proposes a metacognition model for object detection. In this article, metacognition is understood as the ability for a system to monitor its own beliefs and account for its own accuracy. The proposed model learns the probability for a detector to hallucinate (false positive) or fail to detect an object. To this end, the model assumes some prior world knowledge drawn from Psychology: the Spelke principles: Object persistence, continuous motion and non-colocation of objects. \n\nThe proposed model is trained and tested on a synthetic dataset and the experimental results show an improvement in detection performance compared to standard models. ", "summary_of_the_review": "\nIn summary, this is an interesting research topic, but the significance of the article is limited by the rather simple model choices and experimental setup. Because of those choices, it is unclear that the proposed method would generalise to other setups. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bEhHRiVGht", "reviewer": "Reviewer_1uYX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the new datasets collected by the author, the model achieves significant improvement based on the existing detection algorithms. In addition, the idea of using METAGAN to deal with object detection proposes an entire new perspective to address the detection problem, which is very interesting to me. \n\nMy main concern to accept this paper is that the experiments are not convincing. The problem setting in this paper is quite different with the other papers for object detection, and the dataset for evaluation is more like a toy dataset, with simple contexts and limited images for training. For object detection, the complicated context is one of the main challenges, and it is unfair to compare models with detectors for 2D images since they are not designed to capture the trajectory between each pair if the frame. I wonder whether the author can report the results of the proposed module based on the large-scale public video detection datasets (e.g., ImageNet VID [1]) , and compare the results with the detectors for video detection [2] [3], where the models can also capture the relationship between different frames. It will make the proposed framework far more convincing. \n\n[1] Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge[J]. International journal of computer vision, 2015, 115(3): 211-252.\n\n[2] Han M, Wang Y, Chang X, et al. Mining inter-video proposal relations for video object detection[C]//European Conference on Computer Vision. Springer, Cham, 2020: 431-446.\n\n[3] Gong T, Chen K, Wang X, et al. Temporal ROI Align for Video Object Recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(2): 1442-1450.\n\n", "summary_of_the_paper": "This paper proposes a new METAGAN module which captures the relationship between objects and detections in each time slot, and it refines the detection results by eliminating the false predictions. \n", "summary_of_the_review": "Overall, my point is on the fence. As the limitation I mention above, my pre-rebuttal score is slightly below the acceptance bar. If the author can validate the effectiveness of the proposed method on the challenging public benchmarks, I'd like to change the score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XEHe8JIxjXu", "reviewer": "Reviewer_RfQP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1. I think the motivation behind the paper is strong and is a good fit for ICLR. The idea of being able to take a noisy object detector and learn how to denoise its outputs using a set of sensible priors, without retraining or needing to know the inner workings of the object detector itself, is a simple and attractive one to me. The paper does a good job of explaining this motivation and framing the work.\n2. The METAGEN model proposed in Section 3 is clearly explained and seems like a reasonable approach to impose the various desirable metacognitive priors.\n3. The experiment uses a range of object detectors to show that the proposed METAGEN model can provide benefit to any generic noisy black-box detector.\n4. The work appears to be fully reproducible via code at https://anonymous.4open.science/r/MetaGen-0391/ \n\n# Weaknesses\n\n1. What motivated the choice of dataset? Given the simplicity of generating data, why not make more complex scenes, and more complex trajectories? For example, are there any object-to-object occlusions in the dataset? Based on the videos and trajectories provided, it seems possibly not? These are interesting because they can easily confuse detectors and are a good motivation for trying to introduce notions of object permanence into models. Does the model hold up in more cluttered environments where real occlusions occur? The approach would be, I think, considerably more compelling if it was shown to work on more than one dataset, particularly without significant tuning of METAGEN hyperparameters between the two. \n\n2. How were the initial three models trained? I wonder if the main difference in improvement by applying METAGEN is more to do with how strong those models were to begin with. It might be interesting to show e.g. what is the relative improvement if all three models are pre-trained to the same level of test performance? It might also be interesting to see what happens as the detection models are given different amounts of pre-training. How does the performance vs training data size curve look for the models with and without METAGAN?\n\n3. The robotics community has been using methods like particle filtering for quite a long time to deal with issues of unreliable sensing / perception. Although they do not frame their work in the same metacognitive light, the end effect is somewhat similar - the perception model provides noisy frame-by-frame inputs while a dynamics/state space model is used to smooth the estimates enforcing notions like object permanence. I'm blanking on a good reference that's closest to this work, but it would be nice to see some effort to make a connection here. \n\n4. There are certain experimental details which I think could be added to improve understanding.  For example, the order of training scene samples clearly has an effect, but how critical is it? What are the effects of change non-maximum suppression / not filtering to top five highest-confidence detections per frame? How sensitive are the results to the various hyperparameters involved in the SMC optimization?\nWere there any observed or hypothetical failure modes of this metacognitive approach, where it might actually worsen the outcome? One can imagine it could be vulnerable to certain kinds of cascading error, for example.\n\n5. Sec 5 - for the second comparison model, is this thresholding the softmax classification output of the detector? If so, is the 0.00 fitted threshold value for Faster R-CNN rather surprising? Is it close to being perfect in detection performance?\n\n6. I’m unclear why object label-level Jaccard similarity was chosen as the metric. I understand that it would indirectly capture some notion of spatial accuracy, but could this be done more explicitly? e.g. by also measuring whether projected objects or predicted bounding box centroids lie within the ground truth bounding box?\n\n# Minor points\n\n1. Fig 2 caption chould ideally be self-explanatory without referring to text. Why is plant in blue in top panel? \n\n2. The two different colors in Fig 3C is hard to see - maybe use dashed outlines to make it more obvious?\n\n3. Table 2 Faster R-CNN row has a numerical typo\n\n4. It would be good to see Fig 3 for the other two models for comparison, perhaps in the Appendix if they are not particularly interesting?\n\n5. “the assumption of spatio-temporal continuity implies that any moving object must exhibit a continuous trajectory… we do not implement this constraint, although doing so is a solved problem” -- I think calling this “solved” may be a bit reductive; could this statement be qualified a little more?\n\n6. Misc typos; “removing and existing” -> an; “were originally inferring” -> inferred\n\n7. The bounds in Tables 1 and 2 could be simplified to a single +/- number; also probably could be shown to 2 decimal places to improve readability.", "summary_of_the_paper": "This paper proposes a metacognitive framework to augment object detection models to help improve them without human input.\nThe framework is based on learning to detect and fix a model’s output if it violates early object constraints (Spelke principles), namely object permanence, cohesion, and spatiotemporal continuity.\nThe framework, called METAGEN, does not require access to the internal state of the detection model but operates only on its outputs, maintaining a metacognitive representation which summarizes the model’s performance (propensity to hallucinate or incorrectly miss objects of different classes).\nThe representation is initialized for a given scene from a learned and evolving prior, and is updated along with the estimated world state using a particle filter with various rejuvenation steps over objects, locations and metacognitive parameters.\nMETAGEN is tested using single-stage (RetineNet), two-stage (FasterR-CNN), and transformer (DETR) models of object detection within a dataset gathered using the synthetic ThreeDWorld environment, and is shown to improve test performance both during inference and after stopping inference on both training and new test scenes.\n", "summary_of_the_review": "I think this is an interesting paper that could merit publication but I am concerned that the experimental side is lacking for the reasons mentioned above. I would say it is below the threshold for publication as I read it right now, but I look forward to hear clarifications from the authors and will take into consideration the Q&A with all the reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2GFTq4YyfFg", "reviewer": "Reviewer_RHiF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\n\nThe paper introduces an interesting formulation on how to incorporate various consistency constraints in video object detection. The proposed METAGEN framework is simple and intuitive, and can easily be extended to other applications.  \n\n$\\textbf{Weaknesses}$\n1. I am not sure if the formulation used in the paper can be treated as a traditional object detection problem. The authors use a video and take into account the camera trajectory, which is additional information unavailable to the baseline 2D detection method. Therefore directly comparing to 2D object detection approaches seems unfair. Additionally, the use of a video makes it closely aligned to object tracking or video object detection, where the ideas of enforcing cohesions and spatiotemporal continuity have previously been explored. (For example [a], [b], and [c]). The authors should discuss these (and possibly other) methods in their related work, and highlight how their proposed approach is different. Additionally, comparisons to these methods that utilize video/camera information would be a better baseline.\n\n2. The experiments are only shown on a single synthetic dataset, which is extremely small in size (only 100 videos). Additionally, the diversity in objects in limited (5 objects), and the generated scenes don't account for background noises (which is what you would get in a real world setting).\n\n3. Qualitative results showing how the proposed approach improves on the baseline detections is missing. Even though there is an improvement in the numbers, its hard to understand how these improvements translate visually. \n\n4. Evaluating the detection accuracy using only centroids makes it hard to correctly understand the improvements in detection performance as it gives no sense of the bounding box area. As you already assume you know the camera position, it should be possible to map the 3D bounding box to a 2D box. Additionally, as mentioned in the paper, the accuracy metric computed doesn't take into account the spatial correctness. Is there a reason why the authors did not compute the L1 distances between the projected 1D bounding box point and ground truth?\n\n5. I am confused how the true $V$ is obtained to compute the MSE given in Equation (4). It would be great if the authors could clarify this.\n\n[a] Zhu, Xizhou, et al. \"Flow-guided feature aggregation for video object detection.\" Proceedings of the IEEE International Conference on Computer Vision. 2017.\n\n[b] Bertasius, Gedas, Lorenzo Torresani, and Jianbo Shi. \"Object detection in video with spatiotemporal sampling networks.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[c] Wang, Qiang, et al. \"Fast online object tracking and segmentation: A unifying approach.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n", "summary_of_the_paper": "The paper proposes a new method to incorporate Spelke's principals of object perception as constraints to improve the performance of an out-of-the-box object detector. This is done via defining a hierarchical generative model which defines \"metacognitive\" priors over the a set of observations. Through joint inference over these metacognitive priors and new unobserved states, the method outputs better object detections. The authors show improved performance on a synthetic dataset which contains scenes rendered in a virtual environment.", "summary_of_the_review": "The paper presents an interesting way of enforcing consistency constraints in object detection. However, I feel that the experimental evaluation and overall novelty when compared to existing approaches in video object detection/tracking is lacking. Therefore, I feel that this paper, in its current state, is not ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates a metacognition model for object detection. Reviewer RHiF wrote the best summary for this work:

The paper proposes a new method to incorporate Spelke's principles of object perception as constraints to improve the performance of an out-of-the-box object detector. This is done via defining a hierarchical generative model which defines "metacognitive" priors over the a set of observations. Through joint inference over these metacognitive priors and new unobserved states, the method outputs better object detections. The authors show improved performance on a synthetic dataset which contains scenes rendered in a virtual environment.

All reviewers agree that this is a really novel and interesting approach of enforcing consistency constraints in object detection, but had various issues with the experiments. At its current state, I believe it would make a very strong workshop paper, but not read for the ICLR 2020 conference. The authors found the reviews to be helpful, in particular, advice about the dataset construction and metric definitions, and I believe that future versions of this work will be significantly improved. We look forward to reading a revised version of this work in a high impact journal or future ML conference, good luck!
Decision: Reject