Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper defines an entity-monitoring problem where the goal is to identify the distinct objects see in an episode where the agent/model moves through the scene/observes partial state. The paper proposes the OBM-Net model architecture to address this problem and identify all the distinct objects observed over each episode. The key idea of OBM-Net to have fixed set of slots which allow tracking multiple hypothesis over time and use an attention mechanism to update the slots with observations over time. 
main_review: The entity-monitoring problem as formulated in the paper does not seem to pose the challenges outlined in the intro. Post reading the intro I was expecting to see something akin to really long-term tracking with associations across objects even when there are huge gaps in temporal coherence in environments with significant distractors. In contrast, the problem setup focuses on very limited settings with fairly low complexity. 

"In particular, we train a system to construct a memory of the objects in the environment, without prior models of the robot’s sensing, the types of objects to be encountered, or the patterns in which they might move in the environment. "

I am not sure what without prior models of the robot's sensing means. Since the model is being trained on the agent's input. It is just a question of when the learning is happening (online vs. offline). Since the training is being done online, one of the things that is happening is learning a feature representation for clustering/slotting similar objects together. Why does this feature representation need to be learnt online? Why can we not use a pre-trained representation trained in a self supervised manner with observations collected from the robot. Once we have such a representation how would proposed method compare to just clustering observations on a pre-trained representation. For instance in the qualitative examples shown in Figure 5 I would be surprised if unsupervised pre-trained representations cannot make the same association. 

From an experimental stand point all the tasks seem fairly non-standard due to definition of the task. There are other datasets like CATER https://rohitgirdhar.github.io/CATER/ which setup versions of the problem. Why setup a different problem rather than use CATER or some of the long term tracking benchmarks https://github.com/wangdongdut/Long-term-Visual-Tracking which involve object re-association?

From an architecture and modeling standpoint there does not seem to be significantly different from https://arxiv.org/pdf/2006.15055.pdf. Moreover, the paper does not evaluate on the same benchmark tasks Locatello et.al evaluate on. What is the reason for this? Evaluating on the same benchmarks would allow comparison of differences in modeling details more effectively since the evaluation OBM-Net is empirical. 
summary_of_the_review: The entity-monitoring problem as evaluated in the paper does not pose the challenges outlined in the intro. I am not convinced that the proposed method is a better alternative for associating/clustering objects compared to using pre-trained representations on data from the corresponding domains.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an end-to-end system for the data association and filtering (DAF) problem. The architecture is built to mimic components typically found in DAF systems to provide a sort of algorithmic prior to the network. The resulting system is evaluated on several different tasks using synthetic data.
main_review: The idea of encoding "algorithmic priors" into deep learning architectures has been explored in the past for various problems and usually is a sound idea and provides benefits to aspects such as complexity, efficacity, or performance of the resulting method over a pure end-to-end learning-based method. As such, the idea proposed in the paper is sensible and has merit.

Despite this, there are several issues with the paper in its current state. One of them is the apparent disconnect between the data association based description and motivation in the beginning, and the clustering focused experimental validation. While it might be reasonable to consider certain aspects to be clustering-like, these similarities should be described and motivated. There also appears to be no comparison to any DAF method to be presented.

The description of the method itself, while understandable at a high level, lacks detail and specifics. The method seems to utilize various deep learning "building blocks" such as slots, attention, etc., without introducing what they are or how their function captures algorithmic aspects. The capturing of algorithmic aspects is another problematic part of the method description, as the algorithm that is being reproduced is never actually described or shown. Therefore, the reader is left wondering what aspects are being replicated and why.

The experiments are very disconnected from the paper's motivation, and their link to the initially described task is unclear. The second experiment on MNist and planes also seems to be more about the presence or absence of rotational invariance of the employed embeddings, more so than the method itself. Making the comparison confusing and purpose unclear. Another problem with the clustering based evaluation is the metric being used. L2 or similar distance metrics are simply not useful metrics when evaluating clustering results and thus a widely accepted metric such as normalized mutual information would be appropriate to evaluate these results. The last experiment, which seems to be more in the direction of the initially described problem does not appear to have an actual DAF method included in the baselines. Given that the proposed method seemed to be replicating a specific DAF method I would have expected to see at least that method as a baseline.
summary_of_the_review: The general idea of replicating traditional algorithm designs in a deep learning architecture is sensible the lack of detail in the method description and unclear experimental design and evaluation means that the paper is not in a publishable state at this stage.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the “entity monitoring” problem. They propose a neural-network architecture that uses self-attention as a mechanism for data association to construct a memory of the objects in the environment and demonstrate its effectiveness in a set of illustrative problems.
main_review: In general, the paper is easy to follow, the evaluation proves some potential of the proposed method. However, there are some critical concerns of the paper listed below: 

A.	This paper addressed the problem “entity monitoring”. Based on the problem formulation, it is very related to the object re-identification and instance retrieval problem. How does it differ? I would expect some highlights on this regard in the introduction and related works. 

B.	All experiments are performed with synthetic dataset. This does not expose much real-world challenges. Section 5.2 only evaluates on a single object category for 3D objects, i.e.  the airplane. However, often the problem at hand requires the method to handle multiple instances of multiple categories. In an indoor environment, you can easily find multiple chairs in a room together with many other categories of objects. Ref 1) below performed a similar evaluation, but with the full 40 categories of ModelNet40.  Moreover, another critical point from my view is the lack of evaluation on real-world dataset. One can extract the image crops corresponding to each object captured from different viewpoints from some public 3D scene dataset, e.g. ScanNet (http://www.scan-net.org/), as how it is done in Ref 2) below.

C.	In line with the above points, there are methods that are closely related but are not covered in the discussion and in the comparison. Some recent related works are listed below for references: 
1)	Bai, Song, et al. "Re-ranking via metric fusion for object retrieval and person re-identification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
2)	Bansal, Vaibhav, Gian Luca Foresti, and Niki Martinel. "Where Did I See It? Object Instance Re-Identification With Attention." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
3)	He, Shuting, et al. "Transreid: Transformer-based object re-identification." ICCV 2021.
4)	Zhang, Yifu, et al. "Fairmot: On the fairness of detection and re-identification in multiple object tracking." International Journal of Computer Vision (2021): 1-19.
5)	Zhang, Xiao, et al. "Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
6)	Huang, Tsung-Wei, et al. "Multi-View Vehicle Re-Identification using Temporal Attention Model and Metadata Re-ranking." CVPR Workshops. Vol. 2. 2019.
summary_of_the_review: To summaries, the current submission does not situate the method well in the literature because some closely related works are not covered in the paper. The evaluation is also not convincing in terms of how the method can handle real-world challenges and lacks evaluation of real-world data. I therefore vote for a rejection.
review_title: null

Meta-review: This paper proposed a long-term object-based memory system for robots.  The proposed method builds on existing ideas of data association filters and neural-net attention mechanisms to learn transition and observation models of objects from labelled trajectories.  The proposed method was compared with baseline algorithms in a set of experiments.

The initial reviews raised multiple concerns about the paper. Reviewers nrGQ and  V7qP commented on the conceptual gap between the problem proposed in the introduction and the extent of the experiments. Reviewer qPet understood the paper to be a form of object re-identification and was concerned about the limited comparisons with related work.  The author response clarified their goal of estimating the states of the objects in the world, which they state is different from the goals of long-term tracking and object reidentification mentioned by the reviewers.  The authors also clarified the relationship to other work in slot-attention and data association filters.  

The ensuing discussion among the reviewers indicated that the paper's contribution remained unclear even after the author response. Two reviewers noted the paper did not clearly communicate the problem being solved (all reviewers had a different view of the problem in the paper).  These reviewers wanted a better motivation for the problem being addressed in this paper.  The third reviewer remained unconvinced that the problem in the paper was different from long-term object tracking.

Three knowledgeable reviewers indicate reject as the contributions of the paper were unclear to all of them. The paper is therefore rejected.
Decision: Reject