Learning an Object-Based Memory System
Authors: Yilun Du
Joshua B. Tenenbaum
Tomás Lozano-Pérez
Leslie Pack Kaelbling
Venue: ICLR 2022 Submitted (2022)
Keywords: Long Term Memory
State Estimation
Robot Learning
Object-Centric Learning

Abstract: A robot operating in a household makes observations of multiple objects as it moves around over the course of days or weeks.  The objects may be moved by inhabitants, but not completely at random.  The robot may be called upon later to retrieve objects and will need a long-term object-based memory in order to know how to find them.  In this paper, we combine some aspects of classic techniques for data-association filtering with modern attention-based neural networks to construct object-based memory systems that consume and produce high-dimensional observations and hypotheses. We perform end-to-end learning on labeled observation trajectories to learn both the internal transition and observation models.  We demonstrate the system's effectiveness on a sequence of problem classes of increasing difficulty and show that it outperforms clustering-based methods, classic filters, and unstructured neural approaches.

Under review as a conference paper at ICLR 2022
LEARNING AN OBJECT BASED MEMORY SYSTEM
Anonymous authors Paper under double-blind review
ABSTRACT
A robot operating in household makes observations of multiple objects it moves around over the course of days or weeks. The objects may be moved by inhabitants, but not completely at random. The robot may be called upon later to retrieve objects and will need long-term object-based memory in order to know how to find them. In this paper, we combine some aspects of classic techniques for data-association filtering with modern attention-base bbsed neural networks to construct object-based memory systems that consume and produce high- nigh-dimensional observations and hypotheses. We perform end-to-end learning on labeled observation trajectories to learn both the internal transition and observation models. We demonstrate the system's 's effectiveness show that it outperforms clustering- based methods, classic filters, and unstructured neural approaches.
sequence of problem classes of increasing difficulty and
INTRODUCTION
Consider a robot operating in a household, making observations of multiple objects as it moves around over the course of days weeks. The objects may be moved by the inhabitants, even when the robot is not observing them, and we expect the robot to be able to find any of the objects when requested. We will call this type of problem entity monitoring. It occurs in many applications, but we are particularly motivated by the robotics applications where the observations are very high dimensional, such images or point clouds. Such systems need to perform online data association, determining which individual objects generated each observation, and state estimation, aggregating the observations of each individual object to obtain a representation that is lower variance and more complete than any individual observation. This problem can be addressed by an online recursive filtering algorithm that receives a stream of object detections as input and generates, after each input observation, a set of hypotheses corresponding to the actual objects observed by the agent.
When observations are closely spaced in time and objects only briefly go out of view, the entity monitoring problem becomes the well studied problem of object tracking. In contrast, in this paper, we are interested studying the more generalized entity monitoring problem, where a robot must associate set of sparse and temporally separated observations of objects over the course of days or weeks into a coherent estimate of the underlying objects and associated properties in a scene (Figure 1). In such setting, it is important that the system does not depend continuous visual tracking, as any individual object may be seen at one time and then again significantly later. A sub-proble toobproblem of generalized entity monitoring corresponds to object identification, which we seek to consistently re-identify objects across time. However, to solve the generalized entity monitoring problem, system must not only identify similar objects across time, but integrate the observations into an estimate of their properties that may not be directly inferrable from any single observation.
classical solution the entity monitoring problem, developed for the tracking case but extensible to other dynamic settings, is a data association filter (DAF) (the tutorial of Bar-Shalom et al. (2009) provides good introduction). Bayes-optimal solution to this problem can be formulated, but requires representing a number of possible hypotheses that grows exponentially with the number of observations. A much more practical, though less robust, approach is a maximum likelihood DAF (ML-DAF), which commits, maintains a set of object hypotheses, one for each object (generally starting with the empty set) and for each observation it decides either: (a) associate the observation with existing object hypothesis and perform a Bayesian update on that hypothesis with the new data, (b) start a new object hypothesis based on this observation, or (c) discard the observation noise. As the number of entities
each step, to maximum likelihood data association: the algorithm

---

Under review as a conference paper at ICLR 2022
Start
End
Observations (One Segmented Depth Map per Timestep)
House Plan
Figure I: (left) Example observations to the OBM system. At cach time-step, the OBM obtains segmented depth map of single object. (right) Example domain layout. Sample layout with robot trajectory, field of view (in yellow) and tables that can contain objects. Objects in the domain can they are on as well as to different lables (simulating perturbations induced by the inhabitants). The robot moves through the environment acquiring local, partial observations of objects and must predict the number, location, and shape of objects it has seen.
both locally on the table
in the domain and the time between observations of the same entity increase, the problem becomes more difficult and the system can begin to play the role of the long-term object-based memory (OBM) for an autonomous agent.
The engineering approach to constructing such an OBM requires many design choices, including the specification of a latent state-space for object hypotheses, a model relating observations to object states, another model specifying the evolution of object states over time, and thresholds or other decision rules for choosing, for a new observation, whether to associate it with an existing hypothesis, use to start a new hypothesis, or discard it. In any particular application, the engineer must tune all of these models and parameters to build an OBM that performs well. This is time-consuming process that must be repeated for each new application.
In this paper, we develop a method for training neural networks to perform as OBMs for dynamic entity monitoring. In particular, we train a system to construct a memory of the objects in the environment, without explicit models of the robot's 's sensors, the types of objects be encountered, or the patterns in which they might move in the environment. Although it is possible to train an unstructured recurrent neural network (RNN) to solve this problem, find that building in some aspects of the structure of the OBM allows faster learning with less data and enables the system to address problems with a longer horizon. We describe a neural-network architecture that uses self-attention as a mechanism for data association, and demonstrate its effectiveness in several illustrative problems. We first validate that OBM can estimate object states when observations are drawn online from a set of cluster centers. Next, we validate that OBM can estimate object states when observations correspond to high-dimensional images. Finally, we illustrate its application on a realistic simulated robotic domain.
2 RELATED WORK
Online clustering methods In the simple setting, where object state does not change over time, the entity monitoring problem can be seen as a form of online clustering, where the assignment of data points to clusters is done online, with observations arriving sequentially and cumulative set of hypotheses output after each observation. One of the most fundamental online clustering methods is vector quantization, articulated originally by Gray (1984) and understood stochastic gradient method by Kohonen (1995). It initializes cluster centers at random and assigns each new observation the closest cluster center, and updates that center to be closer the observation. We show that our approach can learn to outperform this online clustering method. More recent work has explored theoretical aspects of online clustering with guarantees (Liberty et al., 2016; Bhaskara and Rwanpathirana, 2020; Cohen-Addat et al., 2021).
Data-Association Filters The most classic filter, for the case of a single entity, is the Kalman filter (Welch and Bishop, 2006). In the presence of astooccsoiitionn uncertainty the Kalman filter can be extended by considering assignments of observations to multiple existing hypotheses in a DAF or ML-DAF. These approaches, all of which require hand-tuned transition and observation models, are described by (Bar-Shalom et al., 2009). We show that our approach can learn the underlying transition and observation models and performs comparably to ML-DAF with ground truth system dynamic and observation models.
Visual data-association methods A special case of the entity monitoring problem where observa- tions are closely spaced in time has been extensively explored in the visual object tracking setting (Luo et al., 2014; Xiang et al., 2015; Bewley al., 201 60;6; Frossard and Urtasun, 2018; Brasó and
2

---

Under review as a conference paper at ICLR 2022
Leal-Taixé, 2020). In these problems, there is typically fixed visual field populated with many smoothly moving objects. This enables some specialized techniques that take advantage of the fact that the observations of each object are typically smoothly varying in space-time, and incorporate additional visual appearance cues. In contrast, in our setting, there is no fixed spatial field for obser- vations and they may be temporally widely spaced, as would be the case when robot moves through the rooms of a house, encountering and re-encountering different objects as it does so. While work has studied the detection of repeated objects with similar appearance (Girdhar and Ramanan, 2019; Bai et al., 2019; Bansal et al., 2021; He et al., 2021: Zhang et al., 2021 Huang et al., 2019), our focus is on aggregating and estimating the individual states of objects based on substantially different observations in a different space, and our methods are not competitive with specialized techniques on the much more specialized problems of
tracking or object re-identication.
Learning for data association There is relatively little work in the area of learning for generalized data association, but Liu et al. (2019) provide a recent application of LSTMs (Hochreiter and Schmidhuber, 1997) to rich version of the data association problem, in which batches of observations arrive simultancously, with a constraint that each observation can be assigned to at most one object hypothesis. The sequential structure of the LSTM is used here not for recursive filtering, but to handle the variable numbers of observations and hypotheses. is assumed that Fuclidean distance is an appropriate metric and that the observation and state spaces are the same. Milan et al. (2017) combine a similar use of LSTM for data association with a recurrent network that learns to track multiple targets. It learns dynamics model for the targets, including birth and death processes, but operates in simple state and observation spaces.
Slot Based and Object Centric Learning Our approach to the dynamic entity monitoring task relies on the use of attention over set of object hypothesis slots. Generic architectures for processing such slots can be found in (Shi et al., 2015; Vinyals et al., 2015; Lee et al., 2018), where we use (Lee et al., 2018) 8) as a point of comparison for OBM. We note that these architectures provide generic mechanisms to process sets of inputs, and lack the explicit structure from DAF we build into our model. Our individual hypothesis slots correspond to beliefs over object hypotheses, and thus relates to existing work in object-centric scene learning. Such work has explored the discovery of factorized objects from both static scenes (Burgess et al., 2019; Greff al., 2019; Locatello et al., 2020). Developed concurrently and most similar to our work, (Locatello et al., 2020) also utilizes slots as a means of representing factorized object decomposition of static images. In contrast to (Locatello et al., 2020), our work focuses on the use of a set of slots to represent the evolution of uncertain object hypotheses over time, and incorporates attention and inductive biases from DAF to selectively update beliefs across time to obtain object hypotheses as well as their associated confidences.
Algorithmic priors for neural networks One final comparison is to other methods that integrate algorithmic structure with end-to-end neural network training. This approach has been applied to sequential decision making by Tamar et al. (2016), particle filters by Jonschkowski et al. (2018), and Kalman filters by Krishnan et al. (2015), as well as to complex multi-module robot control system by Karkus et al. (2019). The results generally are much more robust than completely buil- models and much more sample-efficient than completely unstructured deep-learning. We view our work as an instance of this general approach.
3 PROBLEM FORMULATION
We formalize the process of learning an object-based memory system (OBM). Formally, when the OBM is executed online, it receives a stream of input observations 21,... 27 where 2, E RRd, and after each input 2,, it will output two vectors representing set of predicted properties of hypothesized objects y, = [ .Y) and an associated confidence score for each hypothesis, C = [Cea (E()... where YıA E ddd, Cu (0, 1). To ensure that confidences are bounded, we constrain E Cek 1. We limit the maximum number of hypothesis "slots" in advance to K. Dependent on the application, the 2 and y values may be in the same space with the same representation, but this is not necessary.
We have training data representing N different maitt-mmnitoring problem instances, D {(=(0) m, (t) ( (ee(L....) (..))ee(....)) where each training example is an input/output sequence of length L, each element of which consists of a pair of input x and m = {m, (()) which is a set of nominal object hypotheses representing the true current state of objects that have actually been
observed so far in the sequence. It will always be true that (i) C m and j((() < K because the set of objects seen so far is cumulative.
3

---

Under review as a conference paper at ICLR 2022
transition
Algorithun OBM-Net execution.
Observation
Input: observations 2t....
Slot Stnte 92
affend suppress
count n E R state e RKx
update
for timestop =1oo
Slot Count nig
encode(2:)
relevance(e, 5.
Ohs Fmbee
for slot to K de nu. suppressiattende
Update uy
S:
de +- updatet mi, rou (au ra.
Attention agg
4
ny
Relevance
0A
decode( sh..
Output Y/:
transition (4))
Confidence Cj
encode
relevance
end for
end For
Figure 2: Architocture and pscudocode of OBM-Net. Observations are fed sequentially to OBM-Net, and encoded with respect to each hypothesis. A subset of the hypotheses are updated at each time-step, with corresponding slot counts incremented according to attention weight. Slots then decoded, with the confidence of an output proportional underlying slot count.
Our objective is to train a recurrent computational model to perform as an OBM effectively in problems that are drawn from the same distribution over latent domains those in the training set. To do so, we formulate model (described section 4) with parameters 0, which transduces the input sequence 21...... 2L into an output sequence (y1, (1)...,..... (L), and train it to minimize the following loss function:
N
[10 D)
Lobs(yt m, + Lator (y.)) (0) (.))
) Laparso (c.
The Lobj term a chamfer loss (Barrow et al., 1977), which looks for the predicted yee that is closest each actual my and sums their distances, making the model has found good, high-confidence representation for each true object, with < 1:
Lonj(u, e, m)) Emin min 1
1lun my
Ck
The Colot term is similar, but makes sure that each object the model has found is a true object, where we multiply by ck to not penalize for predicted objects in which we have low confidence:
Laton (v, C, m)
Σ
min ci I|
myll
Finally, the sparsity loss discourages the model from using multiple outputs to represent the same true object, by encouraging sparsity in object hypothesis confidences (derivation in Section D):
Cuparso (c) log||e||
OBM-NETS
Inspired by the the basic form of classic DAF algorithms and the ability of modern neural-network techniques to learn complex models, we have designed the OBM-Net architecture for learning OBMs and cuwwwstomized procedure for training it from data, motivated by several design considerations. First, because object hypotheses must be available after each individual input and because observations will generally be too large and the problem too difficult to solve from scratch each time, the network will have the structure of recursive filter, with new memory values computed on each observation and then fed back for the next. Second, because the loss function is set based, that is, it doesn't 't matter what order the object hypotheses are delivered in, our memory structure should also be permutation invariant and independent of the number of objects, and so the memory processing is in the style of an attention mechanism. Finally, in applications where the observations = may in a representation not well suited for hypothesis representation and aggregation, the memory operates on latent representation that is related to observations and output hypotheses via encoder and decoder modules.
Figure 2 shows the architecture of the OBM-Net model. The memory of the system is stored in 8, which consists of K elements, the K hypotheses DAF, each in Id::; the length-K vector n of positive values encodes how many observations have been assigned each slot during the execution so far. New observations are combined with the memory state, and the state updated to reflect the passage of time by a neural network constructed from seven modules with trainable weights.
When an observation 2 arrives, it is immediately encoded into a vector e Rd, which is fed into subsequent modules. First, attention weights computed for each hypothesis slot, using the
4

---

Under review as a conference paper at ICLR 2022
encoded input and the existing content of that slot, representing the degree to which the current input "matches" the current value of each hypothesis in memory (corresponding to hypothesis matching computation in DAF algorithms). To force the network to commit to sparse assignment of observations to object hypotheses while retaining the ability to effectively train with gradient descent, the suppress module sets all but the top M values in w to O and renormalizes, obtain the vector of M values that sum to 1:
expl (attend (sk, nig e))
Wk == exp(attend (zj, MM: e)) ; a suppress
The a vectors are integrated to obtain which is normalized to obtain the output confidence
The update module also operates on the encoded input and the contents of each hypothesis slot, producing a hypothetical update of the hypothesis in that slot under the assumption that the current an observation of the object represented by that slot (corresponding hypothesis updates in DAF algorithms); so for all slots k.
Wk update (Sk, (g.,))
Additionally, a scalar relevance value, (0, 1), is computed from s and e; this value modulates the degree to which slot values are updated, and gives the machine the ability to ignore or downweight an input, corresponding to rejection of outlier observations in DAF algorithms. It is computed as
K
= relevance(e, s, = 2N2 NN1 8k. n4))
k=1
where NN, is a fully connected network with the same input and output dimensions and NN2 is a fully connected network with a single sigmoid output unit. The attention output and relevance r are now used to decide how to combine all possible slot-updates u with the old slot values 8 using the following fixed formula for each slot k:
S14 = (1 rag) + rag
Because most of the ag values have been set to O, this results in sparse update which will ideally concentrate on a single slot to which this observation is being "assigned", and correspond to the DArhypothesis updates in DAF algorithms.
To obtain outputs, slot values sf are then decoded into the outputs, y, using a fully connected network:
yk decode (8/x)
Finally, to handle the setting in which object state evolves over time, we add a transition module, which computes the state St+1 from the new slot values s using an additional neural network, corresponding to DAFtransition updates in DAF algorithms:
St+1k transition
These values are then fed back, recurrently, as inputs to the overall system.
Given a data set D, we train the OBM-Net model end-to-end to minimize loss function C, with slight modification. We find that including the Leparse: term from the beginning of training results in poor learning, but adopting a training scheme in which the Lxparse first omitted then reintroduced over training epochs, results in reliable training that is efficient in both time and data.
EMPIRICAL RESULTS
We evaluate OBM-Net on several different entity monitoring tasks. First, we consider a simple online clustering task and validate the underlying machinery of OBM-Net as well as its ability to generalize inference time to differences in (a) the number of actual objects, (b) the number of hypothesis slots and (c) the number of observations. Next, we evaluate the performance of OBM-Net on an image domain which the underlying observation space is substantially different from the hypothesis space. Finally, we evaluate the performance of OBM-Net on the complex simulated household robot domain shown in Figure and validate the ability OBM-Net to capture an object with underlying dynamics and complex properties, as well as its utility for downstream robotics object-fetching tasks. We provide additional evaluation of our approach in Sections A, B and C.
Baselines and Metrics In each domain, LSTM (Hochreiter and Schmidhuber, 1997) and set transformer (Lee et al., 2018) (details in E.3), as well as task-specific baselines. All learned network architectures structured use 50000
compare OBM-Net online learned baselines of
5

---

Under review as a conference paper at ICLR 2022
Model
Online
Learned
Normal
Elongated
Mixed
Angular
Noise
OBM-Ner
0.157
0.191
0.184
0.794
0.343
Set Transformer
0.407
0.395
0.384
0.794
0.424
0.256
0.272
0.274
VQ
0.173
0.195
0.191
0.992
0.947
Set Transformer
0.226
0.218
0.274
0.816
0,406
Attention
0.254
0.267
0.268
K-meunsi++
0.103
0.139
0.135
1.259
GMM
0.113
0.141
0.136
0.865
1,207
Table I: Quantitative Results on Online Clustering, Comparison of performance on clustering performance across different distributions. Reported error is the L2 distance between predicted and ground truth means. Methods the bottom half of table operate on observations bulk and thus are not directly comparable. Observations
10 Observations
Observations
Observations
Figure 3: Qualitative Visualization of OBM-Net. Illustration of OBM -Net execution on the Normal distribu- tion setting. Decoded value of hypothesis (with size corresponding to confidence) shown in red, with ground truth clusters in black. Observations shown in blue.
parameters. Unless otherwise noted, models except OBM-Net are given and asked to predict the ground truth number of components K, while OBM-Net uses 10 hypothesis slots. Results are reported in terms of MSE error k min5llyx myll (with respect to the most confident K hypotheses for OBM-Net).
5.1 ONLINE CLUSTERING
Setup. To check the basic operation of the model and understand the types of problems for which performs well, we first test approach on simple clustering problems with the same input and output spaces, but different types of data distributions, each a mixture of three components. We train on 1000 problems with observation sequences of length 30 drawn from each problem distribution and test on 5000 problems from the same distribution. In every case, the means of the three components are drawn at random for each problem. We consider set of five problem distributions, a Normal setting in which each component is 2D Gaussian with identical variance across individual dimensions and components, Enlongated and Mixed settings where 2D Gaussians have more variation across different components and Angular and Noise settings where underlying distributions are non-Gaussian in nature. We provide precise details about distributions in Section E.I.
Baselines and Metries. In addition to the online learned baselines described in Section 5, we compare our approach with following task specific clustering methods: Batch, non-learning: K- means++ (Arthur and Vassilvitskii, 2007) and expectation maximization (EM) (Dempster al., 1977) on a Gaussian mixture model (SciKit Learn implementation); Online, non-learning: vector quantization (Gray, 1984); Batch, learning: set transformer (Lee et al., 2018) and slot attention (Locatello et al., 2020).
Results. We compare our approach to each of the baselines for the five problem distributions in Table 1. The results in this table show that on Normal, Mixed, and Elongated tasks, OBM- performs better than learned and constructed online clustering algorithms, but does slightly worse than offline clustering algorithms. Such discrepancy in performance is to be expected due to the fact that OBM-Net is running and being evaluated online. On the Angular and Noise tasks, OBM-Net is able to learn a useful metric for clustering and outperforms both offline and online alternatives.
Next, we provide a qualitative illustration of execution of OBM-Net on the Normal clustering task in Figure 3 as trajectory of observations are seen. We plot the decoded values of hypothesis slots in red, with size scaled according to confidence, and ground-truth cluster locations black. OBM-Net is able to selectively refine slot clusters to be close to ground truth cluster locations even with much longer observation sequences than it was trained on. We provide qualitative visualization of individual modules of OBM-Net in Section A.2 well as performance on increased numbers of clusters in Section A.4. We further provide ablations of each proposed component of OBM-Net in Section A.3. Generalization. We next assess the ability of OBM-Net to generalize inference time to dif- ferences in the number of input observations as well as differences in the underlying number of hypothesis slots used the Normal distribution. On the left side of Figure 4, we plot the error of

---

Under review as a conference paper at ICLR 2022
Error Test Time Observations
04
Model
Slots
Ground Truth Clusters
0.3
7
10
0.162
0.214
0.242
OeM-Net
0,175
0,195
0.213
0.197
0.205
0.1
Set Tronsfoforrr
Set Transformer
0.261
0.279
0.282
0.
Vector Quantization
0.171
0.199
0.205
Observattinss
Figure 4: (left) Generalization with Increased Observations. Plot of LSTM, Set Transformer and OBM-Net errors when executed at test time on different number of observations from the Normal distribution. With increased observations, OBM-Net error continues to decrease while other approaches obtain higher error. (right) Generalization with Different Hypothesis Slots. Error of OBM-Net, when executed at test time with a different number hypothesis slots on test distributions with different numhers of ground true components. In all cases, OBM-Net is trained on 3-component prohlems with 10 slots. OBM-Net achieves good performance with novel number of hypothesis slots, and outperforms different instances of the Set Transformer trained with the ground truth number of cluster components as well vector quantization.
LSTM, Set Transformer, and OBM-Net as a function of the number observations seen at inference time. We find that when OBM-Net given more observations then seen during training time (all models are trained with observations of length 30), it is able to further improve its performance, while both LSTM and Set Transformer results suffer. We believe that such generalization ability due to the inductive bias added to OBM-Net, and provide an analysis in Section A.3. We provide additional analysis of this generalization across all distributions in Table A6 and find similar results.
On the right side of Figure 4, we investigate the generalization ability of OBM-Net at inference time to increases in both the number of hypothesis slots and the underlying number of mixture components from which observations are drawn. We compare to the set transformer and to VQ. both of which know the correct number of components at inference time. We find that OBM-Net generalizes well to increases in hypothesis slots, and exhibits improved performance with large number of underlying components, performing comparably to or better than the VQ algorithm. We further note that none of the learning baselines can adapt to different numbers cluster components inference time, but find that OBM-Net outperforms the set transformer even when it is trained on the ground truth number of clusters in the test. We provide additional generalization analysis in Section A.1.
5.2 IMAGE-BASED DOMAINS
We next validate the ability of OBM-Net to perform entity monitoring on image inputs, which requires OBM-Net to synthesize a latent representation for slots, and learn to perform association, update, and transition operations in that space.
Setup. We experiment with two separate image-based domain, each consisting of a set of similar entities (2D digits or 3D airplanes). We construct entity monitoring problems by selecting K objects in each domain, with the desired y values being images of those objects in a canonical viewpoint. An input observation sequence is generated by randomly selecting one of those K objects, and generating an observation 2 corresponding to a random viewpoint of the object. Our two domains are: (1) MNIST: Each object is a random image in MNIST, with observations corresponding rotated images, and the desired outputs being the un-rotated images; (2) Airplane: Each object is a random object from the Airplane class in ShapeNet (Chang al., 2015), with observations corresponding to airplane renderings (using Blender) at different viewpoints and the desired outputs the objects rendered in canonical viewpoint. We provide details in Section E. and use K 3 components.
Baselines. In addition to our learned baselines, we compare with a task specific baseline, batch K-means, in a latent space that is learned by training an autoencoder on the observations. In this setting, we were unable to train the Set Transformer stably and do not report results for it.
Results. In Table 2, we find that our approach significantly outperforms other comparable baselines in both accuracy and generalization. We further visualize qualitative predictions from our model in Figure 5. We find that our highest confidence decoded slots correspond to ground truth objects.
5.3 SIMULATED HOUSEHOLD ROBOT DOMAINS
Finally, we validate that OBM-Net can solve the entity monitoring task in simulated robotic settings. Setup. We model a robot moving within a house, as pictured in Figure 1, in the PyBullet simulation environment. In this house, each problem will involve following trajectory consisting of sequence of 50 locations. These locations are distributed across 5-6 separate rooms, with later locations potentially revisiting earlier locations. At each location, the robot looks around and if there is table
7

---

Under review as a conference paper at ICLR 2022
Model
Learedd
MNIST
Airplanes
Ohservations
10
30
50
100
10
30
50
100
OBM-Net
7.143
5.593
5.504
5,580
4,558
4.337
4,325
LSTM
9.980
9.208
9,166
9.267
5.106
4,992
4,983
4,998
K-mmans
13.596
12.505
12.261
12.021
7,246
6,943
6,878
6,815
Table 2: Quantitative Results on Image Domain, Comparison of monitoring performance on MNIST
and Airplane datasets across 10, 30, 50, 100 observations. For OBM-Net, L.TTM and K-means we use a convolutional encoder/decoder trained on the data. We train models with 30 ohservations and report MSE error.
Training Objects
Truinipg Ohscrvation
Training Objects
Training Observaiions
7 + 9
>
1
4 1
4 9 4
4
f
S 8
S
3 O
8
8 4 3 7
g
0 7
3
7 0
0 4
4
A
2
4
Groual Truth Objects
Decoled Stote
Ground Trull:
Decoded Stote
Figure 5: Qualitative Visualization of OBM-Net Execution on Images. Qualitative visualization of two
image-based association tasks (left: MNIST, right: airplanes). At the top of each is example training problem, illustrated by the true objects and an observation sequence. Each of the next rows shows an example test problem. with the ground truth objects and decoded slot values. The three highest-confidence hypotheses for each problem are highlighted in red, and correspond to ground-truth objects.
within view (which happens about 50% of the time). it will get an observation of one of the objects on the table or an empty observation otherwise. Each new problem has 8 tables whose locations are drawn from a larger set of potential table locations and on each table there will be two objects drawn from a small set of classes, e.g. lamp, cushion, etc. Each object class has a characteristic stochastic movement pattern, with one object class sequentially teleporting between tables (details in appendices). The goal is for the robot to be able to construct hypotheses for each distinct object it has seen and to be able to predict for each object the table it is currently on and its location relative to the table. More precisely, the input sequence of observations & corresponds to a segmented depth map of single object visible given the camera pose at particular location in the trajectory (or an empty observation in the case no object is visible), as well as which table it is resting on and its positional offset relative the table. The desired output y values are, for each object seen so far, is the predicted table y' it is on currently as well its associated offsets relative to the predicted table, y".
We train on total of 10000 randomly sampled trajectories in the same floor plan, but with new randomly drawn object instances and tables for each trajectory. We test using 1000 trajectories, with test object meshes drawn from set disjoint from the set of object meshes used during training (but sharing the same semantic class). To test the flexibility of the approach, we consider three different configurations of object classes on tables, with the motion pattern of each of the object classes illustrated in Section 1, as well additional setup details and example observations.
Metrics. To test the efficacy of our approach, we measure to what extent each hypothesis slot m, can recover both the table that the associated object is as well as the object's 's position relative to the table. We match a hypothesis slot & with each object label y/i by computing arg minx l| m&l + Lossce(v) m For each match, we report the accuracy of m matching vř: and well the mean absolute error between yf and mf. When the table prediction for y/ is incorrect, we set mean absolute error to be equal half the table size (0.15), reported table offsets are meaningless in that case. In this setting, both OBM-Net and associated baselines use 10 hypothesis slots.
Baselines. In addition to our learned baselines, we compare to two task specific baselines. We construct simple clustering baseline for this problem. Given localized input-segmented depth map, we extract object offsets by averaging all points in the point cloud associated with each segment. To associate objects dynamically across time, we use batch K-means clustering on the inferred object candidate offsets and associated table identities to obtain a set of objects. We further compare OBM- clustering method used in the STRANDS project (Hawes
Net with the more complex et al., 2017) to infer objects in a real robotic setup from our underlying segmented depth maps, as well as a hand-crafted DAF system using ground truth dynamics. For all learned models, we convert the segmented depth maps into downsampled 3D pointclouds.
Results. Table 3 shows that OBM-Net outperforms the baselines in both estimating the supporting tables and regressing the relative position of the objects across different number of observations. Figure 6 (left) shows the prediction error of all methods the robot last saw an object; observe that OBM-Net is substantially better at long-term memory than the LTM and set transformer, and still outperforms the clustering and STRANDS baselines even
function of the number of steps since
8

---

Under review as a conference paper at ICLR 2022
Model
Leumed
Configuration A
Configuration B
Contiguration C
Table Accuracy
Position Ervor
Table Accuracy
Position Liror
Table Accuracy
Postion Error
Obseraatoos
10
25
50
10
25
50
10
25
50
10
25
50
10
25
50
10
25
50
OBM-Net
0.984
0.926
0.809
0,019
0,041
0,078
11.89
0.924
0.745
0,021
0,946
0.082
0,988
0,932
0,873
0,027
0J852
0.880
Set Transfocmer
0.583
0.819
0.476
0,066
0.089
0.919
0.771
0.542
0.924
0,052
0.083
0.885
0.745 0.648
0.057
0,056
0.089
LSTM
0.126
0.123
0.892
0.091
0.130
Clustering
0.761
83070
0.105
0.761
0.770
0.103
0.761
0.695
0.069
0.303
STRANDS
0.900
0.733
0.610
0.033
0.057
0.085
0.940
0.841
0.37
0.033
0.048
0.087
0.973
0.822
0.831
0.055
0.86
DAF
0.059
0.807 0.620
0.022
0.043
0.081
0.537
0.87
0.787
0.021
0.039 0084
0.974 0.914 0.303 0.050
0.055 0083
Table 3: Quantitative Analysis of OBM-Net on Simulated Household Domain. Quantitative comparison of
OBM-Net with baselines across 3 studied household domain configurations across 10, 25, 50 observations.
Orject Recevely as Time
2.8
0.6
SUMM
Shage Prediction
Shape Prodiction (Set Transformer)
Stupe Prodiction (0DM-Net)
Inforcal Ginep (ODM-Net)
Infored Grreg (Applied Real Scome)
Tino
(LSTM)
Figure 6: (left) Object Recovery over Time. Percentage of objects correctly recovered function of timesteps since seeing the object last. OBM-Net performs similarly to oracle with ground truth dynamics. (middle) 3D Reconstructions. Illustration of 3D reconstructions of hypothesis from each model. OBM-Net obtains accurate 3D reconstructions. (right) Estimated Grasps. We utilize the predicted 3D mesh from OBM-Net to infer grasp which successfully enables the grasp of a real object in the ground truth scene. with long inter-observation gaps. As an upper bound, we compare with an oracle model, which knows ground truth object identity and dynamics (ignoring object collision). We find that OBM-Net performs similarly to the oracle model (performance across all models drops due to stochasticity), and in some cases does better, perhaps by taking account object collisions.
By adding shape occupancy prediction head (Mescheder et al., 2019) to OBM-Net, we can also regress the underlying 3D shapes of our objects. We predict each shape at 32 x 32 x 32 resolution, decoding each occupancy at each voxel coordinate using a MLP head conditioned on a hypothesis state. Quantitatively, we find that our approach gets 95.33% accuracy compared to 72.74% accuracy obtained by a LSTM and 73.67% obtained by set transformer when predicting voxels for each test mesh in the test set. We provide visualization of predicted shapes from OBM-Net in Figure A4.
Object fetching. Finally, we verify that object hypotheses from OBM-Net can usefully support a task in which a robot has to retrieve an object it has previously observed. First, we consider the object. We train LSTM, set transformer, and OBM-Net to
task of finding a previouly-encounteen predict underlying object class yf for each object hypothesis, as well as shape estimate and location. Given a desired object class (for example, either a plant, cushion, or bucket in configuration A)) we wish to find, the robot examines each prediction (yi, and navigates in the simulated world to look for an object of the specified class, based on predictions of yf and yf. We measure the number of predictions that need to be queried to find the object, as well in which the robot succeeded within 10 attempts. On this task, we find that LSTM obtains an overall planning success of 68.75% with an average number of 5.38 hypotheses investigated before finding an object. In contrast, the set transformer obtains a planning success of 81. 81.25% with on average 4.88 attempts. We find that OBM-Net performs best and is able to find the object of the desired class 100% of the time, with an average of 2.03 hypotheses examined before finding the object.
overall success percentage of trials
Next, qualitatively analyze the 3D reconstructions of each object hypothesis and its ability to support manipulation. Given a 3D reconstruction, we compute grasps on the underlying shape by looking for parallel planar surfaces large enough to accommodate the gripper. We then try to execute that grasp on the target 3D object we wish to grasp in the (simulated) real world. As illustrated in Figure 6, we find that the 3D reconstruction of object hypotheses from OBM-Net is accurate enough to enable grasping of real 3D shape. In contrast, predictions from LSTM and set transformer baselines are significantly poorer and do not enable downstream manipulation.
Discussion This work has demonstrated that using algorithmic bias inspired by classical solution to the problem of filtering to estimate the state of multiple objects simultancously, coupled with modern machine-learning techniques, we can arrive at solutions that learn to perform and generalize well. Importantly, the same underlying system, with no prior knowledge about the types of obser- vations or desired output hypotheses or the frequency of observations, is able to learn to perform data-association and state estimation to solve a variety of entity monitoring problems as well as to support an based memory system for robot in dynamically changing environment.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Symposium on Discrete Algorithms '07, 2007.
Song Bai, Peng Tang. Philip HS Torr, and Longin Jan Latecki. Re-ranking via metric fusion for object retrieval and person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 740-749, 2019. 3
Vaibhav Bansal, Gian Luca Foresti, and Niki Martinel. Where did see it? object instance re- identification with attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 298-306 2021. 3
Yaakov Bar-Shalom, Fred Daum, and Jim Huang. The probabilistic data association filter. IEEE Control Systems Magazine, December 2009, 14
Harry Barrow, Jay M. Tenenbaum, Robert C. Bolles, and Helen Wolf. Parametric correspondence and chamfer matching: Two new techniques image matching. IJCAI, 1977. 4
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing (ICIP), pages 3464-3468, 2016. 2
Aditya Bhaskara and Aravinda Kanchana Rwanpathirana. Robust algorithms for online k-means clustering. In Algorithmic Learning Theory, pages 148 173. PMLR, 2020.
Guillem Brasó and Laura Leal-Taixé. Learning neural solver for multiple object tracking. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. Monet: Unsupervised scene decomposition and representation. 11Xi:1900.11190, 2019. 3
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. 5 12.00122, 2015.
Vincent Cohen Addad, Benjamin Guedj, Varun Kanade, Guy Rom. Online k-means clustering. In International Conference on Artificial Intelligence and Statistics, pages 1126-1134. PMLR, 2021. 2
Arthur P Dempster, Nan Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em
algorithm. Journal of the Royal Statistical Sociery: Series B (Methodological), 39(1):1-22, 1977. 6
Davi Frossard and Raquel Urtasun, End-to-end learning of multi-sensor 3d tracking by detection. In ICRA, May 2018. 2
Rohit Girdhar and Deva Ramanan. Cater: diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint 191.00449 2019.
Gray. Vector quantization. JEEE ASSP Magazine, 1(2):-22) 29, 1984. 2,
Klaus Greff, Raphaël Lopez Kaufmann Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In ICMI, 2019. 3
Nick Hawes, Christopher Burbridge, Ferdian Jovan, Lars Kunze, Bruno Lacerda, Lenka Mudrova, Jay Young. Jeremy Wyatt, Denise Hebesberger, Tobias Kortner, et al. The strands project: Long-term autonomy in everyday environments. IEEE Robotics & Automation Magazine, 2463):146-156, 2017. 8
Shuting Hc, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer-hasedd object re-identification. arXiv preprint ar arXiv:2102.044 2021. 3
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural Comput., 98)) 1997. 3, 5, 14
Tsung-War Wei Huang, Jiarui Cai, Hao Yang, Hung-Min Hsu, and Jenq-Neng Hwang. Multi-view vehicle re- identification using temporal attention model and metadata re-ranking. In CVPR Workshops, volume 2, 2019.
10

---

Under review as a conference paper at ICLR 2022
Rico Jonschkowski. Divyam Rastogi. and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. ArXiv, abs/1805.111 1112,, 2018.
Peter Karkus, Xiao Ma. David Hsu, Leslie Pack Kaelbling. Wee Sun Lee, and Tomas Lozano-Perez. Differen- tiable algorithm networks for composable robot learning. ArXiv, abs/1905, 1tw./9955.1600, 2019.
Teuvo Kohonen. Self-Organizing Maps. Springer-Verlag. 1995. 2
Rahul G. Krishnan, Uri Shalit, and David A Sontag. Deep kalman filters. ArXiv, abs/151 2015.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Tch. Set transformer:
A framework for attention-based permutation-invariint neural networks. arXiv preprint 1810000825
2018. 5, 6, 14, 19
Edo Liberty, Ram Sriharsha, and Maxim Sviridenko. An algorithm for online k-means clustering. In 2016 Proceedingy of the eighteenth workshop algorithm engineering and experiments (ALENEX), pages 81-89. SIAM, 2016.
Huajun Liu, Hui Zhang. and Christoph Mertz. DeepDA: LSTM-based bardeep data association network for multi-targets tracking in clutter. 2019 22th International Conference on Information Fusion (FUSION), pages 1-8, 2019.
Francesco Locatello, Dirk Weissenhorn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020. 3,
Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao, Tae-Kyun Kim. Multiple object tracking: A literature review. arXiv preprint arXiv: 1409.14499..86 7618, 2014. 2
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer
Vision and Puttern Recognition, pages 4460-4470, 2019. 9
Anton Milan, Seyed Hamid Rezatofighi, Anthony R. Dick, Ian Reid, and Konrad Schindler. Online multi-target tracking using recurrent neural networks. ArXiv, abs/1604,0300 1604,03635, 2017. 3
Adam Santoro. Ryan Faulkner. David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in neural information processing systems, pages 7299-7310, 2018. 18
Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d
shape recognition. IEEE SPL, 22(12):2339 2343, 2015.
Aviv Tamar, Sergey Levine, Pieter Abbeel, Yi Wu, and Garrett Thomas. Value iteration networks. ArXiv, abs/1602.02867, 2016.
Oriol Vinyals. Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint D...vv:1111.039// 2015.
Greg Welch and Gary Bishop. An introduction to the kalman filter. 2006. 2
Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Roboticy and Automation Letters, 5(2):713-720, 2020. 17
Yu Xiang, Alexandre Alahi, and Silvio Savarese. Learning to track: Online multi-object tracking by decision making. In Proceedings of the IEFF international conference on computer vision, pages 4705-4713, 2015.
Xiao Zhang, Yixiao Ge. Yu Qiao, and Hongsheng Li. Refining pseudo labels with clustering consensus over generations for unsupervised object re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3436-3445, 2021a.
Yifu Zhang, Chunyu Wang, Xinggang Wang. Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification multiple object tracking. International Journal of Computer Vision, pages 1-19, 2-21b.. 2021b.
11

---

Under review as a conference paper at ICLR 2022
Appendix
To fully understand the underlying machinery of OBM-Net, we provide additional qualitative and quantitative verification of OBM-Net on online clustering in Section A, on dynamic clusters in Section B and in simulated household domains in Section C. We provide derivation of the sparsity loss utilized in training in Section D. Finally, we provide experimental and architecture details of OBM-Net in Section E.
A ONLINE CLUSTERING RESULTS
We provide quantitative and qualitative results on online clustering to further analyze and understand the underlying performance and algorithmic computation performed by OBM-Net. First, we analyze the generalization of OBM-Net to increased number of clusters of inputs in Section A. Next, we provide qualitative visualization of OBM-Net in Section A.2, and ablations in Section A.3, We further analyze performance on problems with a larger number of clusters in Section A.4.
A.1 GENERALIZATION
We provide additional analysis of the ability of OBM-Net to generalize at test time to increased number of observations and an increased number of clusters.
Generalization to Increased Observation Number. We report performance of different models across different distributions in Table A6 when executed at test time with an increased number of observations. We find that OBM-Net able to obtain better performance with increased number of observations across all different distributions, while other learned baselines perform worse Fur- thermore, we find that OBM-Ne outperforms other learned baselines in all distributions except for rotation. For rotation we find that when training with 10,000 different distributions, OBM-Net exhibits better performance of 0.555 compared Set Transformer performance of 0.647 and L.TTM performance of 0.727 with 30 observations (and similarly outperforms Set Transformer and LSTM at larger number of observations).
Inferring Object Number. In contrast to other algorithms, OBM-Net learns predict both set of object properties y/ of objects and a set of confidences c for each object. This corresponds to the task of both predicting the number of objects in of observations, well as the associated object properties. We evaluate the ability of OBM-Net to regress object number at test time in scenarios where the number of objects (underlying clusters) is different than that of training. We evaluate on the Normal distribution with variable number of component distributions, and measure inferred components through a threshold confidence. OBM-Net is trained on a dataset with underlying components. We find in Figure Al that OBM-Net is able to infer the presence of more component distributions (as they vary from 3 to 6), with improved performance when cluster centers are sharply separated (right figure of Figure A1).
A.2 QUALITATIVE VISUALIZATION
Submodule Visualization. We find that individual modules learned by OBM-Net are interpretable. We visualize the attention weights of each hypothesis slot in Figure A2 and find that each hypothesis slot learns to attend to a local region next to the value it decodes to. We further visualize plot of relevance weights in Figure A3 across an increasing number of observations where individual observations are drawn distributions with different levels of noise with respect to cluster centers. We find that as we see more observations, the relevance weight of new observations decreases over time, indicating that OBM-Net learns to pay the most attention towards the first set of observations it sees. In addition, we find that in distributions with higher variance, the relevance weight decreases more slowly, as later observations are now more informative in determining cluster centers.
A.3 ABALATIONS
We ablate each component of OBM-Net and present results in Table Al on the Normal distribution. We test removing Leparse (sparsity), removing learned slot embeddings (leamed memory) where
12

---

Under review as a conference paper at ICLR 2022
Noisy Cluster
Seperated Cluster
Incoered Componans
60
1.8
Cocsponents
Cumponunts
Figure A1: Generalization to Increased Cluster Number. Plots of inferred number of components using confidence threshold in OBM-Net compared to the ground truth number of clusters (OBM-Net is trained on only 3 clusters). We consider two scenarios, noisy scenario where cluster centers randomly drawn from -1 to (left) and scenario where all added cluster components well seperated from each other (right). OBM-Net able to infer more clusters both scenarios, with better performance when cluster centers are more distinct from each other.
Obs 100
Update Magnitudk over Updates
1.0
Noise 1.0
0.5
0.5
Updere Weight
Noice 2.0
0.0
0.6
-0.5
005
0.4
-1.0
-1.0
0.2
11.
-05
1.0
-0.5
20
40
80
100
Figure A2: Visualization Attention Weights. Plot of decoded values of slots (in red) with confidence shown by the size dot (left), and what slot each input assigns the highest attention towards (right) (cach slot is colored differently, with assigned inputs colored in the same way). Note alignment of regions on the right with the decoded value of slot on the left.
Observations
Figure A3: Visualization of Relevance Weights. Plots of the magnitude of rele- vance weights with increased observation number on different distributions with dif- fering standard deviation (noise).
instead, individual hypothesis slots, we store the explicit values of inputs, removing the suppress modules (suppression) and removing the relevance module (relevance). We find that each of our proposed components enables better performance on the underlying clustering task. Interestingly, we further find that the addition of relevance enables approach generalize test time to larger numbers of observations.
Sparsity
Learned
Supression
Relevance
Observations
Memory
10
30
50
100
0.382
0.452
0.474
0.487
0.384
0.412
0.423
0.335
0.357
0.366
0.387
0.278
0.238
0.157
0.137
0.131
Table A1: Abalation Analysis. We ablate each component of OBM-Net on the Normal distribution. When learned memory is ablated, OBM-Net updates states based observed values (appropriate in the Normal distribution dataset).
A.4 LARGER NUMBER CLUSTERS
We measure the performance of OBM-Net when trained with large number of clusters and slots. We utilize the Normal distribution setting, but generate underlying training input observations from a total of 30 difference components, and train OBM-Net with total of 30 slots. We train OBM-Net with observations, and measure performance at inferring cluster centers with between 50 100 observations. We report performance in Table A2 and find that OBM-Net obtains good performance
13

---

Under review as a conference paper at ICLR 2022
Model
Online
Observations
50
65
80
100
OBM-Net
0.158
0.154
0.151
0.147
VQ
0.162
0.157
0.153
0.148
K- K-means++
0,155
0.151
0.148
0.146
GMM
0.156
0.151
0.149
0.147
Table A2: Performance on Large Number of Clusters. Comparison performance on Normal distribution, when underlying distributions have a large number of components. We use 30 components, and train models with 50 observations. Each cluster observation and center is drawn between -1 and with reported error as the L2 distance between predicted and ground truth means.
in this setting, out-performing the strong online baseline VQ, and performing similarly to K-means++ which directly operates on all input observations at once.
B DYNAMIC DOMAINS
We further verify the ability of OBM-Net to perform entity monitoring in a dynamic setting and compare its performance with that of classical data aato-association filter.
Setup. We evaluate performance of dynamic entity monitoring using moving 2D objects. A problem involves trajectory of observations of the K dynamically moving objects, with desired y values being the underlying object positions. Objects evolve under a linear Gaussian dynamics model, with a noisy observation of single object observed at each step (details in Section E. This task is typical of tracking problems considered by DAF. All learning-based models are trained on observation sequences of length 30. To perform well in this task, a model must discover that needs to estimate the latent velocity of each object, as well as learn the underlying dynamics and observation models. We utilize K 3 for our experiments.
Baselines. We compare with the de-facto standard method, Joint Probabilistic Data Association (JPDA) (Bar-Shalom et al., 2009), which uses hand-built ground-truth models (serving as an oracle). We further compare with our learned online baselines of Set Transformer (Lee et al., 2018) and LSTM (Hochreiter and Schmidhuber, 1997) methods.
Result. Quantitative performance, measured in terms of prediction error true object locations, is reported in Table A3. We can see that the Set Transformer cannot learn a reasonable model at all. The LSTM performs reasonably well for short (length 30) sequences but quickly degrades relative to OBM-Net and JPDA as sequence length increases. We note that OBM-Net performs comparably to, but just slightly worse than, JPDA. This is strong performance because OBM-Net is generic and can be adapted to new domains given training data without the need to aand-design the models in JPDA. We believe that these gains are due the inductive biases built into our architecture.
Model
Observations
20
OBM-Net
0.415
0.395
0.382
0.394
Set Transformer
0.699
0.701
0.854
1.007
LSTM
0.422
0.400
0.445
0.464
JPDA (oracle)
0.683
0.372
0.362
0.322
Table A3: Performance on Dynamic Objects. Comparison of different methods on estimating the state of dynamically moving objects. All learning models are trained with 1000 sequences of 30 obscrvations. We report MSE error. JPDA uses the ground-truth observation and dynamics models.
C HOUSEHOLD DOMAINS RESULTS
We provide additional analysis of the ability of OBM-Net to perform entity monitoring in a simulated household domain.
Additional Simulated Household Robot Results. We present qualitative visualizations of 3D shapes obtained by OBM-Net when executed test trajectories. In Figure A4 we illustrate different
14

---

Under review as a conference paper at ICLR 2022
Grond
Truth
Prod
Figure A4: 3D Reconstructions of OBM-Net. Illustration of predicted 3D shapes from OBM-Net when OBM-Net is executed on a test trajectory. We further visualize ground truth meshes seen in the trajectory and find that predicted shapes coarsely match ground truth meshes.
Figure A5: Illustration of iGibson Environment. (left) Illustration of example RGB input in our iGibson environment. (right) Example configuration of tables in our iGibson environment (tables drawn in blue).
predicted 3D shapes with their associated ground truth 3D shape. We find that reconstructions from OBM-Net appear to coarsely match the underlying shape of ground truth 3D meshes.
We further validate the efficacy of OBM-Net on the more complex household domain of iGibson. We illustrate example input observation of our environment, and the corresponding configuration of tables and objects Figure A5. We utilize the same configuration settings as the Pybullet environment, training models trajectories of length 50, consisting of 8 tables with objects on them each. We utilize object classes and movement patterns from Configuration described in Section 5.3 in the main paper. We provide additional dataset details in Section E. We compare OBM-Net with LSTM and clustering baselines discussed in Section 5.3 of the main paper. We use the same metries described in Section 5.3. In Table A4 report results this household setting. We find that OBM-Net performs significantly better than LSTM and Clustering baselines.
iGibson Environment.
Scanned iGibson Environment. We further validate the efficacy of OBM-Net utilizing real house scans from Figure A5 as the background of our environment. We utilize the same configurat configuration settings as the Pybullet environment, training models trajectories of length 50, consisting of tables with objects on them each. We utilize object classes and movement patterns from Configuration A described in Section 5.3 in the main paper. We provide additional dataset details Section E. We compare OBM-Net with LSTM and clustering baselines discussed in Section 5.3 of the main paper. We use the same metrics as described in Section 5.3. Table A5 we report results on this household setting. We find that OBM-Net performs significantly better than LSTM and Clustering baselines.
Model
Learndd
Accuracy
Position
Observations
25
OBM-Net
0,992
0.925
0.813
0.159
0.234
0.301
LSTM Clustering
0,883
0.625
0.489
0.203
0.294
0.354
Table A4: Quantitative Results iGibson. Comparison of performance of OBM-Net and basclines the iGibson environment.
Model
Learedd
Table Accuracy
Position Error
Observations
10
25
50
10
25
50
OBM-Net
0.970
0.889
0.782
0.153
0.232
0.296
LSTM
0.628
0.499
0.297
Clustering
0,802
0.632
0.576
0.212
0.258
0.304
Table A5: Quantitative Results on Scanned Gibson Houses. Comparison of performance of OBM-Net and baselines scanned Gibson environment.
15

---

Under review as a conference paper at ICLR 2022
Gaussian Domain
Dynamic Domain
Robotic Domain
Figure A6: Qualitative Visualization of Domains. Visualizations of the Normal Gaussian, Dynamic domains and Robotic domains. Observations are transparent while ground truth states are bolded for gaussian and dynamic domains. Four sample image observations shown rohotic domain.
D SPARSITY LOSS
In this section, we show that Capans (c) encourage confidences e to be sparse. Recall that
(c) log//el|
(2)
where ||||| the L2 norm which is convex function. Recall that c, the confidence vector, defines a polyhedron, since is the set of points that non-negative, and whose elements up to one. The maximum of convex function over polyhedra must occur at the vertices, which correspond to an assignment of ||c]| given that its elements sum up to one. This is equivalent to finding the stationary points of the Lagrangian L(c,
single e; and Os every other value of Next consider the minimum of
L(G = =
+ &
(3)
By taking the gradient of the above expression, we find that the stationary value corresponds each being equal. Since the function is convex, this corresponds to the minimum of ||c|]- Thus Lspanse maximized when each individual confidence equal.
E EXPERIMENTAL DETAILS
In this section, we provide details of our experimental approach. We first discuss the details of datasets used Section E.1. Next, we provide the model architectures used in Section E.2, Finally, we provide details on the baselines we compare with in Section E.3.
E. DATASET DETAILS
We first provide detailed experimental settings for each the datasets considered the paper. Online Clustering. In online clustering, utilize observations drawn from the following distri- butions, where Gaussian centers are drawn uniformly from -1 to 1.
1. Normal: Each 2D Gaussian has standard deviation 0.2. The normal setting is illustrated in Figure A6.
2. Mixed: Each distribution is 2D Gaussian, with fixed identical variance across each individual dimension, but with the standard deviation of each distribution drawn from a uniform distribution from (0.04, 0.4).
3. Elongated: Each distribution is 2D Gaussian, where the standard deviation along each dimen- sion is drawn from uniform distribution from (0.04, 0.4), but fixed across distributions.
4. Angular: Each distribution is 2D Gaussian with identical standard deviation across dimension and distribution, but points above и are wrapped around x and points below x wrapped to
π. Gaussian means are selected between (-T,, -t /3) and between (2mr /3, т). The standard
deviation of distributions is 0.3 я.
5. Noise: Each distribution has 2 dimensions parameterized by Gaussian distributions with standard deviation 0.5, but with the values of the remaining 30 dimensions drawn from a uniform distribution from (-1
Dynamic Domains. Next, in the dynamics domain, we implement our dataset using the StoneSoup library". We initialize the location of each cluster from Gaussian distribution with standard deviation
htttp:://ttnnesee wew.hhedosserrccccocccooocoooo. c//e/v0.. 1b3/ stssssooppu..... .htm1
16

---

Under review as a conference paper at ICLR 2022
Plant
Cushion
Basket
Lamp
Trash Can
Cushion
Cushion
Lamp
Plant
Motion Start
Motion
End
Horizontal
Vertical
Teleport Motion
Horizontal Motion
Vertical
Teleport Motion
Horizontal Motion
Vertical
Teleport
Motion
Motion
Motion
Motion
Motion
Configuration A
Configuration B
Configuration C
Figure A7: Object Dynamics in Household Domain. Illustration object dynamics our simulated household environment. Objects exhibit either vertical, horizontal, or teleportation motion, dependent on the object class across configurations A. B, C. We illustrate the motion of each object class, with start and end images corresponding to 20 timesteps of motion of object, except for objects that teleport - these change each timestep.
1.5 and initialize velocity in each directory from Gaussian distribution with standard deviation of 0.02. At each timestep. Gaussian noise added to velocities with magnitude le-4. We show example tracks in Figure A6. Our JPDA implementation is also from the StoneSoup library.
Image Domains. In the image domain, for MNIST, we use the 50000 images in the training set to construct the training problems, and the 10000 images in the non-overlaping test set to construct the test problems. For the Airplane dataset, we use 1895 airplanes to construct the training problems, and 211 different airplanes to construct the test problems. Each airplane is rendered with 300 viewpoints.
Robotics Domains. For the robotics domains, we implement our embodied house environment in Pybullet, and construct house with x and y axis between -1 and 1. We utilize furniture assets from (Xia et al., 2020) for each of the individual classes of objects considered, with 50% of the objects (sorted alphabetically) being used for the training dataset and the 50% of the objects used for the test dataset. Each individual object is scaled by factor of 0.1 to fit on each table. Each table has size 0.15 by 0.1 in our setting. Our constructed house environment uses the floor plan illustrated in Figure of the main paper. Across configurations, objects, which each individual step of the trajectory corresponding a 1/60 of second advancement of simulation time PyBullet. In each configuration, objects in separate object classes move with velocities of (0.6, 0.0). (0.0, 0.6) and (0.3, 0.3) units per second respectively. collision event stop moving, making the underlying dynamics of objects a stochastic process. We illustrate example images observations from our environment Figure A6 (though segmented depth maps of objects instead input in Figure A7. Objects which teleport, teleport at each timestep (provided there is no collision).
the presence of collision, all objects involved in the
model) and illustrations of the underlying dynamics of objects
For the iGibson house, utilize the Pomaria environment in iGibson environment. This house has x and axis roughly between -4 and 4, and thus we scale tables to a size of 0.45 by 0.3 in the environment, and proportionally scale up the size of individual objects as well as their underlying movement speed. To sample trajectories both settings, we sample set of points across rooms in house and utilize motion planning infer paths connecting each individual point.
E.2 MODEL/BASELING ARCHITECTURES
We provide the overall architecture details for the LSTM in Figure A8a, for the Set Transformer in Figure A8b and OBM-Net in Figure A9a. For image experiments, we provide the architecture of the encoder in Figure A10a and decoder in Figure A10b. Both LSTM, OBM-Net, and autoencoding baselines use the same image encoder and decoder. For robotics experiments, we provide the architecture of the shape decoder in Figure A9b.
In OBM-Net memory, the function update(sk: $k. n&. e) is implemented by applying a 2 layer MLP with hidden units h which concatenates the vectors Sk. ny, e as input and outputs a new state ug of dimension h. The value ne is encoded using the function I+ny to normalize the range of input to be between 0 and 1. The function attend (kk Flg. e) is implemented in analogous way to update, using a 2 layer MLP that outputs a single real value for each hypothesis slot.
For the function relevance(sk, (k,, n&, e), apply NN, per hypothesis slot, which is implemented layer MLP with hidden units h that outputs a intermediate state of dimension h. (Sx. n4. e) is fed into NN, in analogous manner update. NN, is applied average of the intermediate representations
17

---

Under review as a conference paper at ICLR 2022
of each hypothesis slot and is also implemented a layer MLP with hidden unit size h, followed by a sigmoid activation. We use the ReLU activation for all MLPs. NNg represented is GRU, which operates the previous slot value.
E.3 BASELINE DETAILS
All baseline models are trained using prediction slots equal to the ground truth of components. To train the Set Transformer and we apply the Set Transformer sequentially on the concatenation of an input observation with the current of hypothesis slots. Hypothesis slots are updated based off the new values of the slots after applying self-attention (Set Transformer Encoder). Hypothesis slots are updated based off the new values of the slots after applying self-attention (Set Transformer Encoder). We use the Chamfer loss to train baseline models, with confidence set to 1.
in online manner, follow the approach in (Santoro al., 2018)
18

---

Under review as a conference paper at ICLR 2022
Dense
Dense - h
Dense +
Dense + h
LSTM(h)
Set Transformer Encoder
Dense +
Set Transformer Decoder
Dense -> output
(b) The model architecture of the Set Transformer bascline. The hidden dimension h is 48 for the syn- thetic Gaussian distributions. We use the encoder and decoder defined in (Lee et al., 2018) with 4 heads and hidden dimension h.
(a) The model architecture of the LSTM baseline. The hidden dimension used is 96 for synthetic Gaussian distributions and 128 for Image datasets. For image experiments, the first 2 and last 2 fully connected layers are replaced with image encoders and decoders.
Figure A8: Architecture of baseline models.
Dense + h
Dense >>
(1, Ms 2) + Dense +
OBM-Net Memory
Concat (h, state)
Dense - h
Dense -
Dense - output
Dense
(a) The model architecture of OBM-Net. The hidden dimension is 64 is for synthetic Gaussian distribu- tions and 128 the image/robotics experiments. For image experiments, the first and last linear layers are replaced with convolutional encoders and decoders.
(b) The shape decoder of OBM-Net used in the robotics experiments. The shape decoder takes input voxel coordinate as well as a value and predicts a occu- pancy for the voxel.
Figure A9: Architecture of OBM-Net and the shape decoder.
5x5 Conv2d, 32, stride 2, padding
Dense 4096
3x3 Conv2d, 64, stride padding
Reshape (256, 4,
3x3 Conv2d, 64, stride padding
4x4 Conv2dTranspose, 128, stride 2, padding
3x3 Conv2d, 64, stride 2, padding
4x4 Conv2dTranspose, 64, stride 2, padding
3x3 Conv2d, 128, stride 2, padding
4x4 Conv2dTranssose, 64, stride 2, padding
Flatten
4x4 Conv2dTranspose, 64, stride 2, padding
Dense - h
3x3 Conv2d, 3, stride 1, padding
(a) The model architecture of the convolutional encoder for image experiments.
(b) The model architecture of the convolutional decoder for image experiments.
Figure A10: The model architecture of the convolutional encoder and decoder for image experiments.
19

---

Under review as a conference paper at ICLR 2022
Type
Model
Online
Observations
10
30
50
100
OBM-Net Sct Transformer
+
0.235
0.162
0.146
0.128
0.390 +
0.388
0.388
0.389
LSTM
+
0.288
0.260
0.269
0.288
Normal
VQ
0.246
0.172
0.147
0.122
Set Transformer
0.295
0.261
0.253
0.247
K-means++
0.183
0.107
0.086
0.066
GMM
0.189
0.118
0.087
0.067
OBM-Net
0.255
0.184
0.164
0.147
LSTM Set Transformer
+
0.306
0.274
0.284
0.290
0.415
0.405
0.407
0.408
Mixed
VQ Set Transformer
0.262
0.192
0.169
0.145
0.309
0.274
0.266
0.261
K-means++
0.206
0.135
0.105
0.088
GMM
0.212
0.136
0.105
0.079
OBM-Net
0.258
0.192
0.173
0.161
LSTM
0.314
0.274
0.288
0.300
Set Transformer
0.394
0.391
0.394
0.394
Enlongated
VQ
0.265
0.194
0.172
0.149
Set Transformer
0.309
0.244
0.240
0.232
K-means++
0.213
0.139
0.113
0.092
GMM
0.214
0.141
0.112
0.086
OBM-Net LSTM
+
0.892
0.794
0.749
0.736
0.799
0.796
0.795
0.794
Set Transformer
0.793
0.794
0.782
0.782
Rotation
VQ
0.956
1.000
1.000
0.984
Set Transformer
0.815
0.784
0.779
0.772
K-means++
0.827
0.834
0.823
0.802
GMM
0.842
0.875
0.867
0.848
OBM-Net
0.375
0.343
0.338
0.334
LSTM Transformer
0.419
0.406
0.405
0.407
0.434
0.424
0.425
0.424
Noisc
VQ Set Transformer
1.479
0,948
0.826
0.720
0.436
0.407
0.398
0.394
K-means+
1.836
1.271
1.091
0.913
GMM
1.731
1.215
1.056
0.856
Table A6: Generalization with Inereased Observations. Error of different models when executed test time different number of observations across different distributions. We train models with components and 30 observations.
20


Reviews:
[{"review_id": "yi_Ztw6qkhW", "reviewer": "Reviewer_V7qP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of encoding \"algorithmic priors\" into deep learning architectures has been explored in the past for various problems and usually is a sound idea and provides benefits to aspects such as complexity, efficacity, or performance of the resulting method over a pure end-to-end learning-based method. As such, the idea proposed in the paper is sensible and has merit.\n\nDespite this, there are several issues with the paper in its current state. One of them is the apparent disconnect between the data association based description and motivation in the beginning, and the clustering focused experimental validation. While it might be reasonable to consider certain aspects to be clustering-like, these similarities should be described and motivated. There also appears to be no comparison to any DAF method to be presented.\n\nThe description of the method itself, while understandable at a high level, lacks detail and specifics. The method seems to utilize various deep learning \"building blocks\" such as slots, attention, etc., without introducing what they are or how their function captures algorithmic aspects. The capturing of algorithmic aspects is another problematic part of the method description, as the algorithm that is being reproduced is never actually described or shown. Therefore, the reader is left wondering what aspects are being replicated and why.\n\nThe experiments are very disconnected from the paper's motivation, and their link to the initially described task is unclear. The second experiment on MNist and planes also seems to be more about the presence or absence of rotational invariance of the employed embeddings, more so than the method itself. Making the comparison confusing and purpose unclear. Another problem with the clustering based evaluation is the metric being used. L2 or similar distance metrics are simply not useful metrics when evaluating clustering results and thus a widely accepted metric such as normalized mutual information would be appropriate to evaluate these results. The last experiment, which seems to be more in the direction of the initially described problem does not appear to have an actual DAF method included in the baselines. Given that the proposed method seemed to be replicating a specific DAF method I would have expected to see at least that method as a baseline.", "summary_of_the_paper": "The paper proposes an end-to-end system for the data association and filtering (DAF) problem. The architecture is built to mimic components typically found in DAF systems to provide a sort of algorithmic prior to the network. The resulting system is evaluated on several different tasks using synthetic data.", "summary_of_the_review": "The general idea of replicating traditional algorithm designs in a deep learning architecture is sensible the lack of detail in the method description and unclear experimental design and evaluation means that the paper is not in a publishable state at this stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "vHEz5iJv5bV", "reviewer": "Reviewer_nrGQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The entity-monitoring problem as formulated in the paper does not seem to pose the challenges outlined in the intro. Post reading the intro I was expecting to see something akin to really long-term tracking with associations across objects even when there are huge gaps in temporal coherence in environments with significant distractors. In contrast, the problem setup focuses on very limited settings with fairly low complexity. \n\n\"In particular, we train a system to construct a memory of the objects in the environment, without prior models of the robot’s sensing, the types of objects to be encountered, or the patterns in which they might move in the environment. \"\n\nI am not sure what without prior models of the robot's sensing means. Since the model is being trained on the agent's input. It is just a question of when the learning is happening (online vs. offline). Since the training is being done online, one of the things that is happening is learning a feature representation for clustering/slotting similar objects together. Why does this feature representation need to be learnt online? Why can we not use a pre-trained representation trained in a self supervised manner with observations collected from the robot. Once we have such a representation how would proposed method compare to just clustering observations on a pre-trained representation. For instance in the qualitative examples shown in Figure 5 I would be surprised if unsupervised pre-trained representations cannot make the same association. \n\nFrom an experimental stand point all the tasks seem fairly non-standard due to definition of the task. There are other datasets like CATER https://rohitgirdhar.github.io/CATER/ which setup versions of the problem. Why setup a different problem rather than use CATER or some of the long term tracking benchmarks https://github.com/wangdongdut/Long-term-Visual-Tracking which involve object re-association?\n\nFrom an architecture and modeling standpoint there does not seem to be significantly different from https://arxiv.org/pdf/2006.15055.pdf. Moreover, the paper does not evaluate on the same benchmark tasks Locatello et.al evaluate on. What is the reason for this? Evaluating on the same benchmarks would allow comparison of differences in modeling details more effectively since the evaluation OBM-Net is empirical. ", "summary_of_the_paper": "The paper defines an entity-monitoring problem where the goal is to identify the distinct objects see in an episode where the agent/model moves through the scene/observes partial state. The paper proposes the OBM-Net model architecture to address this problem and identify all the distinct objects observed over each episode. The key idea of OBM-Net to have fixed set of slots which allow tracking multiple hypothesis over time and use an attention mechanism to update the slots with observations over time. ", "summary_of_the_review": "The entity-monitoring problem as evaluated in the paper does not pose the challenges outlined in the intro. I am not convinced that the proposed method is a better alternative for associating/clustering objects compared to using pre-trained representations on data from the corresponding domains.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LyrpwgQLLcB", "reviewer": "Reviewer_qPet", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is easy to follow, the evaluation proves some potential of the proposed method. However, there are some critical concerns of the paper listed below: \n\nA.\tThis paper addressed the problem “entity monitoring”. Based on the problem formulation, it is very related to the object re-identification and instance retrieval problem. How does it differ? I would expect some highlights on this regard in the introduction and related works. \n\nB.\tAll experiments are performed with synthetic dataset. This does not expose much real-world challenges. Section 5.2 only evaluates on a single object category for 3D objects, i.e.  the airplane. However, often the problem at hand requires the method to handle multiple instances of multiple categories. In an indoor environment, you can easily find multiple chairs in a room together with many other categories of objects. Ref 1) below performed a similar evaluation, but with the full 40 categories of ModelNet40.  Moreover, another critical point from my view is the lack of evaluation on real-world dataset. One can extract the image crops corresponding to each object captured from different viewpoints from some public 3D scene dataset, e.g. ScanNet (http://www.scan-net.org/), as how it is done in Ref 2) below.\n\nC.\tIn line with the above points, there are methods that are closely related but are not covered in the discussion and in the comparison. Some recent related works are listed below for references: \n1)\tBai, Song, et al. \"Re-ranking via metric fusion for object retrieval and person re-identification.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n2)\tBansal, Vaibhav, Gian Luca Foresti, and Niki Martinel. \"Where Did I See It? Object Instance Re-Identification With Attention.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n3)\tHe, Shuting, et al. \"Transreid: Transformer-based object re-identification.\" ICCV 2021.\n4)\tZhang, Yifu, et al. \"Fairmot: On the fairness of detection and re-identification in multiple object tracking.\" International Journal of Computer Vision (2021): 1-19.\n5)\tZhang, Xiao, et al. \"Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n6)\tHuang, Tsung-Wei, et al. \"Multi-View Vehicle Re-Identification using Temporal Attention Model and Metadata Re-ranking.\" CVPR Workshops. Vol. 2. 2019.", "summary_of_the_paper": "This paper addresses the “entity monitoring” problem. They propose a neural-network architecture that uses self-attention as a mechanism for data association to construct a memory of the objects in the environment and demonstrate its effectiveness in a set of illustrative problems.", "summary_of_the_review": "To summaries, the current submission does not situate the method well in the literature because some closely related works are not covered in the paper. The evaluation is also not convincing in terms of how the method can handle real-world challenges and lacks evaluation of real-world data. I therefore vote for a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a long-term object-based memory system for robots.  The proposed method builds on existing ideas of data association filters and neural-net attention mechanisms to learn transition and observation models of objects from labelled trajectories.  The proposed method was compared with baseline algorithms in a set of experiments.

The initial reviews raised multiple concerns about the paper. Reviewers nrGQ and  V7qP commented on the conceptual gap between the problem proposed in the introduction and the extent of the experiments. Reviewer qPet understood the paper to be a form of object re-identification and was concerned about the limited comparisons with related work.  The author response clarified their goal of estimating the states of the objects in the world, which they state is different from the goals of long-term tracking and object reidentification mentioned by the reviewers.  The authors also clarified the relationship to other work in slot-attention and data association filters.  

The ensuing discussion among the reviewers indicated that the paper's contribution remained unclear even after the author response. Two reviewers noted the paper did not clearly communicate the problem being solved (all reviewers had a different view of the problem in the paper).  These reviewers wanted a better motivation for the problem being addressed in this paper.  The third reviewer remained unconvinced that the problem in the paper was different from long-term object tracking.

Three knowledgeable reviewers indicate reject as the contributions of the paper were unclear to all of them. The paper is therefore rejected.
Decision: Reject