Learning Online Data Association
Authors: Yilun Du
Joshua B. Tenenbaum
Tomas Perez
Leslie Pack Kaelbling
Venue:  (2021)
Keywords: Data Association

Abstract: When an agent interacts with a complex environment, it receives a stream of percepts in which it may detect entities, such as objects or people. To build up a coherent, low-variance estimate of the underlying state, it is necessary to fuse information from multiple detections over time. To do this fusion, the agent must decide which detections to associate with one another. We address this data-association problem in the setting of an online filter, in which each observation is processed by aggregating into an existing object hypothesis. Classic methods with strong probabilistic foundations exist, but they are computationally expensive and require models that can be difficult to acquire. In this work, we use the deep-learning tools of sparse attention and representation learning to learn a machine that processes a stream of detections and outputs a set of hypotheses about objects in the world. We evaluate this approach on simple clustering problems, problems with dynamics, and a complex image-based domain. We find that it generalizes well from short to long observation sequences and from a few to many hypotheses, outperforming other learning approaches and classical non-learning methods.

Under review as a conference paper at ICLR 2021
LEARNING ONLINE DATA ASSOCIATION
Anonymous authors
Paper under double-blind review
ABSTRACT
When an agent interacts with a complex environment, it receives a stream of percepts in which it may detect entities, such as objects or people. To build up a coherent, low-variance estimate of the underlying state, it is necessary to fuse information from multiple detections over time. To do this fusion, the agent must decide which detections to associate with one another. We address this data- association problem in the setting of an online filter, in which each observation is processed by aggregating into an existing object hypothesis. Classic methods with strong probabilistic foundations exist, but they are computationally expensive and require models that can be difficult to acquire. In this work, we use the deep- learning tools of sparse attention and representation learning to learn a machine that processes stream of detections and outputs set of hypotheses about objects in the world. We evaluate this approach on simple clustering problems, problems with dynamics, and complex image- based domain. We find that it generalizes well from short to long observation sequences and from a few to many hypotheses, outperforming other learning approaches and classical non-learning methods.
INTRODUCTION
Consider a robot operating in a household, making observations of multiple objects as it moves around over the course of days or weeks. The objects may moved by the inhabitants, even when the robot is not observing them, and we expect the robot to be able to find any of the objects when requested. We will call this type of problem entity monitoring. It occurs in many applications, but we are particularly motivated by the robotics applications where the observations are very high dimensional, such as images. Such systems need to perform online data association, determining which individual objects generated each observation, and state estimation, aggregating the observations of cach individual object to obtain a representation that is lower variance and more complete than any individual observation. This problem can be addressed by online recursive filtering algorithm that receives stream of object detections as input and generates, after each input observation, a set of hypotheses corresponding to the actual objects observed by the agent.
When observations are closely spaced in time, the entity monitoring problem becomes one of tracking and it can be constrained by knowledge of the object dynamics. In many important domains, such as the houschold domain, temporally dense observations are not available, and so it is important to have systems that do not depend on continuous visual tracking.
A classical solution to the entity monitoring problem, developed for the tracking case but extensible other dynamic settings, data association filter (DAF) (the tutorial Bar-Shalm et (2009) provides a good introduction). A Bayes-optimal solution to this problem can be formulated, but it requires representing a number of possible hypotheses that grows exponentially with the number of observations. A much more practical, though much less robust, approach is a maximum likelihood DAF (MI.-DAF), which commits, on each step, to a maximum likelihood data association: the algorithm maintains set of object hypotheses, one for each object (generally starting with the empty set) and for each observation it decides to either: (a) associate the observation with an existing object hypothesis and perform Bayesian update on that hypothesis with the new data, (b) start new object hypothesis based on this observation, or (c) discard the observation as noise.
The engineering approach to constructing a ML-DAF requires many design choices, including the specification of a latent state space for object hypotheses, a generative model relating observations to objects, and thresholds other decision rules for choosing, for new observation, whether to

---

Under review as a conference paper at ICLR 2021
associate it with an existing hypothesis, use it to start new hypothesis, or discard it. In any particular application, the engineer must tune all of these models and parameters to build a DAF that performs well. This is time-consuming process that must be repeated for each new application.
A special case of entity monitoring is one in which the objects' state is static, and does not change over time. In this case, classical solution is online (robust) clustering. Clustering algorithms perform data association (cluster assignment) an state estimation (computing a cluster center).
In this paper we explore training neural networks perform as DAFS for dynamic entity monitoring and as online clustering methods for static entity monitoring. Although it is possible to train an believe that building in some aspects of the structure
unstructured RNN to solve these problems, of the DAF will allow faster learning with less data and allow the system to address problems with longer horizon. We begin by briefly surveying the related literature, particularly focused on learning- based approaches. We then describe a neural-networt architecture that uses self-attention as a mechanism for data association, and demonstrate its effectiveness in several illustrative problems. We find that it outperforms raw RNN as well as domain-agnostic online clustering algorithms, and competitively with batch clustering strategies that can see all available data at once and with state-of-the-an ahrt DAI's for tracking with hand-built dynamics and observation models. Finally, we illustrate its application to problems with images as observations in which both data association and the use of an appropriate latent space are critical.
RELATED WORK
Online clustering methods The typical setting for clustering problems is batch, where all the data is presented to the algorithm at once, and it computes either an assignment of data points to clusters or a set of cluster means, centers, or distributions. We are interested in the online setting, with observations arriving sequentially and a cumulative set of hypotheses output after each observation One of the most basic online clustering methods is vector quantization, articulated originally by Gray (1984) and understood as a stochastic gradient method by Kohonen (1995). It initializes cluster centers at random and assigns each new observation to the closest cluster center, and updates that center to be closer to the observation. Methods with stronger theoretical guaranteees, and those that handle unknown numbers of clusters have also been developed. [Charikar et al.] (2004) formulate the problem of online clustering, and present several algorithms with provable properties. Liberty et al. (2016) explore online clustering in terms of the facility allocation problem, using a probabilistic threshold to allocate new clusters in data. Choromanska and Montelconi (2012) formulate online
clustering as a mixture of separate expert clustering algorithms.
Dynamic domains In the setting when the underlying entities have dynamics, such airplanes observed via radar, large number of DAFS have been developed. The most basic filter, for the case of single entity and no data association problem, is the Kalman filter (Welch and Bishop 2006). In the presence of aastaassooiition uncertainty the Kalman filter can be extended by considering assignments of observations to multiple existing hypotheses under the multiple hypothesis tracking (MHT) filter. A more practical approach that does not suffer from the combinatorial explosion of the MHT is the joint probabilistic data association (JPDA) filter, which keeps only one hypothesis but explicitly reasons about the most likely assignment of observations to hypotheses, [Bar-Shalom al. (2009) provides detailed overview and comparison of these approaches, all of which require hand-tuned transition and observation models.
Learning for clustering There is great deal of work using deep-learning methods to find latent spaces for clustering complex objects, particularly images. Min et al. (0118) provide an excellent survey, including methods with auto-encoders, GANs, and VAES. Relevant our approach are amortized inference methods, including set transformers (Lee et al. 2018) and its specialization to deep amortized clustering (Lee et al. 2019), in which a neural network trained to map directly from data to be clustered into cluster assignments or centers. A related method is neural clustering processes (Pakman al. 2019). which includes an online version, and focuses on generating samples from a distribution on cluster assigmments, including an unknown number of clusters.
Visual data-association methods Data association has been explored in the context of visual object tracking (Luo et al., [2014] Xiang et al., 2015 Bewley et al. 2016 Brasó and Leal- Taixé 2020 Ma et al. 2019 Sun et | [011 Frossard and Urtasun 2018). In these problems, there is typically a fixed visual field populated with many smoothly moving objects. This is an important special case of the general data-association. It enables some specialized techniques that take advantage
2

---

Under review as a conference paper at ICLR 2021
of the fact that the observations of each object are typically smoothly varying in space-time, and incorporate additional visual appearance cues. In contrast, in our setting, there is no fixed spatial field for observations and they may be temporally widely spaced, as would be the case when robot moves through the rooms of a house, encountering and re-encountering different objects as does so. Our emphasis is on this long-term data-association and estimation, and our methods not competitive with specialized techniques on fixed-visual-ieee tracking problems.
Learning for data association There is relatively little work in the area of generalized data association, but et al. (2019) provide recent application of LSTMs (Hochreiter and Schmidhuber 1997) to rich version of the data association problem, in which batches of observations arrive simultancously, with a constraint that each observation can be assigned to at most one object hypothesis. The sequential structure of the LSTM is used here not for recursive filtering, but to handle the variable numbers of observations and hypotheses. is assumed that Euclidean distance is an appropriate metric and that the observation and state spaces are the same. Milan et al. (2017) combine a similar use of LSTM for data association with a recurrent network that learns to track multiple targets. It learns dynamics model for the targets, including birth and death processes, but operates in simple state and observation spaces.
Algorithmic priors for neural networks One final comparison is to other methods that integrate algorithmic structure with end-to-end neural network training. This approach has been applied to sequential decision making by Tamar et al. (2016), particle filters by Jonschkowski et al. (2018). and Kalman filters by Krishnan et al. (2015), as well as to a complex by Karkus al. (2019). The results generally are much more robust than completely hand-buil models and much more sample-efficient than completely unstructured deep-learning. We view our work as instance of this general approach.
robot control system
PROBLEM FORMULATION
The problem of learning to perform online data association requires careful formulation. When the DAF is executed online, will receive a stream of input detections 21 ........ žr where 2 Rd and after each input 2t, it will output two vectors, ye- = [VolkeE(I...) and C2 [ceR)RE(I...R)) where You 1 Cek E (0, 1) and Σn Cak = 1. The y values in the output represent the predicted properties of the hypothesized objects and the e values represent a measure of confidence in the hypotheses, in terms of the proportion of data that each one has accounted for. The maximum number of hypothesis "slots" " is limited in advance to K. In some applications, the z and y values will be in the same space with the same representation, but this is not necessary.
We have training data representing N different data-association problems, D [(=(") mm)))ee(a. L.)(a...)hhea.. where each training example is input/output sequence of length L,, each element of which consists of a pair of input and m {mm)))e(1...(")) which is set of nominal object hypotheses representing the true current state of objects that have actually been observed so far in the sequence. It will always be true that m; C m and _(()) K.
Our objective train a recurrent computational model perform DAF effectively in problems that are drawn from the same distribution as those in the training set. To do so, we formulate model (described in section 4 with parameters 0, which transduces the input sequence 2 , zg into an output sequence (11, ci)..... (UL, CL), and train it to minimize the following loss function:
N L
((0:D)
ΣΣ Cob (v)
m m.)) + Eator (1 ,e)" ..,)))) + Lsparso (1)
The Lubj term is a chamfer loss (Barrow al. 1977). which looks for the predicted y that is closest to each actual my and sums their distances, making sure the model has found good, high-confidence representation for each true object:
Lobj m) = -E min
1
1lve
my
k Ck +
3

---

Under review as a conference paper at ICLR 2021
y
CA
decode
normalize
72
combine
transition
suppress
W0
relevance
update
attention
encode
Figure Architecture of the DAF-Net. Grey boxes represent fixed computations; white boxes represent neural networks with adjustable parameters, those with internal vertical bars represent a replication of the same computation on slot values in parallel. Red lines indicate information derived from input observation, green lines indicate information derived some hypothesis slots values, and blue lines indicate information derived from counts on each hypothesis slot.
The Lalot term is similar, but makes sure that each object the model has found is a true object, where we multiply by Ck to not penalize for predicted objects in which we have low confidence:
Calor (y) c, m)
Σ min ok liyk myll
The sparsity loss discourages the model from using multiple outputs to represent the same true object:
Csparse (c) -oog//cl|
4 DAF-NETS
Inspired by the the basic form of classic DAF algorithms and the ability of modern neural-network techniques learn complex models, we have designed the DAF-Net architecture for learning DAFS and customized procedure for training it from data, inspired by several design considerations. First, because object hypotheses must be available after each individual input and because observations will generally be too large and the problem too difficult to solve from scratch each time, the network will have the structure then fed back for the next. Second, because the loss function is set based, that is, it doesn't 't matter what order the object hypotheses are delivered in, our memory structure should also be permutation invariant, and the memory processing is in the style of attention mechanism. Finally, because in some applications the observations may be in a representation not well suited for hypotheses representation and aggregation, the memory operates a latent representation that is related to observations and hypotheses via encoder and decoder modules.
recursive filter, with new memory values computed on each observation and
Figure shows the architecture of the DAF-Net model. There six modules with adaptable weights and memory that is stored in two recurrent quantities, and n. The main memory is s, which consists of K elements, each in dd,;; the length-K vector positive values encodes how many observations so far have been assigned to each slot. When an input arrives, it is immediately encoded into a vector in dd.. The update network operates on the encoded input and the contents of each hypothesis slot, intuitively producing an update of the hypothesis in that slot under the assumption that the current an observation of the object represented by that slot; so for all slots k.
U update(sk: sk, nig e)
The atlention weights w represent the degree to which the current input "matches" the current value:
exp(attend(ss, e))
Wkk En= exp(attend (k,,)) e))
4

---

Under review as a conference paper at ICLR 2021
To force the network to commit to sparse assignment of observations to object hypotheses while retaining the ability to effectively train with gradient descent, the suppress module sets all but the top M values in w to 0 and renormalizes, to obtain the vector of M values that sum to 1. The vectors are integrated to obtain n, which is normalized to obtain the final output confidence values c.
Additionally, scalar relevance value, (0, is computed from and e; this value is used to modulate the degree to which slot values are updated, and gives the machine the ability to ignore or downweight input. It is computed as
K r= NN1 (avg NN2(e, 8k, 8k, mx)) 1=1
where NN, fully connected network with the same input and output dimensions and NN2 is fully connected network with a sigmoid output unit. The attention output and relevance are now used to decide how combine all possible slot-updates with the old slot values s using the following fixed formula for each slot k:
t1x = raz) 88x + r0g
Because most of the ag values have been set O, this results in a sparse update which will ideally concentrate single slot which this observation is being "assigned."
To compute the outputs, the si slot values are decoded into the representation that required for the outputs, y/:
Vk dccode(s(
Finally, to handle the setting in which object state evolves over time, we can further add dynamics model, which computes the state 8 from the new slot values of using an additional neural network:
S1+1k
NNa(s
These values are fed back, recurrently, as inputs to the overall system.
Given data set D, we train the DAF-Net model end-to-end to minimize loss function L, with slight modification. We find that including the Lsparse term from the beginning of training results in poor learning, but adopting training scheme in which the Leparse first omitted then reintroduced over training epochs, results in reliable training that is efficient in both time and data.
5 EMPIRICAL RESULTS
We evaluate DAF-Net on several entity monitoring tasks, including simple online clustering, monitor- ing objects with dynamics, and high-dimensional image pose prediction in which the observation space is not the same as the hypothesis space. Our experiments aim to substantiate the following claims:
DAF-Net outperforms non-learning clustering methods, even those that operate in batch mode rather than online, because those methods cannot learn from experience to take advantage of information about the distribution observations and true object properties (tables [ [5).
DAF-Net outperforms clustering methods that can learn from previous example problems when
data is limited, because it provides useful structural bias for learning (table 1 and 5).
DAF-Net generalizes to differences between training and testing in (a) the numbers of actual
objects, (b) the numbers of hypothesis slots and (c) the number of observations (tables [ and
DDFF-Net works when significant encoding and decoding are required (table(5).
DAF-Net is able to learn dynamics models and observation functions for the setting when the entities are moving over time (table 4 nearly matching the performance strong data association filters with known ground-truth models.
We compare with the following alternative methods: Batch, non-learning: K-means++ (Arthur and Vassilvitskii] 2007) and expectation maximization (EM) (Dempster et al. 1977) on a Gaussian
mixture model (SciKit Learn implementation): Online, non-learning: vector quantization (Gray 1984); Batch, learning: set transformer (Lee et al., 2018); Online, learning: LSTM (Hochreiter and Schmidhuber, 1997) and an online variant the set transformer (Lee et al. 2018): Dynamic, non-learning: joint probabilistic data association filter (Bar-Shalom al. 2009) All learned network
5

---

Under review as a conference paper at ICLR 2021
Model
Online
Learned
Observations
10
30
50
100
DAF-Net
+
0.235 (0.001)
0.162 (0.001)
0.146 (0.001)
0.128 (0.001)
Set Transformer
0.390 (0.002)
0.388 (0.002)
0.388 (0.002)
0.389 (0.001)
LSTM
+
0.288 (0.001)
0.260 (0.001)
0.269 (0.001)
0.288 (0.001)
VQ
0.246 (0.001)
0.172 (0.001)
0.147 (0.001)
0.122 (0.001)
Set Transformer
+ 0.295 (0.003)
0.261 (0.001)
0.253 (0.001)
0.247 (0.001)
K-means++
0.183 (0.002)
0.107 (0.001)
0.086 (0.001)
0.066 (0.001)
GMM
0.189 (0.002)
0.118 (0.001)
0.087 (0.001)
0.067 (0.001)
Table 1: Comparison performance after training on one thousand Normal distributions for a thousand iterations. We use 3 components, and train models with 30 observations. We report standard error in parentheses. Each cluster observation and center is drawn between -1 and I, with reported error as the L2 distance between predicted and ground truth mean.
Model
Online
Learned
Normal
Elongated
Mixed
Angular
Noise
DAI-Net
0.157 (0.001)
0.191 (0.001)
0.184 (0.001)
0.794 (0.001)
0.343 (0.001)
Transformer
0.407 (0.001)
0.395 (0.001)
0.384 (0.001)
0.794 (0.003)
0.424 (0.001)
LSTM
0.256 (0.001)
0.272 (0.001)
0.274 (0.001)
0.799 (0.002)
0.408 (0.001)
VQ
0.173 (0.002)
0.195 (0.002)
0.191 (0.002)
0.992 (0.004)
0.947 (0.002)
Set Transformer
0,226 (0.001)
0.248 (0.001)
0.274 (0.001)
0.816 (0.001)
0.406 (0.002)
K-means++
0.103 (0.001)
0.139 (0.001)
0.135 (0.001)
0.822 (0.003)
1.259 (0.002)
GMM
0.113 (0.001)
0.141 (0.001)
0.136 (0.001)
0.865 (0.003)
1.207 (0.002)
Table 2: Comparison of performance on clustering after 30 itcration when training on 1000 different distributions for a thousand iterations. We use a total of 3 components, and train models with 30 observations. We report standard error in parentheses. Each cluster observation and center is drawn between -1 and 1, except for angular which is drawn between - and pi with reported error the L2 distance between predicted and ground truth mean.
architectures are set to have about 50000 parameters. We provide additional details about architecture and training the appendix. The set transformer is a standard architecture that has been evaluated on clustering problems in the past.
All models except DAF-Net are given the ground truth number of components K, while DAF-Net uses 10 hypothesis slots. Results are reported in terms of loss >; min& 1| -myll (with the most confident K hypotheses selected for DAF-Net).
Gaussian domains To check the basic operation of the model and understand the types of problems for which it performs well, we tested in simple clustering problems with the same input and output spaces, but different types of data distributions, each mixture of three components. We train on 1000 problems drawn from each problem distribution distribution and test on 5000 from the same distribution. In every case, the means of the three components are drawn at random for each problem.
1. Normal: Each component is a 2D Gaussian with fixed identical variance across each individual dimension and across distributions. This is basic "sanity check."
2. Elongated: Each component is 2D Gaussian, where the variance along each dimension is drawn from uniform distribution, but fixed across distributions.
3. Mixed: Each component is a 2D Gaussian, with fixed identical variance across each individual dimension, but with the variance of each distribution drawn from a uniform distribution.
4. Angular: Each component is 2D Gaussian with identical variance across dimension and distribution, but points above и are wrapped around to - 7 and points below - ग wrapped to æ
5. Noise: Each component has 2 dimensions parameterized by Gaussian distributions, but with the values of the remaining 30 dimensions drawn from a uniform centered at
We compare approach to each of the non-dynamic baselines for the five problem distributions in T a complete listing of results for all the distributions can be found in the Appendix. The results in this table show that Normal, Mixed, and Elongated tasks, DAF-Net performs comparably the offline clustering algorithms, even though it is running and being evaluated online. On the Angular and Noise tasks, Det-Net is able to learn useful metric for clustering and outperforms both offline

---

Under review as a conference paper at ICLR 2021
31
13
Gaussian Domain
Dynamic Domain
Figure 2: Visualizations of Dynamic and Gaussian domains. Observations are transparent while while ground truth states are bolded
Model
Slots
Ground Truth Clusters
7
10
0.162 (0.001)
0.214 (0.001)
0.242 (0.001)
DAF-Net
20
0.175 (0.001)
0.195 (0.001)
0.213 (0.001)
30
0.188 (0.001)
0.197 (0.001)
0,205 (0.001)
Set Transformer
0.261 (0.001)
0.279 (0.001)
0,282 (0.001)
Vector Quantization
0.171 (0.001)
0.199 (0.001) 0.205 (0.001)
Table 3: Quantitative evaluation of DAF-Net on distributions with different numbers of true components and hypothesis slots test time with 30 observations. In all cases, DAF-Net is trained with 3-component problems, 10 slots, and observations. We compare with an offline set transformer trained with different numbers of problem components as well as with vector quantization.
and online alternatives (with additional analysis in the appendix showing DAF-Net outperforms all other learning baselines with more training distributions in the Angular task).
In Table T we evaluate the quality of predictions after 10, 30, 50, and 100 observations in the Normal distribution. We find that DAF-Net generalizes well to increased numbers of observations, with predictions becoming more accurate as the observation sequence length increases, despite the fact that it is trained only observation sequences of length 30. This is in contrast with other online learning baselines, set transformer and LSTM, which both see increases in error after 50 or 100 observations. This pattern holds across all the test problem distributions (see Appendix).
In Table 3 we investigate the generalization ability of DAF-Net to both increases in the number of hypothesis slots and the underlying number of mixture components from which observations are drawn. We compare to the offline set transformer and to VQ, both of which know the correct number of components test time. Recall that, to evaluate DAF-Net even when it has large number of extra slots, we use its K most confident hypotheses. We find that DAF-Net generalizes well to increases hypothesis slots, and exhibits improved performance with large number of underlying components, performing comparably to or better than the VQ algorithm. We note that none of the learning basclines can adapt to different numbers cluster components at test time, but find that DAF-Net outperforms the set transformer even when it is trained the ground truth number of clusters in the test. We also ablated each component of our model and found that each of our proposed components enables both better performance and generalization. Detailed results of the ablations and a figure illustrating the clustering process are the appendix.
Dynamic Domains We next evaluate the ability of DAF-Net to perform data association in domains where objects are moving dynamically over time. This domain is typical of tracking problems considered by data association filters, and we compare with the de-facte standard method, Joint Probabilistic Data Association (JPDA), which uses hand-built ground models. We consider setup consisting of 3 different moving objects in 2D. Their velocity is perturbed at each step by an additive component drawn from Gaussian distribution and observations of their positions (but no made with Gaussian error. To perform well in this task, model must
observations of velocities) discover that it needs to estimate the latent velocity of each object, as well as learn the underlying dynamics and observation models.
7

---

Under review as a conference paper at ICLR 2021
Model
Observations
10
20
30
40
DAF-Net
0.322 (0.009)
0.187 (0.007)
0.168 (0.008)
0.195 (0.014)
Online Set Transformer
4.588 (0.006)
4.499 (0.005)
4.42 (0.006)
4.43 (0.005)
LSTM
0.348 (0.007)
0.390 (0.011)
0.506 (0.011)
0.687 (0.018)
JPDA (ground truth)
0.316 (0.022)
0.157 (0.016) 0.142 (0.008)
0.141 (0.002)
Table 4:
Comparison of performance on position estimation of 3 dynamically moving objects. All learning models are trained with 1000 sequences of 30 observations. We report standard error in parentheses. JPDA uses the ground-truth observation and dynamics models.
Type
Model
Learned
Observations
10
100
DAF-Net LSTM
7.143 (0.006)
5,593 (0.004)
5.504 (0.004)
5,580 (0.004)
9.980 (0.005)
9.208 (0.004)
9.166 (0.004)
9.267 (0.004)
MNIST
K-means (Pixel) K-means (Learne))
13.214 (0.005)
12,010 (0.005)
11.961 (0.004)
11.719 (0.004)
13.596 (0.006)
12.505 (0.005)
12.261 (0.003)
12.021 (0.004)
DAF-Net
4,558 (0.005)
4.337 (0.004)
4.331 (0.004)
4.325 (0.004)
LSTM
5.106 (0.003)
4.992 (0.005)
4.983 (0.003)
4,998 (0.004)
Airplanes
K-means (Pixel)
7.127 (0.006)
6,890 (0.004)
6,603 (0.004)
6,517 (0.004)
K-means (Learned)
7.246 (0.006)
6,943 (0.005)
6,878 (0.005)
6.815 (0.004)
Table 5: Comparison of performance of online clustering on MNIST and on rendered Airplane datasct. For DAF-Net, LSTM and K-means (Learned) use convolutional encodcr/decoder trained on the data; for K-means (Pixcl) there is no encoding. We use a total of 3 components and train models with 30 observations. Models are trained on 20000 problems on both datasets.
We compare our approach to the Set Transformer and LSTM methods, well as JPDA with ground-trute models. The basic clustering methods have no ability to handle dynamic systems so omit them from the comparison. The learning methods (DAF-Net, Set Transformer, and LSTM) are all trained on observation sequences of length 30. We test performance of all four methods on sequences of multiple lengths. Quantitative performance, measured in terms of prediction error on true object locations, is reported in Table 4 We can see that the online Set Transformer cannot learn reasonable model all. The LSTM performs reasonably well for short (length 10) sequences but quickly degrades relative to DAF-Net and JPDA as sequence length increases. We note that DAF-Net performs comparably to but just slightly worse than JPDA. This is very strong performance because DAF-Net is generic and can be adapted to new domains given training data without the need to hand-design the models used by JPDA.
Image-based domains We now evaluate the ability of DAF-Net to perform data association in domains with substantially more complex observation spaces, where the outputs are not simple averages of the inputs. This requires the network to synthesize a latent representation for slots in which the simple additive update performs effectively.
We investigate this with two image-based domains. In each domain, we have set of similar objects (digits or airplanes). A problem is constructed by selecting К objects from the domain, and the desired y values are images of those objects in a canonical viewpoint. The input observation sequence generated by randomly selecting one of those K objects, and then generating an image of it from a random viewpoint as the observation 2.
Each object is a random digit image in MNIST, with observations corresponding to that same image rotated, and the desired outputs being the un-rotated images; (2) Airplane: Fach object a random object from the Airplane class in ShapeNet (Chang et al. 2015). with observations corresponding to airplane renderings (using Blender) at different viewpoints and the desired outputs the objects rendered a canonical viewpoint.
Our two domains are: (1) MNIST:
For MNIST, use the 50000 digit images in the training set to construct the training problems, and the 10000 images in the non-overlaping test set to construct the test problems. For the Airplane dataset, we 1895 airplanes objects to construct the training problems, and 211 different airplanes objects to construct the test problems. Each object is rendered with 300 viewpoints. Of our baseline methods, only batch K-means (in pixel space) can be directly applied this problem with even
8

---

Under review as a conference paper at ICLR 2021
Tminnng Objectss
Triinnn Vaternations
Tmining Ofjects
Training Observation
7 4
X
7
V
9
4
S
8 3
S
8
Y
4
0
>
O
4
7
0
2
2 Decoded Slots
A
Y
4
O
Ground Truth Chjects
Ground Truth Oljects
Decoded Sloes
Figure 3: Results two image-based association tasks (left: MNIST, right: airplanes). At the top of each example training problem, illustrated by the true objects and observation sequence. Each the next rows shows an example test problem. with ground truth objects and decoded slot values. three highest-confidence hypotheses for each problem are highlighted red, and correspond nicely to the ground-truth objects.
reasonable results. We also include versions of LSTM and of batch K-means that operate latent representation that is learned first using an auto-encoder. In Table [ we find that our approach significantly outperforms other comparable baselines in both accuracy and generalization. We visualize qualitative predictions from our model in Figure 3
DISCUSSION
This work has demonstrated that using algorithmic bias inspired by a classical solution to the problem of filtering estimate the state of multiple objects simultaneously. coupled with modern machine- learning techniques, we can arrive at solutions that learn to perform and generalize well from comparatively small amount of training data.

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++:: the advantages of careful seeding. In Symposium on Discrete Algoritims '07, 2007.
Yaakov Bar-Shalom, Fred Daum, and Jim Huang. The probabilistic data association filter. IEEE Control Systems Magazine, December 2009.
Harry Barrow. Jay M. Tenenbaum, Robert C. Bolles, and Helen Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In IJCAI, 1977.
Alex Bewley, Zongyuan Ge, Lionel Out, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking, In 2016 IEEE International Conference on Image Processing (ICIP), pages 3464 3464-3468, 2016.
Guillem Brasó and Laura Leal-Taixé. Learning neural solver for multiple object tracking. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv:15/2.03012, 2015.
Moses Charikar, Chandra Chekuri, Tomás Feder, and Rajeev Motwani. Incremental clustering and dynamic information retrieval. SIAM Journal on Computing, 33(6):1417-1440, 2004.
Anna Choromanska and Claire Monteleoni. Online clustering with experts. In Artificial Intelligence and Statistics, pages 227-235, 2012.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
Davi Frossard and Raquel Urtasun, End-to-end learning of multi-sensor 3d tracking by detection. In ICRA, May 2018.
R. Gray. Vector quantization. JEEE ASSP Magazine, 1(2):-29,, 1984.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput.. 933::1735 1997.
Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. ArXiv, ahs/1805.11111 11122, 2018.
Peter Karkus, Xiao Ma, David Hsu, Leslie Pack Kaelbling, Wee Sun Lee, and Tomas Lozano-Perez. Differen- tiable algorithm networks for composable robot learning. ArXiv, abs/1905. 16602, 2019.
Teuvo Kohonen. Self-Organizing Maps. Peringer-Verlage 1995.
Rahul Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. ArXiv, abs//511..0000 1,0511,, 2015.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework attention based permutation iomuttaionninvariint neural networks. arXiv preprint arXiv:1810.00825 2018.
Juho Lee, Yoonho Lee, and Yee Whye Teh. Deep amortized clustering. ArXiv, abs/1909.1 13433, 2019.
Edo Liberty. Ram Sriharsha. and Maxim Sviridenko. An algorithm for online k-means clustering. In 2016 Proceedings of the eighteenth workshop algorithm engineering and experiments (ALENEX), pages 81-89. SIAM, 2016.
Huajun Liu, Hui Zhang, and Christoph Mertz. DeepDA: LSTM-based deep data association network for multi-targets tracking in clutter. 2019 22th International Conference Information Fusion (FUSION), pages 1-8, 2019.
Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao, and Tac Kyun Kim. Multiple object tracking: literature review, arXiv preprint 140.:::1199763811 2014.
Cong Ma, Yuan Li, Fan Yang. Ziwei Zhang, Yueqing Zhuang, Huizhu Jia, and Xiaodong Xie. Deep association: End-to-end graph-based learning for multiple object tracking with conv-graph neural network. In International Conference on Multimedia Retrieval (ICMR). June 2019.
Anton Milan, Seyed Hamid Rezatofighi, Anthony R. Dick, Ian Reid, and Konrad Schindler. Online multi-target tracking using recurrent neural networks. ArXiv, abs/1604.03633. 2017.
10

---

Under review as a conference paper at ICLR 2021
Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514, 2018.
Ari Pakman, Yueqi Wang. Catalin Mitelut, Jinhyung Lee, and Liam Paninski. Neural clustering processes. arXiv: Machine Learning, 2019.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. Advances neural information processing systems, pages 7299-73111 9319, 2018.
ShiJie Sun, Naveed Akhtar, HuanSheng Song, Ajmal Mian, and Mubarak Shah. multiple object tracking. In TPAMI, June 2019.
Deep affinity network for
Aviv Tamar, Sergey Levine, Pieter Abbeel, Wu, and Garrett Thomas. abs/1602.02867, 2016.
Value iteration networks. ArXiv,
Greg Welch and Gary Bishop. An introduction to kalman filter. 2006.
Yu Xiang, Alexandre Alahi, and Silvio Savarese. Learning track: Online multi-obeect tracking by decision making. Proceedings the IEEE international conference on computer vision, pages 4705-4713, 2015.

---

Under review as a conference paper at ICLR 2021
A.1 APPENDIX
A.2 DISCOVERY OF OBJECTS
In contrast to other algorithms, DAF-Net learns predict both a of object properties yıe of objects and set of confidences for each object. This corresponds to the task of both predicting the number of objects in of observations, as well as associated object properties. We evaluate the ability to regress object number in DAF-Net in scenarios where the number of objects is different than that of training. We evaluate on the Normal distribution with variable number of component distributions, and measure inferred component through threshold confidence. DAF-Net is trained dataset with underlying components.
We find in Figure A that DAF-Net is able to infer the presence of more component distributions (as they vary from 3 to 6), with improved performance when cluster centers are sharply separated (right figure of Figure A1
Noisy Cluster
Seperated Cluster
35
Inferred Componens
Inferred
3.5
40
Companents
Components
Figure Al: Plots of inferred number of components using confidence threshold DAF-Net compared to the ground truth number of clusters (DAF-Net is trained only 3 clusters). We consider two scenarios, noisy scenario where cluster centers are randomly drawn from -1 (left) and scenario where all added cluster components well seperated from each other (right). DAF-Net is able to infer more clusters in both scenarios, with better performance when cluster centers are more distinct from each other.
A.3 QUALITATIVE VISUALIZATIONS
We provide an illustration of our results on the Normal clustering task in Figure A2 decoded values of hypothesis slots in red, with size scaled according confidence, and ground-truth cluster locations in black. DAF-Net able to selectively refine slot clusters to be close to ground truth cluster locations even with much longer observation sequences than it trained
We plot the
We find that each component learned by DAF-Net is interpretable. We visualize attention weights of each hypothesis slot in Figure A3 and find that each hypothesis slot learns attend to local region next to the value decodes to. We further visualize a plot of relevance weights in Figure A4 across increasing number observations over different levels of noise in each distribution. We find that as
Obs 10
Obs 30
Obs 50
Obs 100
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
-0.5
1.0
1.0
1.0
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Figure A2: Illustaatoon the clustering process. Decoded value of hypothesis (with size correspond- ing to confidence) shown in red, with ground truth clusters in black. Observations are shown in blue.
12

---

Under review as a conference paper at ICLR 2021
more observations, the relevance weight of new observations decreases time, indicating that DAF-Net learns to pay the most attention towards the first set of observations sees. addition, find that in distributions with higher variance, the relevance weight decreases more slowly, later observations are now more informative in determining cluster centers.
Update Magribude over Updattes
Obs 100
1.0
1.0
Nosw
1.0
Noise
0.5
0.8
Nonee
0.5
9.6
0.0
0.0
Update:
9.4
-0.5
-0.5
-1.0
-1.0
-1.0
-0.5
0.0
1.0
-1.0
0.5
0.0
0.5
9.0
60
80
100
Disvervations
Figure A3: Plot of (left), and what slot each input assigns
the highest attention towards (right) (each slot is colored differently, Figure A4: Plots of magnitude with assigned inputs colored in the same way). Note alignment of relevance weights with increased obser- vation number on different distributions with higher standard deviation (noise),
regions on the right with point density the left.
A.4 QUANTITATIVE RESULTS
We report full performance of each different model across different distributions in Table 6 We find that DAF-Net is able obtain better performance with increased number of observations across different distributions. In addition DAf-Net out-performs neural network baselines when evaluated on 30 observations across distributions except for rotation. For rotation we find that when training with 10,000 different distribution, DAF-Net exhibits better performance of 0.555 compared to Set Transformer Online performance of 0.647 and LSTM performance of 0.727.
A.5 SPARSITY LOss
In this section, we show that Lypame (c) encourage confidences to sparse. Recall that
Laparso (c) = log//e||
(1)
where ||c|| is the L2 norm which is convex. Recall that c, the confidence vector, defines polyhedron, since is the set of points that are non-negative, and whose element sum up to one. The maximum of convex function
a polyhedra must occur at the vertices, which correspond to assignment of I to a single & and Os to every other value of e. Next we consider the minimum of |[c|] given that it's elements sum up to one. This is equivalent finding the stationary points of the Legragian
E
Σ G 1)
By taking the gradient of the above expression, we find that the stationary value corresponds to each e; being equal. Since the function is convex, this corresponds to the minimum of ||c|]- Thus Esparse is maximized when each individual confidence is equal.
A.6 PERFORMANCE USING MORE CLUSTERS
We measure the performance DAF-Net the presence of a large number of clusters and slots. We consider the Normal distribution setting, where input observations are generated by total of 30 difference clusters. We train DAF-Net with 50 observations, and measure performance at inferring cluster centers with either 50 or 100 observations. We report performance in Table 7 and find that DAF-Net approach obtains good performance in this setting, out-performing both online and offline baselines.
13

---

Under review as a conference paper at ICLR 2021
Type
Model
Online
Observations
10
30
50
100
DAF-Net
0.235 (0.001)
0.162 (0.001)
0.146 (0.001)
0.128 (0.001
Set Transformer
0.390 (0.002)
0.388 (0.002)
0.388 (0.002)
0.389 (0.001)
LSTM
0.288 (0.001)
0.260 (0.001)
0.269 (0.001)
0.288 (0.001)
Normal
VQ
0.246 (0.001)
0.172 (0.001)
0.147 (0.001)
0.122 (0.001)
Set Transformer
0.295 (0.003)
0.261 (0.001)
0.253 (0.001)
0.247 (0.001)
K-means++
0.183 (0.002)
0.107 (0.001)
0.086 (0.001)
0.066 (0.001)
GMM
0.189 (0.002)
0.118 (0.001)
0.087 (0.001)
0.067 (0.001)
DAF-Net
0.255 (0.002)
0.184 (0.001)
0.164 (0.001)
0.147 (0.001)
LSTM
0,306 (0.002)
0.274 (0.001)
0.284 (0.001)
0.290 (0.001)
Set Transformer
0.415 (0.002)
0.405 (0.001)
0.407 (0.001)
0.408 (0.001)
Mixed
VQ
0.262 (0.002)
0. (0.001)
0.169 (0.001)
0,145 (0.001)
Set Transformer
0.309 (0.002)
0.274 (0.001)
0.266 (0.001)
0.261 (0.001)
K-means++
0.206 (0.003)
0.135 (0.001)
0.105 (0.001)
0.088 (0.001)
GMM
0.212 (0.003)
0.136 (0.001)
0.105 (0.001)
0.079 (0.001)
DAF-Net LSTM
0,258 (0.002)
0.192 (0.001)
0.173 (0.001)
0.161 (0.001)
Enlongated
0.314 (0.003)
0.274 (0.002)
0.288 (0.001)
0.300 (0.001)
Set Transformer
0.394 (0.003)
0.391 (0.003)
0.394 (0.003)
0.394 (0.003)
VQ
0.265 (0.003)
0.194 (0.002)
0.172 (0.001)
0.149 (0.001)
Set Transformer
0.309 (0.002)
0,244 (0.002)
0.240 (0.001)
0.232 (0.001)
K-means++
0,213 (0.002)
0.139 (0.001)
0.113 (0.001)
0.092 (0.001)
GMM
0.214 (0.002)
0.141 (0.001)
0.112 (0.001)
0.086 (0.001)
DAF-Net
0.892 (0.001)
0.794 (0.001)
0.749 (0.002)
0.736 (0.001)
Rotation
LSTM
0.799 (0.003)
0.796 (0.002)
0.795 (0.002)
0.794 (0.002)
Set Transformer
0.793 (0.003)
0.794 (0.002)
0.782 (0.002)
0.782 (0.002)
VQ
0.956 (0.003)
1.00 (0.003)
1.00 (0.003)
0.984 (0.003)
Set Transformer
0.815 (0.003)
0.784 (0.002)
0.779 (0.002
0.772 (0.002)
K-means+
0.827 (0.004)
0.834 (0.003)
0.823 (0.002)
0.802 (0.001)
GMM
0.842 (0.004)
0.875 (0.001)
0.867 (0.003)
0.848 (0.002)
DAF-Net LSTM
0.375 (0.001)
0.343 (0.001)
0.338 (0.001)
0,334 (0.001)
Noise
0.419 (0.001)
0.406 (0.001)
0.405 (0.001)
0.407 (0.001)
Set Transformer
0.434 (0.001)
0.424 (0.001)
0.425 (0.001)
0.424 (0.001)
VQ Set Transformer
1.479 (0.002)
0.948 (0.002)
0.826 (0.001)
0.720 (0.001)
0.436 (0.001)
0,407 (0.002)
0.398 (0.001)
0.394 (0.001)
K-means+
1.836 (0.002)
1,271 (0.002)
1.091 (0.002)
0,913 (0.002)
GMM
1.731 (0.002)
1,215 (0.002)
1.056 (0.002)
0.856 (0.002)
Table Comparison of performance under different settings after training on different distribution for thousand iterations. We use total of 3 components, and train models with 30 observations. We report standard error in parentheses.
Model
Online
Observations
50
65
80
100
DAF-Net
0.158 (0.001)
0.154 (0.001)
0.151 (0.001)
0.147 (0.001)
VQ
0.162 (0.001)
0.157 (0.001)
0.153 (0.001)
0.148 (0.001)
K-means++
0.155 (0.001)
0.151 (0,001)
0.148 (0.001)
0.146 (0.001)
GMM
0.156 (0.001)
0.151 (0.001)
0.149 (0.001)
0.147 (0.001)
Table Comparison performance on Normal distribution. We use 30 components, and train models with 50 observations. We report standard error in parentheses. Each cluster observation and center is drawn between -1 and 1, with reported error as the L2 distance between predicted and ground truth means.
A.7 DISTRIBUTIONS DETAILS
We provide detailed quantitative values for each distribution below. Gaussian centers are drawn uniformly from -1 to I.
14

---

Under review as a conference paper at ICLR 2021
Model
Learned
Observations
30
100
DAF-Nct
+
13.207 (0,003)
13.214 (0.004)
18.318 (0.006)
14.743 (0.003)
LSTM
15.076 (0. 008)
15.132 (0.006)
19.134 (0.004)
16.071 (0.006)
K-means (Pixel)
18.378 (0.005)
18.817 (0.005)
20.269 (0.005)
18.407 (0.006)
K-means (Learned)
31.127 (0.003)
31.127 (0.003)
31.127 (0.003)
31.127 (0.003)
Table 8: Comparison of performance on regressing the ground truth un-rotated version of an MNIST digit where digits move over time. For DAF-Net, LSTM and K-means (Learned) we use a convolutional encoder/decoder trained on the data; for K-means (Pixel) there is encoding. We total of 3 components and train models with 30 observations. We report MSE error with respect to ground truth unrotated images.
1. Normal: Each 2D Gaussian has standard deviation 0.2.
2. Mixed: Each distribution is a 2D Gaussian, with fixed identical variance across each individual dimension, but with the standard deviation of each distribution drawn from a uniform distribution from (0.04, 0.4).
3. Elongated: Each distribution a 2D Gaussian, where the standard deviation along each dimen- sion is drawn from uniform distribution from (0.04, 0.4), but fixed across distributions.
4. Angular: Each distribution is 2D Gaussian with identical standard deviation across dimension and distribution, but points above are wrapped around to -x and points below -x wrapped to
n. Gaussian means are selected between (-7T, -2m/3) and between (2 /3, T). The standard deviation of distributions is 0.3
5. Noise: Each distribution has 2 dimensions parameterized by Gaussian distributions with standard deviation 0.5, but with the values of the remaining 30 dimensions drawn from uniform distribution from (-1,1).
A.8 DYNAMIC IMAGES
We further compare DAF-Net with other baselines in the setting where the rendered images move over time. We follow the same setup described in image-basee based domain, but now consider the MNIST setup with each digit centered at a random position in the image (with parts of a digit that are outside of the image wrapped around to the other side of the image). At each timestep, the center of each moves with a constant velocity, with the goal to predict the un-rotated image at the current center of digit. We report results in Table and find that our approach performs well in this setting also.
A.9 MODEL/BASELINE ARCHITECTURES
Dense + h
Dense + h
Dense - h
Dense -> h
Dense - h
Dense +
LSTM(h)
DAF-Net Memory
Set Transformer Encoder
Dense + h
Dense
Set Transformer Decoder
Dense + output
Dense - output
(b) The model architecture of the Set Transformer baseline. The hidden dimension i 48 for the synthetic Gaussian dis- tributions. We use the encoder and decoder defined in (Lee) al.] (0118) with 4 heads and hidden dimension h.
(a) The model architecture of the L.ST baseline. The hid- den dimension h used is 96 for synthetic Gaussian distri- butions and 128 for Image datasets. For image experi- ments, the first 2 and last 2 fully connected layers placed with image encoders and decoders.
(c) The model architecture of DAF-Net. The hidden dimension h is 64 is for syn- thetic Gaussian distributions and 128 for the image experiments. We detail in component the memory of DAF-Net memory below. For image experiments, the first 2 and last 2 fully connected lay- are replaced with image encoders and decoders.
Figure A5: Architecture of different models.
15

---

Under review as a conference paper at ICLR 2021
5x5 Conv2d, 32, stride padding
Dense -+ 4096
3x3 Conv2d, 64, stride 2, padding
Reshape (256, 4, 4)
3x3 Conv2d, 64, stride 2, padding
4x4 Conv2dTranspose, 128, stride 2. padding I
3x3 Conv2d, 64, stride 2, padding
4x4 Conv2dTraansoose, 64, stride 2, padding
3x3 Conv2d, 128, stride 2, padding
4x4 Cony2dTrrasspose, 64, stride 2, padding 1
Flatten
4x4 Cons2dTrrassoose, 64, stride 2, padding
Dense
3x3 Conv2d, stride I, padding
(a) The model architecture of the convolutional encoder for image experiments.
(b) The model architecture of the convolutional decoder for image experiments.
Figure A6: Architectures of encoder and decoder models image experiments.
We provide overall architecture details for LSTM in Figure A5a for the set-transformer in Figure and DAF-Net in Figure A5c For image experiments, we provide the architecture of the encoder in Figure A6a and decoder Figure Both LSTM, DAF-Net, and autoencoding baselines use the same image encoder and decoder.
In DAF-Net memory, the function update(ss, (sk, rie e) implemented by applying 2 layer MLP with hidden units h which concatenates the vectors 81; 12, as input and outputs new state ua of dimension h. The value nt is encoded using the function I+*, to normalize the range of input to be between 0 and 1. The function attend Fig. e) is implemented an analogous way to update, using 2 layer MLP that outputs a single real value for each hypothesis slot.
For the function relevance (8k; Tik, e), we apply NN1 per hypothesis slot, which is implemented as 2 layer MLP with hidden units that outputs intermediate state of dimension h. (5k fgg e) are fed into NN; in an analogous manner to update. NN₂ is applied to average of the intermediate representations each hypothesis slot and is also implemented as layer MLP with hidden unit size h, followed by sigmoid activation. We use the ReLU activation for all MIPs.
A.10 BASELINE DETAILS
All baseline models are trained using prediction slots equal to the ground truth of components. To modify the transformer to act in an online manner, follow the approach in (Santoro al.) 2018) and we apply the Set Transformer sequentially on the concatenation of an input observation with hypothesis slots. Hypothesis slots updated based off new values of the slots after applying self-attention (Set Transformer Encoder). We use the Chamfer loss to train baseline models, with confidence set 1.
A.11 ABLATION
We investigate ablations of our model in Table 9 We ablate the components of sparsity loss, learned memory update, suppression attention weights and relevance weights. We find that each component of our model contributes to improved performance.
16

---

Under review as a conference paper at ICLR 2021
Sparsity
Learned Memory
Supression
Relevance
Observations
10
100
0.382 (0.003)
0.452 (0.003)
0.474 (0.003)
0.487 (0.003)
0.384 (0.001)
0.412 (0.001)
0.423 (0.001)
0.430 (0.003)
0.335 (0.002)
0.357 (0.002)
0.366 (0.003)
0.387 (0.001)
0.279 (0.001)
0.274 (0.001)
0.278 (0.001)
0.282 (0.001)
0.238 (0.001)
0.157 (0.001)
0.137 (0.001)
0.131 (0.001)
Table 9: We ablate each components of DAF-Net on the Normal distribution. When learned memory ablated, DAF-Net updates states based on observed values (appropriate in the Normal Distribution dataset). We report Leluster of predictions and report standard error in parentheses. We find that each proposed component of our model is important for improved performance.
17


Reviews:
[{"review_id": "YiuZXrKS_Al", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims to solve the data association problem in a general setting, which is suitable for various scenarios such as online clustering and online tracking. The motivation of having this setting is that temporal dense observations are not always available. \n\nThe high level idea of this work is to learn a neural network which updates the hypothesis ($y$) and outputs the association ($c$). I like the design that are well motivated by the problem itself and the ability of neural networks: 1) recursive filter for online data; 2) order invariant memory structure; 3) representation learning.\n\nThe experiments are conducted on many domains and the proposed approach is compared with many methods from different domains. Ablation study is also conducted to demonstrate the effectiveness of the design decisions. It seems thorough. However, I think the experiments are more like toy experiments and it would be great if it could demonstrate its superiority on a more practical dataset such as a multiple object tracking (MOT) benchmark. In my opinion, data association is the key for MOT. But definitely other problems with a practical setting can also convince me. Another concern about the experiment is that the loss is used as the metric, which seems weired to me. The clustering metrics like F-measure or Rand index and/or precision and recall might be better.\n\n\nOther suggestions and clarifications:\n\n1. I am very interested in the sparsity loss. Is there any references or justifications which shows that it will leads to a sparse result. \n\n2. Is it possible to use a classification (cross entropy) loss for $c$? Will the order invariant property cause any issue here?\n\n3. An additional question about $c$: how is it normalized? \n\n4. Why \"the basic clustering methods have no ability to handle dynamic systems\"? I assume they don't have the dynamic part, but as long as they can observe the object `\"feature\", they are applicable. Though I guess the performance won't be good.\n\n5. In the image-based domains, is it generally better than the classification methods? \n\nMissing reference:\nGuillem Braso and Laura Leal-Taixe. Learning a Neural Solver for Multiple Object Tracking. In CVPR 2020. This work aims to solve the data association problem for Multiple Object Tracking based on Message Passing Networks. \n\n=== Post rebuttal ===\n1. As reviewer 2, I am also not convinced by the reason why it could not be evaluated on a MOT dataset. The explanation suggests that the proposed method is more general and it should be able to apply it to broader domains (and thus MOT could be one of them). \n2. The evaluation metric still looks suspicious to me. I can imagine we can usually get better numbers if we can directly minimize the metrics we want to evaluate. However, in my opinion, data assocision usually need to do hard assginments if we want to use it in practice (unless it is a intermediate task and in that case we can do soft assignments), so I do think it is better to set a hard threshold or draw a curve of F1 v.s. threhold. In the meantime,  I think it is unfair for some other methods, like k-means++ becaue it is minimizing a different objective.\n\nI still think the method has some merits so I am at borderline, though I will not defend it if it is rejected.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V1GG4vq0PKW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes DAF-Net, an attention-based architecture for online filtering, which is used for a flexible set of tasks: online clustering, tracking, and image association.\n\n**Strength**: I like the idea of using attention for updating temporal beliefs, which is natural and potentially a backbone for a wide range of temporally extended tasks. The architecture looks reasonable and handles observation encoding, dynamics updating, and online association altogether, and the presented numbers look good across three experiments.\n\n**Weakness**: See **Clarity** and **Questions**.\n\n**Clarity**: I find the paper not fully accessible at the first sight.\n- Section 3 \"Problem Formulation\" is quite abstract and unintuitive, and some motivation before description might be good. I'm not sure why the loss function is part of problem formulation rather than the method.\n- Section 4 \"DAF-Nets\" presents some parts I do not understand, e.g. the format of $n$ vector, and the sentence \"The $a$ vectors are integrated to obtain $n$, which is normalized to obtain the final output confidence values $c$\". The use of $s_k$ and $s_{tk}$ are inconsistent. \n- Figure 1 looks complicated and the meaning of colors of edges are not annotated.\n- Section 5 \"Empirical Results\" is informative, but in general hard to access. To start with, the first two tasks are described in bulks of text, and I believe any visualization would be valuable for intuition (esp. task 2). Tables lack bolded parts for readability, and I'd appreciate if they can also label what models are trained and what models are unsupervised (though it's in text).\n\n**Questions**:\n- How to handle multi-object tracking when the observation encoding is just one piece, but not object-factorized? 3 objects are still okay but I guess more objects would be intractable?\n- Task 1 and 3 are not really problems with a temporal line. So why is this online approach *supposed* to be better than the batch approach? I know the empirical result is better but I still want to understand it conceptually. \n- I'm not sure if baselines are strong or designed for these tasks, especially task 3 seems a bit ad hoc. It might be more interesting to render MNIST and airplane into tracking tasks if the aim is to show ability to handle high-dim inputs.\n\n**Originality**: I'm not familiar with related work but the method looks original, possibly as the abstract problem formulation is novel.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NEvn8hLfRCp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to train a deep network to perform dynamic data association and static online clustering. The proposed architecture relies on a recurrent structure while also adding knowledge about the data association problem into the architecture design. \n\nThe majority of the paper is well written with an easy to understand and follow motivation and a concise yet complete review of related work. The problem formulation and description of the network are well written and provide a good intuition for the choices made. While some of the choices made in the architecture design are motivated this is not done for all aspects, and it would have been nice to see this as it could provide greater insight into what works and what does not.\n\nThe experimental section of the paper is probably the weakest part. This is partly due to the writing getting less clear in certain aspects and some choices made not being clear. Despite this, the methods used in the comparison seem adequate to demonstrate the ability of the proposed method.\n\nThe paper states that in the Gaussian domains, the proposed method's performance is comparable to that of the non-online methods such as k-means. Looking at the numbers, there is a difference of 0.5 at times (Table 1). Given the scales of the values, this would appear to be a substantial difference. As these results effectively evaluate clustering performance, the chosen metric seems a bit odd. Usually, one would expect normalized mutual information or similar clustering metrics to be used. Overall it just is not clear what the magnitudes reported represent.\n\nAnother aspect that always remained somewhat mysterious was the way the number of observations was used. Is it that the different algorithms were given N observations to process at once or N observations in sequence. In either case, how is the number reported obtain, is it the correctness of every single point in the sequence? While the overall aspect that the different experiments attempt to capture was understandable, the finer details never become entirely clear, which is a shame given the clarity of the rest of the paper.\n\nThe paper concludes that the proposed method scales well to an increased number of hypothesis slots and a large number of underlying clusters. I do not see where this conclusion is supported by the experiments if anything the opposite is shown. Table 3 shows what appears to be a relation between the number of slots and true clusters for performance. Though in any case, the performance degrades with an increase of true clusters present. This leads to another question; the experiments seem to focus on a low number of true clusters and a low number of slots. In the robotic scenarios outlined at the beginning of the paper, one would expect there to be hundreds of objects that need to be tracked. This leads to several questions. How does the proposed method handle cases where the number of slots is 100 or more and the number of clusters is similarly high. Furthermore, is there a reason not to use a model with an overly large number of slots, say 1000, to be sure there is sufficient capacity for complex environments?\n\nThe image domain based experiments were initially quite confusing, mainly because these datasets are used in a non-traditional setup, i.e. clustering or classification. After reading the explanation carefully twice, I could understand what is being done, though initially, I was quite confused. \n\n=== Post rebuttal ===\n\nThe clarification and additional experiments are greatly appreciated and answered some of the points which were not entirely clear to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ir6aFOqvRBC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary: \nThis paper proposes a deep neural network for online data association. Specifically, the proposed network includes an encoder, attention module, transitional model, and an output decoder, which can associate each observation with the existing hypothesis. The proposed method is evaluated on a few toy datasets and two small-scale image datasets.\n\nPaper Strengths:\n1. The technical explanation is very clear to me and seems sound\n2. The experiments are extensive, where the proposed method has been evaluated on three toy datasets and two image domain datasets\n\nPaper Weaknesses:\n1. Overall, I feel the presentation of the paper’s contributions is not very clear. What is the core novelty of this paper, conceptually or technically? For example, what are the differences of the proposed method versus deep Kalman filter, deep state-space model (SSM) [A1]? There should be a detailed discussion about the differences with prior work in related work, which however I could not find. In the last paragraph of the related work, the paper ends with “We view our work as an instance of this approach”, but provides no comparison with other instances of this type of approach which are the closest to the proposed method.\n2. Although I appreciate that this paper has provided extensive experiments, the data used in the experiments seem to be generated randomly (e.g., one thousand normal distribution), which is hard to be reproduced and compared by the follow-up work. Although I am not familiar with the clustering literature, I believe there are public benchmark datasets where I would suggest to evaluate the proposed method, in order to increase the reproducibility and fairness of the comparison. Also, the image-based datasets (MNIST, Airplane) used for evaluation are too simple. It would be nice to see the proposed method is evaluated on real-world datasets for online data association such as MOT Challenges, which makes the results more convincing.\n3. In Table 1, the proposed method seems to perform worse than classical methods K-means++ and GMM, it would be nice to see some analysis to explain why this is the case. Also, it would be nice to add a column for the mean of the results over 5 subsets in Table 2 for an overall understanding of the performance\n4. In the abstract, the paper criticizes prior work is computationally expensive, which is not really convincing. Classical methods like Kalman filter can run very fast without GPU, while the proposed method is a neural network-based approach, which I guess is slower than classical Kalman filter, though there is no inference speed analysis in the main paper. \n5. In the last paragraph of the introduction, this paper mentions that it is possible to solve the target problem by training a standard RNN, but without discussing what are the disadvantages of this RNN-based approach. As a result, there is no justification and motivation for the proposed approach \n6. In the related work, this paper mentions that there is little work in the area of learning for data association, which is not true. For example, in the domain of the visual multi-object tracking domain, there are approaches designed for deep or end-to-end data association such as [A2-A5]. There should be a discussion with such approaches.\n\nJustification:\nMy decision is made mainly because I feel there lacks a detailed discussion about the differences/novelty over prior work and also I am not fully convinced by the experimental results especially the data that is used for evaluation\n\nReferences\n\n[A1] Hafner et al. Learning latent dynamics for planning from pixels. ICML 2019\n\n[A2] Xu et al. How to Train Your Deep Multi-Object Tracker. CVPR 2020\n\n[A3] Ma et al. Deep Association: End-to-End Graph-Based Learning for Multiple Object Tracking with Conv-Graph Neural Network. ICMR 2019\n\n[A4] Sun et al. Deep Affinity Network for Multiple Object Tracking. TPAMI 2019.\n\n[A5] D. Frossard and R. Urtasun. End-to-End Learning of Multi-Sensor 3D Tracking by Detection. ICRA 2018.\n\n[A6] Dendorfer et al. MOTChallenge: A Benchmark for Single-camera Multiple Target Tracking. IJCV 2020", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was on the borderline. While there was some support for the ideas presented, concerns were raised about the experiments. The exposition would also need to better demonstrate the significance of the contribution.
Decision: Reject