Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method to train a deep network to perform dynamic data association and static online clustering. The proposed architecture relies on a recurrent structure while also adding knowledge about the data association problem into the architecture design. 

The majority of the paper is well written with an easy to understand and follow motivation and a concise yet complete review of related work. The problem formulation and description of the network are well written and provide a good intuition for the choices made. While some of the choices made in the architecture design are motivated this is not done for all aspects, and it would have been nice to see this as it could provide greater insight into what works and what does not.

The experimental section of the paper is probably the weakest part. This is partly due to the writing getting less clear in certain aspects and some choices made not being clear. Despite this, the methods used in the comparison seem adequate to demonstrate the ability of the proposed method.

The paper states that in the Gaussian domains, the proposed method's performance is comparable to that of the non-online methods such as k-means. Looking at the numbers, there is a difference of 0.5 at times (Table 1). Given the scales of the values, this would appear to be a substantial difference. As these results effectively evaluate clustering performance, the chosen metric seems a bit odd. Usually, one would expect normalized mutual information or similar clustering metrics to be used. Overall it just is not clear what the magnitudes reported represent.

Another aspect that always remained somewhat mysterious was the way the number of observations was used. Is it that the different algorithms were given N observations to process at once or N observations in sequence. In either case, how is the number reported obtain, is it the correctness of every single point in the sequence? While the overall aspect that the different experiments attempt to capture was understandable, the finer details never become entirely clear, which is a shame given the clarity of the rest of the paper.

The paper concludes that the proposed method scales well to an increased number of hypothesis slots and a large number of underlying clusters. I do not see where this conclusion is supported by the experiments if anything the opposite is shown. Table 3 shows what appears to be a relation between the number of slots and true clusters for performance. Though in any case, the performance degrades with an increase of true clusters present. This leads to another question; the experiments seem to focus on a low number of true clusters and a low number of slots. In the robotic scenarios outlined at the beginning of the paper, one would expect there to be hundreds of objects that need to be tracked. This leads to several questions. How does the proposed method handle cases where the number of slots is 100 or more and the number of clusters is similarly high. Furthermore, is there a reason not to use a model with an overly large number of slots, say 1000, to be sure there is sufficient capacity for complex environments?

The image domain based experiments were initially quite confusing, mainly because these datasets are used in a non-traditional setup, i.e. clustering or classification. After reading the explanation carefully twice, I could understand what is being done, though initially, I was quite confused. 

=== Post rebuttal ===

The clarification and additional experiments are greatly appreciated and answered some of the points which were not entirely clear to me.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work aims to solve the data association problem in a general setting, which is suitable for various scenarios such as online clustering and online tracking. The motivation of having this setting is that temporal dense observations are not always available. 

The high level idea of this work is to learn a neural network which updates the hypothesis ($y$) and outputs the association ($c$). I like the design that are well motivated by the problem itself and the ability of neural networks: 1) recursive filter for online data; 2) order invariant memory structure; 3) representation learning.

The experiments are conducted on many domains and the proposed approach is compared with many methods from different domains. Ablation study is also conducted to demonstrate the effectiveness of the design decisions. It seems thorough. However, I think the experiments are more like toy experiments and it would be great if it could demonstrate its superiority on a more practical dataset such as a multiple object tracking (MOT) benchmark. In my opinion, data association is the key for MOT. But definitely other problems with a practical setting can also convince me. Another concern about the experiment is that the loss is used as the metric, which seems weired to me. The clustering metrics like F-measure or Rand index and/or precision and recall might be better.


Other suggestions and clarifications:

1. I am very interested in the sparsity loss. Is there any references or justifications which shows that it will leads to a sparse result. 

2. Is it possible to use a classification (cross entropy) loss for $c$? Will the order invariant property cause any issue here?

3. An additional question about $c$: how is it normalized? 

4. Why "the basic clustering methods have no ability to handle dynamic systems"? I assume they don't have the dynamic part, but as long as they can observe the object `"feature", they are applicable. Though I guess the performance won't be good.

5. In the image-based domains, is it generally better than the classification methods? 

Missing reference:
Guillem Braso and Laura Leal-Taixe. Learning a Neural Solver for Multiple Object Tracking. In CVPR 2020. This work aims to solve the data association problem for Multiple Object Tracking based on Message Passing Networks. 

=== Post rebuttal ===
1. As reviewer 2, I am also not convinced by the reason why it could not be evaluated on a MOT dataset. The explanation suggests that the proposed method is more general and it should be able to apply it to broader domains (and thus MOT could be one of them). 
2. The evaluation metric still looks suspicious to me. I can imagine we can usually get better numbers if we can directly minimize the metrics we want to evaluate. However, in my opinion, data assocision usually need to do hard assginments if we want to use it in practice (unless it is a intermediate task and in that case we can do soft assignments), so I do think it is better to set a hard threshold or draw a curve of F1 v.s. threhold. In the meantime,  I think it is unfair for some other methods, like k-means++ becaue it is minimizing a different objective.

I still think the method has some merits so I am at borderline, though I will not defend it if it is rejected.
review_title: Good method but the experiment settings are not practical

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes DAF-Net, an attention-based architecture for online filtering, which is used for a flexible set of tasks: online clustering, tracking, and image association.

**Strength**: I like the idea of using attention for updating temporal beliefs, which is natural and potentially a backbone for a wide range of temporally extended tasks. The architecture looks reasonable and handles observation encoding, dynamics updating, and online association altogether, and the presented numbers look good across three experiments.

**Weakness**: See **Clarity** and **Questions**.

**Clarity**: I find the paper not fully accessible at the first sight.
- Section 3 "Problem Formulation" is quite abstract and unintuitive, and some motivation before description might be good. I'm not sure why the loss function is part of problem formulation rather than the method.
- Section 4 "DAF-Nets" presents some parts I do not understand, e.g. the format of $n$ vector, and the sentence "The $a$ vectors are integrated to obtain $n$, which is normalized to obtain the final output confidence values $c$". The use of $s_k$ and $s_{tk}$ are inconsistent. 
- Figure 1 looks complicated and the meaning of colors of edges are not annotated.
- Section 5 "Empirical Results" is informative, but in general hard to access. To start with, the first two tasks are described in bulks of text, and I believe any visualization would be valuable for intuition (esp. task 2). Tables lack bolded parts for readability, and I'd appreciate if they can also label what models are trained and what models are unsupervised (though it's in text).

**Questions**:
- How to handle multi-object tracking when the observation encoding is just one piece, but not object-factorized? 3 objects are still okay but I guess more objects would be intractable?
- Task 1 and 3 are not really problems with a temporal line. So why is this online approach *supposed* to be better than the batch approach? I know the empirical result is better but I still want to understand it conceptually. 
- I'm not sure if baselines are strong or designed for these tasks, especially task 3 seems a bit ad hoc. It might be more interesting to render MNIST and airplane into tracking tasks if the aim is to show ability to handle high-dim inputs.

**Originality**: I'm not familiar with related work but the method looks original, possibly as the abstract problem formulation is novel.


review_title: potentially important direction

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper Summary: 
This paper proposes a deep neural network for online data association. Specifically, the proposed network includes an encoder, attention module, transitional model, and an output decoder, which can associate each observation with the existing hypothesis. The proposed method is evaluated on a few toy datasets and two small-scale image datasets.

Paper Strengths:
1. The technical explanation is very clear to me and seems sound
2. The experiments are extensive, where the proposed method has been evaluated on three toy datasets and two image domain datasets

Paper Weaknesses:
1. Overall, I feel the presentation of the paper’s contributions is not very clear. What is the core novelty of this paper, conceptually or technically? For example, what are the differences of the proposed method versus deep Kalman filter, deep state-space model (SSM) [A1]? There should be a detailed discussion about the differences with prior work in related work, which however I could not find. In the last paragraph of the related work, the paper ends with “We view our work as an instance of this approach”, but provides no comparison with other instances of this type of approach which are the closest to the proposed method.
2. Although I appreciate that this paper has provided extensive experiments, the data used in the experiments seem to be generated randomly (e.g., one thousand normal distribution), which is hard to be reproduced and compared by the follow-up work. Although I am not familiar with the clustering literature, I believe there are public benchmark datasets where I would suggest to evaluate the proposed method, in order to increase the reproducibility and fairness of the comparison. Also, the image-based datasets (MNIST, Airplane) used for evaluation are too simple. It would be nice to see the proposed method is evaluated on real-world datasets for online data association such as MOT Challenges, which makes the results more convincing.
3. In Table 1, the proposed method seems to perform worse than classical methods K-means++ and GMM, it would be nice to see some analysis to explain why this is the case. Also, it would be nice to add a column for the mean of the results over 5 subsets in Table 2 for an overall understanding of the performance
4. In the abstract, the paper criticizes prior work is computationally expensive, which is not really convincing. Classical methods like Kalman filter can run very fast without GPU, while the proposed method is a neural network-based approach, which I guess is slower than classical Kalman filter, though there is no inference speed analysis in the main paper. 
5. In the last paragraph of the introduction, this paper mentions that it is possible to solve the target problem by training a standard RNN, but without discussing what are the disadvantages of this RNN-based approach. As a result, there is no justification and motivation for the proposed approach 
6. In the related work, this paper mentions that there is little work in the area of learning for data association, which is not true. For example, in the domain of the visual multi-object tracking domain, there are approaches designed for deep or end-to-end data association such as [A2-A5]. There should be a discussion with such approaches.

Justification:
My decision is made mainly because I feel there lacks a detailed discussion about the differences/novelty over prior work and also I am not fully convinced by the experimental results especially the data that is used for evaluation

References

[A1] Hafner et al. Learning latent dynamics for planning from pixels. ICML 2019

[A2] Xu et al. How to Train Your Deep Multi-Object Tracker. CVPR 2020

[A3] Ma et al. Deep Association: End-to-End Graph-Based Learning for Multiple Object Tracking with Conv-Graph Neural Network. ICMR 2019

[A4] Sun et al. Deep Affinity Network for Multiple Object Tracking. TPAMI 2019.

[A5] D. Frossard and R. Urtasun. End-to-End Learning of Multi-Sensor 3D Tracking by Detection. ICRA 2018.

[A6] Dendorfer et al. MOTChallenge: A Benchmark for Single-camera Multiple Target Tracking. IJCV 2020
review_title: Novelty is unclear and experiments are not fully convincing

Meta-review: This paper was on the borderline. While there was some support for the ideas presented, concerns were raised about the experiments. The exposition would also need to better demonstrate the significance of the contribution.
Decision: Reject