Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper deals with the problem of multiple object tracking and trajectory prediction in multiple frames of videos. The main focus is adding a relation-reasoning building block to the original HART framework. With multiple objects, the key is to be able to learn the permutation invariant representation during potential changing and dynamic object trajectories. The paper also uses toy examples to show that the proposed block of relation reasoning is not necessarily beneficial when the object trajectory is less random and more static. Finally, experiments on real data demonstrate that the proposed method that accounts for relation reasoning is helpful by a limited magnitude.
 
The main contribution of this paper is the novel relation reasoning block. However, there are three key concerns of mine:
 
1. It is not clear when the new architecture would be helpful. I am really happy to see a more careful study of the success and failure cases of the method. I also appreciate the honesty that the current model is not competitive with the ones that have an accurate bounding box as input. But I think a more detailed study can be conducted, especially in the toy example case.  Maybe it is possible to design different levels of randomness in the trajectory and further figure out when is the reasoning block helpful.

2. Even the real data experiments do not have very impressive results, from my biased observation. (I am not in the field, so maybe I am wrong here.) 

3. The end-to-end approach is also another perspective where the authors try to differentiate their methods from others. I think it is potentially an interesting problem that why end-to-end is something you would prefer (which is definitely a harder problem), and when. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper presents a class-agnostic method for tracking multiple moving objects (MOHART) that extends an existing single-object tracking method (Hierarchical Attentive Recurrent Tracking, HART). Similarly to HART, MOHART utilizes an attention mechanism and LSTM units. The extension form HART to MOHART is done in two main steps: HART is applied to multiple objects in parallel, with a presence variable attached to each, and a permutation-invariant network that learns the interactions between the objects.

The method is tested in two groups of experiments: synthetic data of moving circles, and on several real datasets. The results on the moving circles are enlightening. The paper shows that although MOHART outperforms HART in all circumstances, if the forces between the circles change randomly every number of steps, MOHART outperforms HART by a very large margin. This is because the forces between the circles are determined by their color, which is observable by the network, and MOHART, which capable of learning their interaction, learns the pattern of changing forces. HART, being an independent-particle model, cannot learn the interactions.

The results on the real datasets demonstrate superiority over HART, and in a series of ablation experiments, the Authors are able to show that MOHART's ability to utilize interactions between objects is responsible for the improvement. For example, the improvement over HART is greater when egomotion is significant, because egomotion creates correlations between the movement of objects in the camera frame, which MOHART can utilize.

I found the paper enlightening, based on a neat idea. The experimental results are extensively analyzed and the Author's insights about their algorithm are substantiated by the experiments. 

However the difficulty I had with the paper is that the results are only compared to HART. HART was published in 2017, and the field of scene understanding and tracking had been addressed in multiple papers since then. It is one of the more competitive and application-driven fields in computer vision. Comparing MOHART only to its sister-method from over two years ago significantly limits the usefulness of paper IMHO. The expectation IMHO is not that MOHART outperform each and every method out there, but that the reader know where MOHART stands compared to other methods (e. g. tracking by detection).

I will refrain from pointing to specific papers to compare to, because I believe the Authors should chose the settings, datasets, criteria, and metrics that are the most convenient for them for comparison. The papers surveyed in Section 2, especially under "Tracking-by-Detection", have many followups. Recent papers from those that claim state of the art could good candidates for comparison.

In view of the above, I am inclined at this stage to reject the paper.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: I'm somewhat out of area for this review: I study relational models, but have little experience with computer vision in general and object tracking in particular. 

--------------------

This paper presents an extension to HART [Kosiorek 2017] to track multiple objects. It builds on a simple baseline: run multiple HART models in parallel, each tracking one of the objects. They extend this by adding a relational reasoning module to allow interaction between the parallel models. The relational reasoning module uses Lee et al. (2019)'s self-attention block (similar to Vaswani et al. 2017). They find that the simple baseline is surprisingly effective, but that MOHART (their model) improves performance in environments with stochastic interactions and crowded settings which tend to be more noisy.

The novelty is somewhat limited: they're plugging together two existing approaches (HART and SAB) to allow for interaction, but I am recommending acceptance because I found the experiments surprising (as a negative result) - they show that independent HART models are a strong baseline for tracking in settings that involve interactions, but that are nevertheless solvable without knowledge about the state of other objects. For the synthetic experiments, I would have expected the fact there are repulsive forces between the objects would have been sufficient for the relational module to be helpful. The paper also provides further evidence of the usefulness of non-local networks for these sorts of problems.
review_title: Official Blind Review #1

Meta-review: The authors propose an end-to-end object tracker by exploiting the attention mechanism. Two reviewers recommend rejection, while the last reviewer is more positive. The concerns brought up are novelty (last reviewer), and experiments (second reviewer). Furthermore, the authors seem to overclaim their contribution. There indeed are end-to-end multi-object trackers, see Frossard & Urtasun's work for example. This work needs to be cited, and possibly a comparison is needed. Since the paper did not receive favourable reviews and there are additional citations missing, this paper cannot be accepted in current form. The authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject