Improving End-to-End Object Tracking Using Relational Reasoning
Authors: Fabian B. Fuchs
Adam R. Kosiorek
Li Sun
Oiwi Parker Jones
Ingmar Posner
Venue:  (2020)
Keywords: Relational Reasoning
Tracking
Intuitive Physics
Real-World Application
Permutation Invariance

Abstract: Relational reasoning, the ability to model interactions and relations between objects, is valuable for robust multi-object tracking and pivotal for trajectory prediction. In this paper, we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning. We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment. We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs. To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature. 

Under review conference paper at ICLR 2020
IMPROVING END-TO-END OBJECT TRACKING USING RELATIONAL REASONING
Anonymous authors Paper under double-blind review
ABSTRACT
Relational reasoning-the ability model interactions and relations between objects- is valuable for robust multi-object tracking and pivotal for trajectory prediction. In this paper we propose MOHART, a class-agnostic, end-to-end multi-object tracking and tra- jectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning. We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better ac- challenging toy experiment. We show on
counts for complex physical interactions relational reasoning capabilities in this way three real-world tracking datasets that adding increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs. To the best of our knowledge, MOHART the first fully end-to-end multi-object tracking from vision ap- proach applied to real-world data reported in the literature.
INTRODUCTION
Real-world environments can be rich and contain countless types of interacting objects. Intelligent au- tonomous agents need to understand both the objects and interactions between them if they are to operate in those environments. This motivates the need for class-agnostic algorithms for tracking multiple objects- backing-by-detectte
capability that is not supported by the popular tracking-by-dee -y-deetttnn paradigm. objects are detected in each frame independently. g.. by a pre-trained deep convolutional neural network (CNN) such as YOLO (Redmon et al. (2016), and then linked across frames. Algorithms from this family
selationel Reasening
Extraction
Module
FIGURE 1: Multi-object hierarchical attentive recurrent tracking (MOHART), A glimpse is extracted for each object using (fully differentiable) spatial attention mechanism. These glimpses are further processed with a CNN and fed into relational reasoning module. A recurrent module which iterates over time steps allows for capturing of complex motion patterns. It also outputs spatial attention parameters and a feature vector per object for the relational reasoning module. Dashed lines indicate temporal connections (from time step t to t + 1). The entire pipeline operates in parallel for the different objects, only the relational reasoning module allows for exchange information between tracking states each object. MOHART is an extension of HART (a single-object tracker), which features the same pipeline without the relational reasoning module.

---

Under review conference paper at ICLR 2020
can achieve high accuracy, provided sufficient labelled data train the object detector, and given that all encountered objects can be associated with known classes, but fail when faced with objects from previously unseen categories.
Hierarchical attentive recurrent tracking (HART) is recently-pree proposed, alternative method for single-object tracking (SOT), which can track arbitrary objects indicated by the user (Kosiorek et al. (017)). This done by providing an initial bounding-box, which may be placed over any part of the image, regardless of whether contains an object or what class the object is. HART efficiently processes just the relevant part of an image using spatial attention; it also integrates object detection, feature extraction, and motion modelling into one network, which is trained fully end-to-end. Contrary to frame is typically processed at any given time generate bounding box proposals, end-to-end learning in HART allows for discovering complex visual and spatio- patterns in videos, which is conducive to inferring what an object is and how moves.
where only one video
In the original formulation, HART is limited to the single-object modality- as are other existing end-to-end trackers (Kahou et al. (2017); Rasouli Danesh al. (2019); Gordon et al. (2018)). In this work, we present MOHART, class-agnostic tracker with complex relational reasoning capabilities provided by multi-header self-attention module (Vaswani et al.] (2017); Lee et al. 2019)). MOHART infers the latent state of every tracked object in parallel, and uses sell-attention to inform per-object states about other tracked objects. This helps to avoid performance loss under self-occlusions of tracked objects or strong camera motion. Moreover, since the model is trained end-to-end, it is able to learn how to manage faulty or missing sensor inputs. See fig. for a high-level illustration of MOHART.
In order to track objects, MOHART estimates their states, which can be naturally used to predict future tra- jectories over short temporal horizons, which is especially useful for planning in the context of autonomous agents. MOHART can be trained simultancously for object tracking and trajectory prediction at the same time, thereby increasing statistical efficiency of learning. In contrast to prior art, where trajectory predic- tion and object tracking are usually addressed as separate problems with unrelated solutions, our work show trajectory prediction and object tracking are best addressed jointly.
Section 2 describes prior art in tracking- by -eteecti,,,, end-to and-to-end tracking and predestrian trajectory predic- tion. In Section 3, we describe our approach, which uses a permutation-innariant self-attention module to enable tracking multiple objects end-to-end with relational reasoning. Section|Jcontrann our approach with multi-object trackers which do not explicitly enforce permutation invariance but have the capacity to learn it, simpler permutation-invariin architectures, as well as multiple single-object trackers running in parallel. We show that multi-headed self-attention significantly outperforms other approaches. Finally, in Section [ we apply MOHART to real world datasets and show that permutation crmmutaaionnnnvariannt invariant relational reasoning leads to consistent performance improvement compared to HART both in tracking and trajectory prediction.
2 RELATED WORK
Tracking-by-Detecction Vision- based tracking approaches typically follow a bucking-by-detectin paradigm: objects are first detected in each frame independently, and then a tracking algorithm links the detections from different frames to propose coherent trajectory (Zhang et al. (2008); Milan et al. (2014); Bae and Yoon (2077): Keuper et al. (20T8)). Motion models and appearance are often used to improve the association between detected bounding- bounding-boxes in postprocessing step. Tracking-by-dee by-dttccionnn algorithms currently provide the state-of-the-art in multi-object tracking on common benchmark suites, and we fully acknowledge that MOHART is not competitive at this stage in scenarios where high-quality detections are available for each frame. MOHART can in principle be equipped with the ability to use bounding boxes provided by an object detector, but this is beyond the scope of this project
2

---

Under review conference paper at ICLR 2020
End-to-End Tracking A newly established and much less explored stream of work approaches tracking in an end-to-end fashion. A key difficulty here is that extracting an image crop (according to bounding-boxe -boeess provided by detector), is non-differentiable and results in sigh-variance gradient estimators. [Kahou al. (2017) propose an end-to-end tracker with soft spatial -patial-attention using a 2D grid of Gaussians instead of a hard bounding-box. HART draws inspiration from this idea, employs an additional attention mechanism, and shows promising performance on the real-world KITTI dataset (Kosiorek et al. (2017)) HART forms the foundation of this work. It has also been extended to incorporate depth information from RGBD cameras (Rasouli Danesh et al. (2019)). Gordon et al. (2018) propose an approach in which the crop corresponds to the scaled up previous bounding-box. This simplifies the approach, but does not allow the model to learn where to look e.. no gradient is backpropagated through crop coordinates. To the best of our knowledge, there are no successful implementations of any such end-to-end approaches for multi-object tracking beyond SQAIR (Kosiorek al. (2018)), which works only on datasets with static backgrounds. On real-world data, the only end-to-end approaches correspond to applying multiple single-object trackers in parallel-a method which does not leverage the potential of scene context or inter-object interactions.
Predicting pedestrian trajectories has long history computer vision and robotics. Initial research modelled social forces using hand-crafted features (Lerner et al. (2007); PPelle grini et al. (2009); Trautman and Krause (2010): Yamaguchi et al. (2011)) or MDP-based motion transition models (Rudenko et al. (2018)), while more recent approaches learn from context information, c. g.. posi- tions of other pedestrians or Tandmarks in the environment. Social-T.STM (Alahi et al. (2016)) employs a long short-term memory (LSTM) predict pedestrian trajectories and uses max-pooling to model global social context. Attention mechanisms have been employed to query the most relevant information, such as neighbouring pedestrians, in a learnable fashion (Su al. (2016); Fernando et al. (2018) Sadeghian al. (2019). Apart from relational learning, context (Varshneya and Srinivasaraghaaan information (Sun et al. (2018)), and constant motion priors (Schöller et al. (2019) have proven effective in predicting long-term trajectories.
Pedestrian trajectory prediction
2017 (). periodical time
Our work stands apart from this prior art by not relying on ground truth tracklets. addresses the more challenging task of working directly with visual input, performing tracking, modelling interactions, and, depending on the application scenario, simultaneously predicting future motions. As such, it can also be compared to Visual Interaction Networks (VIN) (Watters et al. (2017)), which use CNN to encode three consecutive frames into state vectors-one per object- and feed these into a recurrent neural network (RNN), which has an Interaction Network (Battaglia Mittaggla al. (2016)) at its core. More recently, Relational Neural Expectation Maximization (R-NEM) has been proposed as an unsupervised approach which combines scene segmentation and relational reasoning (van Steenkiste et al. )2018). Both VINS and R-NEM make accurate predictions in physical scenarios, but, to the best of our knowledge, have not been applied to real world data.
3 RECURRENT MULTI-OBJECT TRACKING WITH SELF-ATTENTION
This section describes the model architecture in fig. T We start by describing the hierarchical attentive recurrent tracking (HART) algorithm (Kosiorek et al. (017)). and then follow with an extension of HART to tracking multiple objects, where multiple instances of HART communicate with each other using multi- headed attention to facilitate relational reasoning. We also explain how this method can be extended to trajectory prediction instead of just tracking.
3.1 HIERARCHICCA ATTENTIVE RECURRENT TRACKING (IART)
HART is an attention- recurrent algorithm, which can efficiently track single objects in a video. It uses a spatial attention mechanism extract a glimpse gr, which corresponds at time-step f, containing the object of interest. This allows it to dispense with the processing of the whole image and can significantly decrease the amount of computation required. HART uses CNN to convert the
small crop of the image x,
3

---

Under review conference paper at ICLR 2020
do for each object in:
Concat
Recurrent Module
[obj
obj
abj
Scalar Product
Softmax
CNN
Feature
Concat
Linear
weighted
Extraction
Sum
Sum
Positional
Linear
Encoding
FIGURE 2: The relational reasoning module in MOHART based multi-headed self-attention. Here, we show the computation of the interaction of the red object with all other objects. Object representations ft. are computed using visual features, positional encoding and the hidden state from the recurrent module. These are linearly projected onto keys (k), queries (q), and values (v) to compute weighted sum inter- actions between objects, yielding interaction vector mt,m- Subscripts t, dropped from all variables for clarity of presentation, so is the splitting into multiple heads.
glimpse into features f,, which then update the hidden state h, of a LSTM core. The hidden state is used estimate the current bounding-box b; spatial attention parameters for the next time-step ar+1 as well object appearance. Importantly, the recurrent core can learn to predict complicated motion conditioned on the past history of the tracked object, which leads relatively small attention glimpses- conttaary to CNN-based approaches (Held et al. (2016): Valmadre et al. (2017)), HART does not need to analyse large regions-of-interes. to search for tracked objects. the original paper, HART processes the glimpse with an additional ventral and dorsal stream top of the feature extractor. Early experiments have shown that this does not improve performance on the MOTChallenge dataset, presumably due to the oftentimes small objects and overall small amount of training data. Further details provided in Appendix B
The algorithm is initialised with a bounding-bor/ bb for the first time-step, and operates on a sequence raw images X1:7. For time-steps 2 2, it recursively outputs bounding-bue -box estimates for the current time-step and predicted attention parameters for the next time-step. The performance is measured as intersection-over. union (IoU) averaged over all time steps in which an object present, excluding the first time step.
Although HART can track arbitrary objects, it is limited to tracking one object at a time. While it can be deployed several objects in parallel, different HART instances have results in performance loss, as it is more difficult to identify occlusions, ego-motion and object interactions. Below, we propose an extension of HART which remedies these shortcomings.
of communication. This
3.2 MUJETI-OBJECT HIERARCHICAL ATTENTIVE RECURRENT TRACKING (MOHART)
Multi-object support HART requires the following modifications. Firstly, in order to handle a dynamically changing number of objects, we apply HART to multiple objects in parallel, where all parameters between HART instances are shared. We refer to each HART instance as a tracker. Secondly, we introduce a presence variable Pt.m for object It is used to mark whether object should interact with other objects, well as to mask the loss function (described in (Kosiorek et al. (201))) )) for the given object when it not present. In this setup, parallel trackers cannot exchange information and conceptually still single-object trackers, which we use as a baseline, referred to as HART (despite it being an extension of the original algorithm). Finally, enable communication between trackers, we augment HART with additional step between feature extraction and the LSTM.
We either a ground-truie truth bounding box provided by external detector; the only requirement is that it contains the object of interest.
4

---

Under review conference paper at ICLR 2020
For each object, glimpse is extracted and processed by a CNN (see fig. D. Furthermore, spatial attention parameters are linearly projected on a vector of the same size and added to this representation, acting as a positional encoding. This then concatenated with the hidden state of the recurrent module of the respective object (see fig. 2). Let f. denote the resulting feature vector corresponding to the m object, and let fr, I:M be the set of such features for all objects. Since different objects can interact with each other, it is necessary to use a method that can inform each object about the effects of their interactions with other objects. Moreover, since features extracted from different objects comprise a set, this method should be permutation--cquiaaiiii i. i..., the results should not depend on the order in which object features are processed. Therefore, we use the multi-head self-attention block (SAB, (Le et al, (201)), which is able to account for higher-order interactions between set elements when computing their representations. Intuitively, in our case, SAB allows any of the trackers to query other trackers about attributes of their respective objects, e. g., distance between objects, their direction of movement, or their relation to the camera. This is implemented as follows,
Q = W fl.M + bg,
K = Waff..M + bb
V = W.fii.M + by,
(1)
O = softmax (Q:k,) Vi, = H
(2)
01:M = O = concat .o...t(O........ ,OO).
(3)
where Om is the output of the relational reasoning module for object m. Time-step subscripts are dropped
to decrease clutter. In Eq. [ each of the extracted features fe is linearly projected into a triplet of key mm query Qr.m and value Vi,m vectors. Together, they comprise K, Q and V matrices with M rows and dg. dg, dy columns, respectively. K, Q and V are then split up into multiple heads H E N.,, which allows to query different attributes by comparing and aggregating different projection of features. Multiplying Q,K?' in Eq. [2 alllow to compare every query vector qt,m, to all key vectors I;,,;M,. where the value of the cor- responding dot-prou not-products represents the degree of similarity. Similarities are then normalised via a softmax operation and used to aggregate values V. Finally, outputs of different attention heads concatenated in Eq. 3 S AB produces M output vectors, one for each input, which are then concatenated with corresponding inputs and fed into separate I.STMS for further processing, as in HART-sec fig-
MOHART is trained fully end-to-end, contrary to other tracking approaches. It maintains a hidden state, which can contain information about the object's motion. One benefit is that in order to predict future trajectories, one can simply feed black frames into the model. Our experiments show that the model learns fall back on the motion model captured by the LSTM in this case.
3.3 MULTI-OBJECT BASELINES
Multilayer perceptron (MLP) In this version, the representations of all objects concatenated and fed into a fully connected layer followed by ELU activations. The output is then again concatenated to the unaltered feature vector of each object. This concatenated version is then fed to the recurrent module of HART. This way of exchanging information allows for universal function approximation (in the limit of infinite layer sizes) but does not impose permutation invariance.
DeepSets Here, the learned representations of the differentt objects summed up instead of concatenated and then divided by total number of objects. This is closely related to DeepSets (Zaheer et al. (2017 7) ) and
allows for universal function approximation of all permutation invariant functions (Wagstaff et al. (2019))
Max-Pooling Similar to DeepSets, but using max-pooling as the permutation invariant operation. This way of exchanging information is used, e.g., by Alaai et al. (2016) who predict future pedestrian trajectories from ground truth tracklets in coordinate space.
4 VALIDATION ON SIMULATED DATA
We test and compare the relational reasoning capabilities of the proposed algorithms on a toy domain. The domain is 2D squared box which contains circular objects with approximated elastic collisions (energy
5

---

Under review conference paper at ICLR 2020
HART
0.44
0.75
0.66
0.64
6.60
MOHART
0.08
0.19
0.22
0.23
0.226
8-00
FIGURE scenario constructed be impossible solve without relational reasoning. Circles of the same colour repel each other, circles of different colour attract each other. Crucially, each circle is randomly assigned its identity in each time step. Hence, the algorithm can not infer the forces exerted on one object without knowledge of the state of the other objects in the current time step. The forces in this scenario scale with 1/vr and algorithm was trained to predict one time step into the future. HART (top) indeed unable to predict the future location of the objects accurately. The achieved average ToU is 47%, which is only slightly higher than predicting the objects to have the same position in the next time step in the current meaningful predictions (76% IoU). The numbers in the bottom row indicate the self-attention weights from the perspective of the top left tracker (yellow number box). Interestingly, the attention scores have strong correlation with the interaction strength (which scales with distance) without receiving supervision.
(34%). Using the relational reasoning module, MOHART (bottom) is able to make
and momentum conservation) between objects and with walls. To investigate how the model understands motion patterns and interactions between objects, train it to predict future object locations in contrast to traditional tracking.
In the first experiment, each circle exerts repulsive forces each other, where the force scales with 1/r, being the distance between them. Predicting the future location just using the previous motion of one object (i.e. without relational reasoning) accurately is therefore challenging. We show that HART end single-object tracker is able to capture complex motion patterns and leverage these to make accurate predictions (see Appendix C). This indicates that HART is able draw conclusions about the (deterministic, but not static) force field.
end-to-
the second experiment, introduce randomness, rendering the scenario not solvable for single object tracker as it requires knowledge about the state of the other objects and relational reasoning (see fig. 3 each time step, we assign colour-coded identity to the objects. Objects of the same identity repel each other, object of different identities attract each other (the objects can be thought of as electrons and protons). The qualitative results fig. 3 show that MOHART, using self-attention for relational reasoning, is able to capture these interactions with high accuracy. Figure 4 (left) shows quantitative comparison of augmenting HART with different relational reasoning modules when identities are re-assigned in every timestep (randomness space with an MLP leads to slightly worse performance than the HART bascline, while simple max-pe max-pooling performs significantly better (AloU 17%). This be explained through the permutation invariance of the problem: the list latent representation of the different objects has no meaningful order and the output of the model should therefore be invariant the ordering of the objects. The MLP is in itself not permutation
1.0). Exchanging information between trackers of different objects in the latent
6

---

Under review conference paper at ICLR 2020
0.8
0.8
0.7
0.7
0.6
0.6
16U
loU
Self-Attention
0.4
0.4
Max Pooling
HART
DoepSets
rel. reasoning)
20
30
40
50
60
90
0.2
0.4
0.6
0.8
1.0
Sequence Length
Randomness
FIGURE 4: Left: average IoU over sequence length for different implementations of relational reasoning on the toy domain shown in fig- 3 (randomness 1.0). Right: performance depending how often agents are re-assigned identities randomly (sequence length 15). The higher the randomness, the less static the force field is and the more vital relational reasoning For randomness 0.0, identities still have to be reassigned in some cases in order to prevent deadlocks, this leads to a performance loss for all models, which explains lower performance of self-attention for randomness 0.0.
invariant and therefore prone to overfit to the (meaningless) order of the objects in the training data. Max- pooling, however, permutation invariant and in theory, despite its simplicity, be used approximate any permutation invariant function given a sufficiently large latent space (Wagstaff et al. (2019). Max- pooling is often used to exchange information between different tracklets, e.g., in the trajectory prediction domain (Alahi et al. (2016); Gupta et al. (2018)) However, self-attention, allowing for learned querying and encoding of information, solves the relational reasoning task significantly more accurately. In fig. (right), the frequency with which object identities are reassigned randomly is varied. The results show that, deterministic environment, tracking does not necessarily profit from relational reasoning even the presence of long-range interactions. The less random, the more static the force field is and static force field can be inferred from a small number of observations (see fig. (). This does not mean that all stochastic environments profit from relational reasoning. What these experiments indicate is that tracking can not be expected to profit from relational reasoning by default in any environment, but instead in environments which feature (potentially non-deterministic) dynamics and predictable interactions.
5 RELATIONAL REASONING IN REAL-WORLD TRACKING
Having established that MOHART is capable of performing complex relational reasoning, we now the algorithm on three real world datasets and analyse the effects of relational reasoning performance de- pending on dataset and task. We find consistent improvements MOHART compared to HART throughout. Relational reasoning yields particularly high gains for scenes with ego-motion, crowded scenes, and simu- lated faulty sensor inputs.
5.1 EXPERIMENTAL DETAILS
We investigate three qualitatively different datasets: the MOTChallenge dataset Milan et al. (006)).. the UA-DETRAC dataset (Wen et al.] (2015)), and the Stanford Drone dataset (Robicquet et al. (2016). To increase scene dynamics and make the tracking/prediitionn problems challenging, sub-sample some the high framerate scenes with a stride of two, resulting in scenes with 7-15 frames per second. Training and architecture details given in Appendices A Aand We conduct experiments in three different modes:
7

---

Under review conference paper at ICLR 2020
mitp
uudpn
m
1-16
22
FIGURE 5: Camera blackout experiment on a street scene from the MOTChallenge dataset with strong ego- motion. Solid boxes are MOHART predictions (for > 2), faded bounding boxes indicate object locations in the first frame. As the model is trained end-to-end, MOHART learns to fall back onto its internal motion model if new observations are available (black frames). As soon as new observations come in, the model 'snaps' back onto the tracked objects.
Tracking. The model is initialised with the ground truth bounding boxes for set of objects in the first frame. It then consecutively sees the following frames and predicts the bounding boxes. The sequence length is 30 time steps and the performance is measured as intersection over union (IoU) averaged over the entire sequence excluding the first frame. This algorithm is either applied to the entire dataset or subsets of it to study the influence of certain properties of the data.
Camera Blackout. This simulates unsteady or faulty sensor inputs. The setup is the same as in Tracking, but sub-sequences of the input are replaced with black images. The algorithm is expected recognise that no new information is available and that it should resort to its internal motion model.
Prediction. Testing MOHART's ability capture motion patterns, only the first two frames are shown the model followed by three black frames. IoU is measured seperately for each time step.
5.2 RESULTS AND ANALYSIS
On the MOTChallenge dataset, HART achieves 66.6% intersection over union (see Table D. which in itself is impressive given the small amount of training data of only 5225 training frames and no pre-training. MOHART achieves 68.5% 5% (both numbers are averaged over runs, independent samples (-test resulted P 0.0001). The performance gain increases when only considering ego- ego-motion data. This is readily explained: movements objects in the image space due to ego-motion are correlated and can therefore be better understood when combining information from movements of multiple objects, i.e. performing relational reasoning. In another ablation, we filtered for only crowded scenes by requesting five objects to be present for, average, 90% of the frames in a sub-sequence. For the MOT-Challenge dataset, this only
8

---

Under review conference paper at ICLR 2020
TABLE 1: Tracking performance on the MOTChallenge dataset measured in loU.
Entire Dataset
Only Ego-Motion
No Ego-Motion
Crowded Scenes
Camera Blackout
MOHART HART
68.5%
66.9% 66.6%
64.7%
69.1
63.6%
64.0%
62.9%
66.9%
60.6%
1.9%
2.9%
1.8%
2.2%
3.0%
TABLE 2: UA-DETRAC Dataset
TABLE 3: Stanford Drone Dataset
All
Crowded Scenes
Camera Blackout
All
Camera Blackout
CamBlack Bikes
MOHART HART
68.15 1
69.5%
64.2% %
57.3%
53.3%
53.3%
68.4%
68.6%
53.8%
56. 1
52.6%
50.7%
A
-0.3%
0.9%
0.4%
1.2%
0.7%
2.6%
leads to a minor increase of the performance gain of MOHART indicating that the dataset exhibits a sufficient density of objects to learn interactions. The biggest benefit from relational reasoning can be observed in the camera blackout experiments (sctup explained in Section 5.11. Both HART and MOHART learn to rely their internal motion models when confronted with black frames and propagate the bounding boxes according the previous movement of the objects. It is unsurprising that this scenario profits particularly from relational reasoning. Qualitative tracking and camera blackout results are shown in fig. and Appendix
E
Tracking performance on the UA-DETRAC dataset only profits from relational reasoning when filtering for crowded scenes (see Table [}. The fact that the performance of MOHART is slightly worse on the vanilla dataset (A- = - 0.3%) can be explained with more overfitting. As there is no exchange between trackers for each object, each object constitutes an independent training sample.
The Stanford drone dataset (see Table 3) is different to the other two-it filmed from birds-eye view. The scenes are more crowded and each object covers a small number of pixels, rendering a difficult problem for tracking. The dataset was designed for trajectory prediction- setup where an algorithm is typically provided with ground -trtt tracklets in coordinate space and potentially an image as context information. The task is then to extrapolate these tracklets into the future. The tracking performance profits from rela- tional reasoning more than on the UA-DETRAS dataset but less than on the MOTChallenge dataset. The performance gain the camera blackout experiments are particularly strong when only considering cyclists.
the prediction experiments (see Appendix D). MOHART consistently outperforms HART. On both datasets, the model outperforms baseline which uses momentum linearly extrapolate the bounding boxes from the first two frames. This shows that even from just two frames, the model learns to capture motion models which are more complex than what could be observed from just the bounding boxes (i.e. momentum), suggesting that it uses visual information (HART & MOHART) as well as relational reasoning (MOHART).
6 CONCLUSION
With MOHART, introduce end-to-end multi-object tracker that is capable of capturing complex in- teractions and leveraging these for precise predictions experiments both on toy and real world data show. However, the experiments also show that the benefit of relational reasoning strongly depends on the nature of the data. The toy experiments showed that in an entirely deterministic world relational reasoning was much less important than in stochastic environment. Amongst the real-world dataset, the highest perfor- mance gains from relational reasoning were achieved on the MÖTChallenge dataset, which features crowded scenes, ego-motion and occlusions.
9

---

Under review conference paper at ICLR 2020
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR, 2016.
Geung-Hwan Bae and Kuk-Jin Yoon. Confidence-based data association and discriminative deep appearance learning for robust online multi-object tracking. IEEE Trunsactions Pattern Analysis and Machine Intelligence, 2017.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physies. NIPS, 2016.
Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Soft+ hardwired attention: An Istm frame- work for human trajectory prediction and abnormal event detection. Neural networks, 2018.
Daniel Gordon, Ali Farhadi, and Dicter Fox. Re3 Real-Time Recurrent Regression Networks for Visual Tracking of Generie Objects. RA-L, 2018.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: socially acceptable jectories with generative adversarial networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
David Held, Sebastian Thrun, and Silvio Savarese. Learning to track at 100 fps with deep regression networks. In European Conference on Computer Vision, 2016.
Samira Ebrahimi Kahou, Vincent Michalski, and Roland Memisevic. RATM: recurrent attentive tracking model. IEEE
Conference on Computer Vision and Pattern Recognition Workshops, 2017.
Margret Keuper, Siyu Tang, Bjor Andres, Thomas Brox, and Bernt Schielc. Motion segmentation & multiple object
tracking by correlation co-clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018.
Adam R. Kosiorek, Alex Bewley, and Ingmar Posner. Hierarchical attentive recurrent tracking. Neural Information Processing Systems, 2017.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer. In International Conference on Machine Learning, 2019.
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer Graphics Forum, 2007.
A. Milan, L. Leal-Taixé, Reid, S. Roth, and K. Schindler. MOT16: benchmark for multi-object tracking. arXiv, 2016.
Anton Milan, Stefan Roth, and Konrad Schindler. Continuous energy minimization for multitarget tracking. PAMI, 2014.
Stefano Pellegrini, Andreas Ess, Konrad Schindler. and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In ICCV, 2009.
Maryam Rasouli Danesh, Srishti Yadav, Sachini Herath, Yasaman Vaghei, Shahram Payandeh. Deep attention models for human tracking using rgbd. Sensors, 19:750, 02 2019.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Conference on Computer Vision and Pottern Recognition, 2016.
A. Robiequet, A. Sadeghian, A. Alahi, and S. Savaresci. Learning social etiquette: Human trajectory prediction in crowded scenes. European Conference Computer Vision, 2016.
10

---

Under review conference paper at ICLR 2020
Andrey Rudenko, Luigi Palmieri, and Kai Arras. Joint long-term prediction human motion using planning based social force approach. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.
Amir Sadeghian. Vineet Kosaraju, Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Christoph Schöller, Vincent Aravantinos, Florian Lay. and Alois Knoll. The simpler the better: Constant velocity for pedestrian motion prediction. arXiv preprint arXiv: 0et//9933079... 2019.
Hang Su, Yinpeng Dong, Jun Zhu, Haibin Ling, and Bo Zhang. Crowd scene understanding with coherent recurrent neural networks. In International Joint Conferences on Artificial Intelligence, 2016.
Sun, Zhi Yan, Sergi Molina Mellado, Mare Hanheide, and Tom Duckett. 3dof pedestrian trajectory prediction learned from long-term autonomous mobile robot deployment data. In 2018 IEEE International Conference on Robotics and Automation. IEEE, 2018.
Peter Trautman and Andreas Krause. Unfreezing the robot: Navigation in dense, interacting crowds. In IROS, 2010.
Jack Valmadre, Luca Bertinetto, João F. Henriques, Andrea Vedaldi, and Philip Hilaire Sean Torr. End-to-end rep- resentation learning for correlation filter based tracking. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sjoerd van Steenkiste. Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximiza- tion: Unsupervised discovery of objects and their interactions. ICLR, 2018.
Daksh Varshneya and G Srinivasaraghavan. Human trajectory prediction using spatially aware deep attention models. arXiv preprint: 1atpam::1550994994 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Information Processing Systems, 2017.
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osborne. On the limitations of representing functions sets. International Conference on Machine Learning, 2019.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual Interaction Networks: Learning Physies Simulator from Video. NIPS, 2017.
Longyin Wen, Dawei Du. Zhaowei Cai, Zhen Lei. Ming-Ching Chang, Honggang Qi. Jongwoo Lim, Ming-Hsuan Yang, and Siwei Lyu. DETRAC: A new benchmark and protocol for multi-object tracking. arXiv, 1511.04136, 2015.
Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara I. Berg. Who you with and where are you going? In CVPR, 2011.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander Smola. Deep Sets. In Advances in Neural Information Processing Systems, 2017.
Li Zhang, Yuan Li, and Ramakant Nevatia. Global data association for multi-object tracking using network flows. CVPR, 2008.
11

---

Under review conference paper at ICLR 2020
A EXPERIMENTAL DETAILS
The MOTChallenge and the UA-DETRAC dataset discussed in this section are intended to be used a benchmark suite for multi-objject-rackkng in a macking-by-detectingg paradigm. Therefore, ground truth bounding boxes are only available for the training datasets. The user encouraged to upload their model which performs tracking in a data association paradigm leveraging the provided bounding box proposals from an external object detector. As we are interested in a different analysis (IoU given inital bounding boxes), we divide the training data further into training and test sequences. To make up for the smaller training data, we extend the MOTChallenge 2017 dataset with three sequences from the 2015 dataset (ETH- Sunnyday, PETS09-S2L1, ETH-Bahnhof). We use the first 70% of the frames of each of the ten sequences for training and the rest for testing. Sequences with high frame rates (30Hz) are sub-sampled with a stride of two. For the UA-DETRAC dataset, split the 60 available sequences into 44 training sequences and 16 test sequences. For the considerably larger Stanford Drone dataset we took three videos of the scene deathCirele for training and the remaining two videos from the same scene for testing. The videos of the drone dataset were also sub-sampled with a stride of two to increase scene dynamics.
B ARCHITECTURE DETAILS
The architecture details were chosen optimise HART performance on the MOTChallenge dataset. They deviate from the original HART implementation (Kosiorek et al. (2017)) as follows: A presence variable predicts whether an object is in the scene and successfully tracked. This is trained with a binary cross entropy loss. The maximum number of objects to be tracked simultancously was set to 5 for the UA-DETRAC and MOTChallenge dataset. For the more crowded Stanford drone dataset, this number was set to 10. The feature extractor is a three layer convolutional network with kernel size of 5, a stride of 2 in the first and last layer, 32 channels the first two layers, 64 channels in the last layer, ELU activations, and skip connections. This converts the initial 32 x 32 x 3 glimpse into 7 x 7 x 64 feature representation. This is followed by a fully connected layer with a 128 dimensional output and an elu activation. The spatial attention parameters are linearly projected onto 128 dimensions and added this feature representation serving positional encoding. The LSTM has a hidden state size of 128. The self-attention unit in MOHART comprises linear projects the inputs to dimensionality 128 for each keys, queries and values. For the real-world experiments, addition to the extracted features from the glimpse, the hidden states from the previous LSTM state are also fed an input by concatinating them with the features. In all cases, the output of the attention module is concatenated to the input features of the respective object.
As an optimizer, we used RMSProp with momentum set to 0.9 and learning rate 5 10-6, For the MOTChal- lenge dataset and the UA-DETRAC dataset, the models were trained for 100,000 iterations of batch size 10 and the reported IoU is exponentially smoothed over iterations achieve lower variance. For the Stanford Drone dataset, the batch size was increased to 32, reducing time to convergence and hence model training to 50,000 iterations.
C DETERMINISTIC TOY DOMAIN
In our first experiment in the toy domain (Figure 6). four circles each exert repulsive forces on each other, where the force scales with 1/r, r being their distance. HART applied four times in parallel and is trained to predict the location of each circle three time steps into the future. The different forces from different objects lead to a non-trivial force field at each time step. Predicting the future location just using the previous motion of one object (Figure 6 shows that each spatial attention box covers only the current object) accurately is therefore challenging. Surprisingly, the single object tracker solves this task with an average of 95% loU over sequences of 15 time steps. This shows the efficacy of end-to-end tracking to capture complex motion
12

---

Under review conference paper at ICLR 2020
FIGURE 6: HART single object tracking applied four times parallel and trained to predict the location each circle three time steps into the future. Dashed lines indicate spatial attention, solid lines are predicted bounding boxes, faded circles show ground truth location + 3. Each circle exerts repulsive forces on each other, where the force scales with 1/r, being their distance.
input images unseen by model
MOHART
86.4%
80.0%
74.0%
68.4%
HART
85.3%
79.2%
73.1%
67.4%
Momentum
78.2%
70.9%
63.9%
(A) Prediction results the MOTChallenge dataset Milan al. (2016).
input images unseen by model
MOHART
87.8%
81.3%
75.5%
70.0%
HART
86.6%
79.6%
72.6%
66.1%
Momentum
80.5%
73.6%
67.2%
(B) Prediction results the UA-DETRAC dataset (crowded only) Wen et al, 2015
FIGURE 7: Peeking into the future. Only the first two frames shown the tracking algorithm followed by three black frames. MOHART learns to fall back on its internal motion model when no observation (i.c. only black frame) is available. The reported IoU scores show the performance for the respective frames 0, 1, 2, and 3 time steps into the future.
patterns and them predict future locations. This, of course, could also be used to generate robust bounding boxes for a tracking task.
D PREDICTION EXPERIMENTS
the results from the prediction experiments (see Figure 7 MOHART consistently outperforms HART. both datasets, the model outperforms bascline which uses momentum to linearly extrapolate the bounding boxes from the first two frames. This shows that even from just two frames, the model Iearns to capture motion models which
more complex than what could be observed from just the bounding boxes (i.e. momentum), suggesting that it uses visual information (HART & MOHART) as well as relational reasoning (MOHART). The strong performance gain of MOHART compared to HART the UA-DETRAC dataset,
13

---

Under review conference paper at ICLR 2020
despite the small differences for tracking on this dataset, can be explained as follows: this dataset features little interactions but strong correlations in motion. Hence when only having access to the first two frames, MOHART profits from estimating the velocities of multiple cars simultaneously.
QUALITATIVE TRACKING RESULTS
HART (Singlc-Objer Tracking)
A
MOHIART (Multi-Objeet Tracking)
B
c
time
FIGURE 8: Tracking examples of both HART and MOHART. Coloured boxes are bounding boxes predicted by the model, arrows point challenging aspects of the scenes. (A) & (C): Each person being tracked is temporarily occluded by a woman walking across the scene (blue arrows). MOHART, which includes a relational reasoning module, handles this more robustly (compare red arrows).
we tested MOHART on three different real world data sets and in a number of different setups. Figure 8 shows qualitative results both for HART and MOHART the MOTChallenge dataset.
Section 5
Furthermore, we conducted a set of camera blackout experiments to test MOHART's capability of dealing with faulty sensor inputs. While traditional pipeline methods require careful consideration of different types corner cases to properly handle erroneous sensor inputs, MOHART is able to capture these automatically. especially when confronted with similar issues the training scenarios. To simulate this, we replace sub- sequences of the images with black frames. Figure and Figure 5 show two such examples from the test data together with the model's prediction. MOHART with black frames and instead uses the LSTM to propagate the bounding boxes. When proper sensor input is available again, the model uses this make rapid adjustment its predicted location and 'snap' back onto the object. This works remarkably well in both the presence of occlusion (Figure 9 and ego-motion (Figure (5.. Tables [ to 3 show that benefit relational reasoning particularly high these scenar- ios specifically. These experiments also be seen proof of concept of MOHART's capabalities of predicting future trajectories and how this profits from relational reasoning.
not to update its internal model when confronted
14

---

Under review conference paper at ICLR 2020
FIGURE 9: Camera blackout experiment on a pedestrian street scene from the MOTChallenge dataset with- out ego-motion. Subsequent frames displayed going from top left bottom right. Shown are the inputs to the model (some them being black frames, i.e. arrays of zeroes) and bounding boxes predicted by MOHART (coloured boxes). This scene is particularly challenging occlusion and missing sensor input coincide (fourth row).
15


Reviews:
[{"review_id": "S1xCmqbaFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a class-agnostic method for tracking multiple moving objects (MOHART) that extends an existing single-object tracking method (Hierarchical Attentive Recurrent Tracking, HART). Similarly to HART, MOHART utilizes an attention mechanism and LSTM units. The extension form HART to MOHART is done in two main steps: HART is applied to multiple objects in parallel, with a presence variable attached to each, and a permutation-invariant network that learns the interactions between the objects.\n\nThe method is tested in two groups of experiments: synthetic data of moving circles, and on several real datasets. The results on the moving circles are enlightening. The paper shows that although MOHART outperforms HART in all circumstances, if the forces between the circles change randomly every number of steps, MOHART outperforms HART by a very large margin. This is because the forces between the circles are determined by their color, which is observable by the network, and MOHART, which capable of learning their interaction, learns the pattern of changing forces. HART, being an independent-particle model, cannot learn the interactions.\n\nThe results on the real datasets demonstrate superiority over HART, and in a series of ablation experiments, the Authors are able to show that MOHART's ability to utilize interactions between objects is responsible for the improvement. For example, the improvement over HART is greater when egomotion is significant, because egomotion creates correlations between the movement of objects in the camera frame, which MOHART can utilize.\n\nI found the paper enlightening, based on a neat idea. The experimental results are extensively analyzed and the Author's insights about their algorithm are substantiated by the experiments. \n\nHowever the difficulty I had with the paper is that the results are only compared to HART. HART was published in 2017, and the field of scene understanding and tracking had been addressed in multiple papers since then. It is one of the more competitive and application-driven fields in computer vision. Comparing MOHART only to its sister-method from over two years ago significantly limits the usefulness of paper IMHO. The expectation IMHO is not that MOHART outperform each and every method out there, but that the reader know where MOHART stands compared to other methods (e. g. tracking by detection).\n\nI will refrain from pointing to specific papers to compare to, because I believe the Authors should chose the settings, datasets, criteria, and metrics that are the most convenient for them for comparison. The papers surveyed in Section 2, especially under \"Tracking-by-Detection\", have many followups. Recent papers from those that claim state of the art could good candidates for comparison.\n\nIn view of the above, I am inclined at this stage to reject the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1lqFdj0tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the problem of multiple object tracking and trajectory prediction in multiple frames of videos. The main focus is adding a relation-reasoning building block to the original HART framework. With multiple objects, the key is to be able to learn the permutation invariant representation during potential changing and dynamic object trajectories. The paper also uses toy examples to show that the proposed block of relation reasoning is not necessarily beneficial when the object trajectory is less random and more static. Finally, experiments on real data demonstrate that the proposed method that accounts for relation reasoning is helpful by a limited magnitude.\n \nThe main contribution of this paper is the novel relation reasoning block. However, there are three key concerns of mine:\n \n1. It is not clear when the new architecture would be helpful. I am really happy to see a more careful study of the success and failure cases of the method. I also appreciate the honesty that the current model is not competitive with the ones that have an accurate bounding box as input. But I think a more detailed study can be conducted, especially in the toy example case.  Maybe it is possible to design different levels of randomness in the trajectory and further figure out when is the reasoning block helpful.\n\n2. Even the real data experiments do not have very impressive results, from my biased observation. (I am not in the field, so maybe I am wrong here.) \n\n3. The end-to-end approach is also another perspective where the authors try to differentiate their methods from others. I think it is potentially an interesting problem that why end-to-end is something you would prefer (which is definitely a harder problem), and when. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkgBQO1aFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm somewhat out of area for this review: I study relational models, but have little experience with computer vision in general and object tracking in particular. \n\n--------------------\n\nThis paper presents an extension to HART [Kosiorek 2017] to track multiple objects. It builds on a simple baseline: run multiple HART models in parallel, each tracking one of the objects. They extend this by adding a relational reasoning module to allow interaction between the parallel models. The relational reasoning module uses Lee et al. (2019)'s self-attention block (similar to Vaswani et al. 2017). They find that the simple baseline is surprisingly effective, but that MOHART (their model) improves performance in environments with stochastic interactions and crowded settings which tend to be more noisy.\n\nThe novelty is somewhat limited: they're plugging together two existing approaches (HART and SAB) to allow for interaction, but I am recommending acceptance because I found the experiments surprising (as a negative result) - they show that independent HART models are a strong baseline for tracking in settings that involve interactions, but that are nevertheless solvable without knowledge about the state of other objects. For the synthetic experiments, I would have expected the fact there are repulsive forces between the objects would have been sufficient for the relational module to be helpful. The paper also provides further evidence of the usefulness of non-local networks for these sorts of problems.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an end-to-end object tracker by exploiting the attention mechanism. Two reviewers recommend rejection, while the last reviewer is more positive. The concerns brought up are novelty (last reviewer), and experiments (second reviewer). Furthermore, the authors seem to overclaim their contribution. There indeed are end-to-end multi-object trackers, see Frossard & Urtasun's work for example. This work needs to be cited, and possibly a comparison is needed. Since the paper did not receive favourable reviews and there are additional citations missing, this paper cannot be accepted in current form. The authors are encouraged to strengthen their work and resubmit to a future venue.
Decision: Reject