Improving End-to-End Object Tracking Using Relational Reasoning

1 INTRODUCTION . Real-world environments can be rich and contain countless types of interacting objects . Intelligent autonomous agents need to understand both the objects and interactions between them if they are to operate in those environments . This motivates the need for class-agnostic algorithms for tracking multiple objects—a capability that is not supported by the popular tracking-by-detection paradigm . In tracking-by-detection , objects are detected in each frame independently , e. g. , by a pre-trained deep convolutional neural network ( CNN ) such as YOLO ( Redmon et al . ( 2016 ) ) , and then linked across frames . Algorithms from this family can achieve high accuracy , provided sufficient labelled data to train the object detector , and given that all encountered objects can be associated with known classes , but fail when faced with objects from previously unseen categories . Hierarchical attentive recurrent tracking ( HART ) is a recently-proposed , alternative method for single-object tracking ( SOT ) , which can track arbitrary objects indicated by the user ( Kosiorek et al . ( 2017 ) ) . This is done by providing an initial bounding-box , which may be placed over any part of the image , regardless of whether it contains an object or what class the object is . HART efficiently processes just the relevant part of an image using spatial attention ; it also integrates object detection , feature extraction , and motion modelling into one network , which is trained fully end-to-end . Contrary to tracking-by-detection , where only one video frame is typically processed at any given time to generate bounding box proposals , end-to-end learning in HART allows for discovering complex visual and spatio-temporal patterns in videos , which is conducive to inferring what an object is and how it moves . In the original formulation , HART is limited to the single-object modality—as are other existing end-to-end trackers ( Kahou et al . ( 2017 ) ; Rasouli Danesh et al . ( 2019 ) ; Gordon et al . ( 2018 ) ) . In this work , we present MOHART , a class-agnostic tracker with complex relational reasoning capabilities provided by a multi-headed self-attention module ( Vaswani et al . ( 2017 ) ; Lee et al . ( 2019 ) ) . MOHART infers the latent state of every tracked object in parallel , and uses self-attention to inform per-object states about other tracked objects . This helps to avoid performance loss under self-occlusions of tracked objects or strong camera motion . Moreover , since the model is trained end-to-end , it is able to learn how to manage faulty or missing sensor inputs . See fig . 1 for a high-level illustration of MOHART . In order to track objects , MOHART estimates their states , which can be naturally used to predict future trajectories over short temporal horizons , which is especially useful for planning in the context of autonomous agents . MOHART can be trained simultaneously for object tracking and trajectory prediction at the same time , thereby increasing statistical efficiency of learning . In contrast to prior art , where trajectory prediction and object tracking are usually addressed as separate problems with unrelated solutions , our work show trajectory prediction and object tracking are best addressed jointly . Section 2 describes prior art in tracking-by-detection , end-to-end tracking and predestrian trajectory prediction . In Section 3 , we describe our approach , which uses a permutation-invariant self-attention module to enable tracking multiple objects end-to-end with relational reasoning . Section 4 contrasts our approach with multi-object trackers which do not explicitly enforce permutation invariance but have the capacity to learn it , simpler permutation-invariant architectures , as well as multiple single-object trackers running in parallel . We show that multi-headed self-attention significantly outperforms other approaches . Finally , in Section 5 , we apply MOHART to real world datasets and show that permutation-invariant relational reasoning leads to consistent performance improvement compared to HART both in tracking and trajectory prediction . 2 RELATED WORK . Tracking-by-Detection Vision-based tracking approaches typically follow a tracking-by-detection paradigm : objects are first detected in each frame independently , and then a tracking algorithm links the detections from different frames to propose a coherent trajectory ( Zhang et al . ( 2008 ) ; Milan et al . ( 2014 ) ; Bae and Yoon ( 2017 ) ; Keuper et al . ( 2018 ) ) . Motion models and appearance are often used to improve the association between detected bounding-boxes in a postprocessing step . Tracking-by-detection algorithms currently provide the state-of-the-art in multi-object tracking on common benchmark suites , and we fully acknowledge that MOHART is not competitive at this stage in scenarios where high-quality detections are available for each frame . MOHART can in principle be equipped with the ability to use bounding boxes provided by an object detector , but this is beyond the scope of this project . End-to-End Tracking A newly established and much less explored stream of work approaches tracking in an end-to-end fashion . A key difficulty here is that extracting an image crop ( according to bounding-boxes provided by a detector ) , is non-differentiable and results in high-variance gradient estimators . Kahou et al . ( 2017 ) propose an end-to-end tracker with soft spatial-attention using a 2D grid of Gaussians instead of a hard bounding-box . HART draws inspiration from this idea , employs an additional attention mechanism , and shows promising performance on the real-world KITTI dataset ( Kosiorek et al . ( 2017 ) ) . HART forms the foundation of this work . It has also been extended to incorporate depth information from RGBD cameras ( Rasouli Danesh et al . ( 2019 ) ) . Gordon et al . ( 2018 ) propose an approach in which the crop corresponds to the scaled up previous bounding-box . This simplifies the approach , but does not allow the model to learn where to look— i. e. , no gradient is backpropagated through crop coordinates . To the best of our knowledge , there are no successful implementations of any such end-to-end approaches for multi-object tracking beyond SQAIR ( Kosiorek et al . ( 2018 ) ) , which works only on datasets with static backgrounds . On real-world data , the only end-to-end approaches correspond to applying multiple single-object trackers in parallel—a method which does not leverage the potential of scene context or inter-object interactions . Pedestrian trajectory prediction Predicting pedestrian trajectories has a long history in computer vision and robotics . Initial research modelled social forces using hand-crafted features ( Lerner et al . ( 2007 ) ; Pellegrini et al . ( 2009 ) ; Trautman and Krause ( 2010 ) ; Yamaguchi et al . ( 2011 ) ) or MDP-based motion transition models ( Rudenko et al . ( 2018 ) ) , while more recent approaches learn from context information , e. g. , positions of other pedestrians or landmarks in the environment . Social-LSTM ( Alahi et al . ( 2016 ) ) employs a long short-term memory ( LSTM ) to predict pedestrian trajectories and uses max-pooling to model global social context . Attention mechanisms have been employed to query the most relevant information , such as neighbouring pedestrians , in a learnable fashion ( Su et al . ( 2016 ) ; Fernando et al . ( 2018 ) ; Sadeghian et al . ( 2019 ) ) . Apart from relational learning , context ( Varshneya and Srinivasaraghavan ( 2017 ) ) , periodical time information ( Sun et al . ( 2018 ) ) , and constant motion priors ( Schöller et al . ( 2019 ) ) have proven effective in predicting long-term trajectories . Our work stands apart from this prior art by not relying on ground truth tracklets . It addresses the more challenging task of working directly with visual input , performing tracking , modelling interactions , and , depending on the application scenario , simultaneously predicting future motions . As such , it can also be compared to Visual Interaction Networks ( VIN ) ( Watters et al . ( 2017 ) ) , which use a CNN to encode three consecutive frames into state vectors—one per object—and feed these into a recurrent neural network ( RNN ) , which has an Interaction Network ( Battaglia et al . ( 2016 ) ) at its core . More recently , Relational Neural Expectation Maximization ( R-NEM ) has been proposed as an unsupervised approach which combines scene segmentation and relational reasoning ( van Steenkiste et al . ( 2018 ) ) . Both VINs and R-NEM make accurate predictions in physical scenarios , but , to the best of our knowledge , have not been applied to real world data . 3 RECURRENT MULTI-OBJECT TRACKING WITH SELF-ATTENTION . This section describes the model architecture in fig . 1 . We start by describing the hierarchical attentive recurrent tracking ( HART ) algorithm ( Kosiorek et al . ( 2017 ) ) , and then follow with an extension of HART to tracking multiple objects , where multiple instances of HART communicate with each other using multiheaded attention to facilitate relational reasoning . We also explain how this method can be extended to trajectory prediction instead of just tracking . 3.1 HIERARCHICAL ATTENTIVE RECURRENT TRACKING ( HART ) . HART is an attention-based recurrent algorithm , which can efficiently track single objects in a video . It uses a spatial attention mechanism to extract a glimpse gt , which corresponds to a small crop of the image xt at time-step t , containing the object of interest . This allows it to dispense with the processing of the whole image and can significantly decrease the amount of computation required . HART uses a CNN to convert the glimpse gt into features ft , which then update the hidden state ht of a LSTM core . The hidden state is used to estimate the current bounding-box bt , spatial attention parameters for the next time-step at+1 , as well as object appearance . Importantly , the recurrent core can learn to predict complicated motion conditioned on the past history of the tracked object , which leads to relatively small attention glimpses—contrary to CNN-based approaches ( Held et al . ( 2016 ) ; Valmadre et al . ( 2017 ) ) , HART does not need to analyse large regions-of-interest to search for tracked objects . In the original paper , HART processes the glimpse with an additional ventral and dorsal stream on top of the feature extractor . Early experiments have shown that this does not improve performance on the MOTChallenge dataset , presumably due to the oftentimes small objects and overall small amount of training data . Further details are provided in Appendix B . The algorithm is initialised with a bounding-box1 b1 for the first time-step , and operates on a sequence of raw images x1 : T . For time-steps t ≥ 2 , it recursively outputs bounding-box estimates for the current time-step and predicted attention parameters for the next time-step . The performance is measured as intersection-overunion ( IoU ) averaged over all time steps in which an object is present , excluding the first time step . Although HART can track arbitrary objects , it is limited to tracking one object at a time . While it can be deployed on several objects in parallel , different HART instances have no means of communication . This results in performance loss , as it is more difficult to identify occlusions , ego-motion and object interactions . Below , we propose an extension of HART which remedies these shortcomings . 3.2 MULTI-OBJECT HIERARCHICAL ATTENTIVE RECURRENT TRACKING ( MOHART ) . Multi-object support in HART requires the following modifications . Firstly , in order to handle a dynamically changing number of objects , we apply HART to multiple objects in parallel , where all parameters between HART instances are shared . We refer to each HART instance as a tracker . Secondly , we introduce a presence variable pt , m for object m. It is used to mark whether an object should interact with other objects , as well as to mask the loss function ( described in ( Kosiorek et al . ( 2017 ) ) ) for the given object when it is not present . In this setup , parallel trackers can not exchange information and are conceptually still single-object trackers , which we use as a baseline , referred to as HART ( despite it being an extension of the original algorithm ) . Finally , to enable communication between trackers , we augment HART with an additional step between feature extraction and the LSTM . 1We can use either a ground-truth bounding-box or one provided by an external detector ; the only requirement is that it contains the object of interest . For each object , a glimpse is extracted and processed by a CNN ( see fig . 1 ) . Furthermore , spatial attention parameters are linearly projected on a vector of the same size and added to this representation , acting as a positional encoding . This is then concatenated with the hidden state of the recurrent module of the respective object ( see fig . 2 ) . Let ft , m denote the resulting feature vector corresponding to the mth object , and let ft,1 : M be the set of such features for all objects . Since different objects can interact with each other , it is necessary to use a method that can inform each object about the effects of their interactions with other objects . Moreover , since features extracted from different objects comprise a set , this method should be permutation-equivariant , i. e. , the results should not depend on the order in which object features are processed . Therefore , we use the multi-head self-attention block ( SAB , Lee et al . ( 2019 ) ) , which is able to account for higher-order interactions between set elements when computing their representations . Intuitively , in our case , SAB allows any of the trackers to query other trackers about attributes of their respective objects , e. g. , distance between objects , their direction of movement , or their relation to the camera . This is implemented as follows , Q = Wqf1 : M + bq , K = Wkf1 : M + bk , V = Wvf1 : M + bv , ( 1 ) Oi = softmax ( QiK T i ) Vi , i = 1 , . . . , H , ( 2 ) o1 : M = O = concat ( Oi , . . . , OH ) , ( 3 ) where om is the output of the relational reasoning module for object m. Time-step subscripts are dropped to decrease clutter . In Eq . 1 , each of the extracted features ft , m is linearly projected into a triplet of key kt , m , query qt , m and value vt , m vectors . Together , they comprise K , Q and V matrices with M rows and dq , dk , dk columns , respectively . K , Q and V are then split up into multiple heads H ∈ N+ , which allows to query different attributes by comparing and aggregating different projection of features . Multiplying QiKTi in Eq . 2 allows to compare every query vector qt , m , i to all key vectors kt,1 : M , i , where the value of the corresponding dot-products represents the degree of similarity . Similarities are then normalised via a softmax operation and used to aggregate values V . Finally , outputs of different attention heads are concatenated in Eq . 3 . SAB produces M output vectors , one for each input , which are then concatenated with corresponding inputs and fed into separate LSTMs for further processing , as in HART—see fig . 1 . MOHART is trained fully end-to-end , contrary to other tracking approaches . It maintains a hidden state , which can contain information about the object ’ s motion . One benefit is that in order to predict future trajectories , one can simply feed black frames into the model . Our experiments show that the model learns to fall back on the motion model captured by the LSTM in this case .

Summary: The paper presents a class-agnostic method for tracking multiple moving objects (MOHART) that extends an existing single-object tracking method (Hierarchical Attentive Recurrent Tracking, HART). Similarly to HART, MOHART utilizes an attention mechanism and LSTM units. The extension form HART to MOHART is done in two main steps: HART is applied to multiple objects in parallel, with a presence variable attached to each, and a permutation-invariant network that learns the interactions between the objects.