Self-Supervised Multi-Object Tracking with Cross-input Consistency

1 Introduction . Multi-object trackers identify all instances of a particular object type in video , and track each instance through the segment of video in which it is visible in the camera frame . Annotating training data for multi-object tracking is tedious and costly ; for example , annotation of pedestrian tracks in just six minutes of video in the training set of the MOT15 Challenge [ 14 ] requires an estimated 22 hours [ 20 ] of human labeling time using LabelMe [ 28 ] . While unsupervised , heuristic detect-to-track methods [ 2 , 4 ] have been proposed that group detections into tracks by estimating motion using a combination of spatial and visual cues , these methods suffer low-accuracy in scenarios with frequent occlusion where heuristics are insufficient . Recent work has proposed applying self-supervised learning for training single-object tracking models on unlabeled video [ 24 , 25 ] . These approaches train a model to propagate instance labels from a reference frame through the rest of a video sequence . In contrast to work on self-supervised representation learning from video , these fully unsupervised approaches do not require fine-tuning to apply the model for single-object tracking . However , a significant limitation in prior work is that the model independently compares pairs of frames at a time . In multi-object tracking , a key challenge is robustly re-localizing tracks across potentially long occlusions , especially when an object instance is occluded by other instances of the same object type . Pairwise frame comparisons are thus insufficient for high-accuracy multi-object tracking ; instead , learning recurrent features that encode the history of a track is crucial for enabling robust re-localization . However , extending prior work to learn RNN parameters is challenging . For example , Wang et al . [ 25 ] propose training using forward-backward consistency : from a patch in an initial frame , after tracking forwards through video and then backwards to return to the initial frame , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the final patch should align with the original patch . Training an RNN in this way would be ineffective as the RNN could simply memorize the features of the original patch . To address this challenge , we propose a novel self-supervised learning method , cross-input consistency . We first compute object detections in each frame of unlabeled video ( like unsupervised , heuristic detect-to-track methods , we assume that a robust detector is available ) . Then , we derive a learning signal from the unlabeled video by sampling a short sequence of contiguous frames from the video , constructing two input variations of that sequence that each hide different information about objects detected in the sequence , and training the tracker to produce consistent tracking outputs when applied independently on each of the two inputs . We propose two alternative input-hiding schemes for computing the input variations : visual-spatial hiding and occlusion-based hiding . Visual-spatial hiding applies the tracker once when only observing spatial inputs ( bounding box coordinates in the video frame ) , and once when only observing visual inputs ( pixel values inside detection boxes ) . Occlusion-based hiding eliminates information about object detections in random intermediate subsequences of frames to simulate occlusion incidents ; thus , it constructs two inputs by eliminating different subsequences of detections in each input . After sampling a sequence of video and computing the two input variations under the chosen input-hiding scheme , we apply the tracker model independently on each input , and back-propagate a learning signal that measures the consistency between tracks computed across the two inputs . To attain high consistency , the model must accurately group detections that correspond to the same object : if the model were to instead arbitrarily group detections into tracks , then variations in the inputs would cause the tracker to produce inconsistent outputs . To implement cross-input consistency , we adapt a now standard RNN model and tracker architecture from prior work [ 12 ] : the tracker processes each frame in sequence by matching detections in the current frame with tracks computed up to the previous frame . In prior work , this model is trained under a supervised procedure : they sample a video sequence 〈I0 , . . . , In〉 and a track t in that sequence , and apply the tracker on t over the sequence . On each frame Ij , the RNN outputs a probability distribution indicating the likelihood that the prefix of a track t up to Ij matches with each detection in Ij . Prior work back-propagates the label ( i.e. , the correct detection of t in Ij ) under cross entropy loss . In contrast , under our method , on each training iteration , we propose to sample a sequence 〈I0 , . . . , In〉 from a corpus of unlabeled video , and apply the RNN model to compute a transition matrix that specifies the probability that each detection in I0 ( rows ) matches with each detection in In ( columns ) . We select the sequence length n so that most objects in I0 are still visible in In . Then , when applying the tracker on two input variations extracted from the sequence , we obtain two transition matrices ( one for each input ) . We compute the dot-product similarity to measure the consistency between these matrices , and back-propagate the negative similarity as a loss function . We evaluate our approach on the MOT17 and KITTI benchmarks against 9 baselines , including both unsupervised and supervised methods . We train our tracker model using cross-input consistency over a corpus of unlabeled video , which can be cheaply obtained . Like other unsupervised methods , we use an object detector trained on image-level bounding box annotations in COCO [ 17 ] , but do not use any expensive video-level annotations . We find that , on MOT17 , our approach improves both IDF1 and MOTA accuracy over the unsupervised baselines by 14 % to 18 % . Moreover , remarkably , our fully unsupervised approach outperforms five of the seven supervised methods we compared , even though these methods train on expensive video-level bounding box and track annotations . Our code is available at https : //favyen.com/uns20/ . 2 Related Work . Self-supervised learning over video has been studied extensively in many contexts . Most work focuses on learning representations of video that can be applied through fine-tuning for tasks such as activity recognition , image classification , and object detection [ 6 , 7 , 9 , 15 , 23 , 26 ] . More closely related to our work , several recent approaches have proposed leveraging widely available unlabeled video to directly train single-object tracking models , without needing fine-tuning [ 13 , 16 ] . Vondrick et al . [ 24 ] train a model to colorize gray-scale video by propagating colors from a colored reference frame . The model is then applied to track objects at inference time by propagating instance IDs instead of colors . Wang et al . [ 25 ] train a model to capture correspondence by applying a cycle-consistent loss : from a patch in an initial frame , after tracking forwards through video and then backwards to return to the initial frame , the final patch should align with the original patch . As we discussed in Section 1 , self-supervisory signals used in prior work such as color propagation and cycle-consistency are not effective for training RNN models , which are needed to achieve accurate MOT . Our work is also related to unsupervised , heuristic detect-to-track multi-object tracking methods such as SORT [ 2 ] and V-IOU [ 4 ] . These methods group detections across different frames using a combination of heuristic spatial cues ( e.g. , Kalman filter over bounding box coordinates ) and visual cues ( e.g. , optical flow ) to track objects . Like our approach , these methods assume that a robust detector is available ; however , because they rely on heuristics to group detections into tracks , they suffer low-accuracy in scenarios with frequent occlusion . Multi-object tracking has been studied extensively in supervised settings , where methods are trained on video-level bounding box and track annotations [ 1 , 5 , 8 , 12 , 18 , 30 ] . However , such annotations are expensive to hand-label , and so these methods are costly to extend to new types of video . Other work explores using unsupervised and self-supervised learning to further improve the performance of fully supervised methods . SimpleReID [ 11 ] proposes improving the performance of one supervised method , CenterTrack [ 30 ] , by training a re-identification model through unsupervised learning . However , while the model can in principle be trained only on image-level annotations through hallucinated motion techniques , their SimpleReID+CenterTrack tracking method depends on expensive video-level annotations to attain high-accuracy . In contrast , our method achieves competitive results without any video-level supervision . 3 Cross-Input Consistency . In our novel cross-input consistency method , we derive an learning signal for training an RNN tracker model through a three-step process . We assume that a corpus of unlabeled video is available , along with an object detector for the object category of interest . During pre-processing , we apply the detector on each frame of unlabeled video to compute object detections . Then , during training , we first repeatedly randomly sample a sequence of contiguous frames from the video , 〈I0 , . . . , In〉 , where each Ik is a video frame . Let Dk by the detections automatically computed in Ik by the detector , and let dki = ( im , x , y , w , h ) be a detection in Dk , where ( x , y , w , h ) are the 4D spatial coordinates ( center point and lengths ) of the detection bounding box , and im is the window of Ik corresponding to that box . We apply an input-hiding scheme to select two input variations A ( D ) , B ( D ) for the video segment , where each variation is a modified sequence of detections in the frames , A ( D ) = 〈DA0 , . . . , DAn 〉 , B ( D ) = 〈DB0 , . . . , DBn 〉 . For example , some detections may be removed entirely , while others may be partially hidden . Second , we apply the tracker model independently on each input variation to derive two probabilistic tracking outputs ( one per input ) , represented as transition matrices . Third , we compare the transition matrices with dot-product similarity to update the RNN parameters . Figure 1 summarizes our approach . Below , we first introduce the model architecture that we adapt from prior work in Section 3.1 . We then detail our novel training procedure , including the computation of the transition matrices and dot-product loss , in Section 3.2 . Finally , we propose two input-hiding schemes for selecting the input variations required by our approach in Section 4 . 3.1 Background : Tracker Model . We adopt a tracker model that is similar to prior work [ 12 ] . We summarize the architecture in Figure 2 . Given a video sequence 〈I0 , . . . , In〉 , and sets of detections Dk = { dk1 , . . . , dkmk } detected in each frame Ik , to initialize the tracking process , we create a length-1 track ti = 〈d0i 〉 for each detection d0i in the first video frame I0 . When processing subsequent frames , we will match the new detections with existing tracks , extending existing tracks if there is a match and initializing new tracks otherwise . Specifically , on each subsequent frame Ik , the model outputs a probability pi , j that each track ti corresponds to each detection dkj ∈ Dk . At inference time , we formulate the problem of matching tracks with detections in Ik as a bipartite matching problem , where the cost of matching ti with dkj is 1− pi , j . We solve this problem and compute a minimum-cost matching using the Hungarian algorithm ; for each pair ( ti , dkj ) in the matching , we append d k j to ti . For each detection in Ik that no track matches to , we create a new track for that detection . The model consists of a CNN , RNN , and matcher network . Together , these components score the likelihood that the ith track , ti = 〈d1 , . . . , dm〉 , matches with the jth detection in Ik , dkj . We first apply the CNN to derive detection-level features . Given a detection d = ( im , x , y , w , h ) , the CNN inputs im resized to 64× 64 , and consists of 6 strided convolutional layers , with ReLU activation in the first 5 layers and linear activation in the last layer . It outputs a 64-vector , which we concatenate with the 4D spatial coordinates to derive a 68-vector detection representation f ( d ) . Then , we compute track-level features f ( ti ) by applying the RNN ( an LSTM with 64 hidden states ) over the sequence of detection-level features of detections in the track , 〈f ( d1 ) , . . . , f ( dm ) 〉 . We use the output of the RNN on the last timestep as the track-level features f ( ti ) . Finally , we apply a matching network to score the likelihood that ti matches dkj . The matching network inputs the concatenation of f ( ti ) and f ( dkj ) , applies four fully-connected layers , and outputs a match score .

Summary: This paper proposes a self-supervised learning scheme for training Multi-Object Tracking (MOT) models using only unlabeled video. The proposed scheme constructs two distinct inputs by hiding different information about the sequence in each input and trains the model to produce consistent tracks across the two inputs. The proposed algorithm is evaluated on the MOT17 and KITTI datasets with comparisons against several supervised methods.