Self-Supervised Multi-Object Tracking with Cross-input Consistency

1 Introduction . Multi-object trackers identify all instances of a particular object type in video , and track each instance through the segment of video in which it is visible in the camera frame . Annotating training data for multi-object tracking is tedious and costly ; for example , annotation of pedestrian tracks in just six minutes of video in the training set of the MOT15 Challenge [ 14 ] requires an estimated 22 hours [ 20 ] of human labeling time using LabelMe [ 28 ] . While unsupervised , heuristic detect-to-track methods [ 2 , 4 ] have been proposed that group detections into tracks by estimating motion using a combination of spatial and visual cues , these methods suffer low-accuracy in scenarios with frequent occlusion where heuristics are insufficient . Recent work has proposed applying self-supervised learning for training single-object tracking models on unlabeled video [ 24 , 25 ] . These approaches train a model to propagate instance labels from a reference frame through the rest of a video sequence . In contrast to work on self-supervised representation learning from video , these fully unsupervised approaches do not require fine-tuning to apply the model for single-object tracking . However , a significant limitation in prior work is that the model independently compares pairs of frames at a time . In multi-object tracking , a key challenge is robustly re-localizing tracks across potentially long occlusions , especially when an object instance is occluded by other instances of the same object type . Pairwise frame comparisons are thus insufficient for high-accuracy multi-object tracking ; instead , learning recurrent features that encode the history of a track is crucial for enabling robust re-localization . However , extending prior work to learn RNN parameters is challenging . For example , Wang et al . [ 25 ] propose training using forward-backward consistency : from a patch in an initial frame , after tracking forwards through video and then backwards to return to the initial frame , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the final patch should align with the original patch . Training an RNN in this way would be ineffective as the RNN could simply memorize the features of the original patch . To address this challenge , we propose a novel self-supervised learning method , cross-input consistency . We first compute object detections in each frame of unlabeled video ( like unsupervised , heuristic detect-to-track methods , we assume that a robust detector is available ) . Then , we derive a learning signal from the unlabeled video by sampling a short sequence of contiguous frames from the video , constructing two input variations of that sequence that each hide different information about objects detected in the sequence , and training the tracker to produce consistent tracking outputs when applied independently on each of the two inputs . We propose two alternative input-hiding schemes for computing the input variations : visual-spatial hiding and occlusion-based hiding . Visual-spatial hiding applies the tracker once when only observing spatial inputs ( bounding box coordinates in the video frame ) , and once when only observing visual inputs ( pixel values inside detection boxes ) . Occlusion-based hiding eliminates information about object detections in random intermediate subsequences of frames to simulate occlusion incidents ; thus , it constructs two inputs by eliminating different subsequences of detections in each input . After sampling a sequence of video and computing the two input variations under the chosen input-hiding scheme , we apply the tracker model independently on each input , and back-propagate a learning signal that measures the consistency between tracks computed across the two inputs . To attain high consistency , the model must accurately group detections that correspond to the same object : if the model were to instead arbitrarily group detections into tracks , then variations in the inputs would cause the tracker to produce inconsistent outputs . To implement cross-input consistency , we adapt a now standard RNN model and tracker architecture from prior work [ 12 ] : the tracker processes each frame in sequence by matching detections in the current frame with tracks computed up to the previous frame . In prior work , this model is trained under a supervised procedure : they sample a video sequence 〈I0 , . . . , In〉 and a track t in that sequence , and apply the tracker on t over the sequence . On each frame Ij , the RNN outputs a probability distribution indicating the likelihood that the prefix of a track t up to Ij matches with each detection in Ij . Prior work back-propagates the label ( i.e. , the correct detection of t in Ij ) under cross entropy loss . In contrast , under our method , on each training iteration , we propose to sample a sequence 〈I0 , . . . , In〉 from a corpus of unlabeled video , and apply the RNN model to compute a transition matrix that specifies the probability that each detection in I0 ( rows ) matches with each detection in In ( columns ) . We select the sequence length n so that most objects in I0 are still visible in In . Then , when applying the tracker on two input variations extracted from the sequence , we obtain two transition matrices ( one for each input ) . We compute the dot-product similarity to measure the consistency between these matrices , and back-propagate the negative similarity as a loss function . We evaluate our approach on the MOT17 and KITTI benchmarks against 9 baselines , including both unsupervised and supervised methods . We train our tracker model using cross-input consistency over a corpus of unlabeled video , which can be cheaply obtained . Like other unsupervised methods , we use an object detector trained on image-level bounding box annotations in COCO [ 17 ] , but do not use any expensive video-level annotations . We find that , on MOT17 , our approach improves both IDF1 and MOTA accuracy over the unsupervised baselines by 14 % to 18 % . Moreover , remarkably , our fully unsupervised approach outperforms five of the seven supervised methods we compared , even though these methods train on expensive video-level bounding box and track annotations . Our code is available at https : //favyen.com/uns20/ . 2 Related Work . Self-supervised learning over video has been studied extensively in many contexts . Most work focuses on learning representations of video that can be applied through fine-tuning for tasks such as activity recognition , image classification , and object detection [ 6 , 7 , 9 , 15 , 23 , 26 ] . More closely related to our work , several recent approaches have proposed leveraging widely available unlabeled video to directly train single-object tracking models , without needing fine-tuning [ 13 , 16 ] . Vondrick et al . [ 24 ] train a model to colorize gray-scale video by propagating colors from a colored reference frame . The model is then applied to track objects at inference time by propagating instance IDs instead of colors . Wang et al . [ 25 ] train a model to capture correspondence by applying a cycle-consistent loss : from a patch in an initial frame , after tracking forwards through video and then backwards to return to the initial frame , the final patch should align with the original patch . As we discussed in Section 1 , self-supervisory signals used in prior work such as color propagation and cycle-consistency are not effective for training RNN models , which are needed to achieve accurate MOT . Our work is also related to unsupervised , heuristic detect-to-track multi-object tracking methods such as SORT [ 2 ] and V-IOU [ 4 ] . These methods group detections across different frames using a combination of heuristic spatial cues ( e.g. , Kalman filter over bounding box coordinates ) and visual cues ( e.g. , optical flow ) to track objects . Like our approach , these methods assume that a robust detector is available ; however , because they rely on heuristics to group detections into tracks , they suffer low-accuracy in scenarios with frequent occlusion . Multi-object tracking has been studied extensively in supervised settings , where methods are trained on video-level bounding box and track annotations [ 1 , 5 , 8 , 12 , 18 , 30 ] . However , such annotations are expensive to hand-label , and so these methods are costly to extend to new types of video . Other work explores using unsupervised and self-supervised learning to further improve the performance of fully supervised methods . SimpleReID [ 11 ] proposes improving the performance of one supervised method , CenterTrack [ 30 ] , by training a re-identification model through unsupervised learning . However , while the model can in principle be trained only on image-level annotations through hallucinated motion techniques , their SimpleReID+CenterTrack tracking method depends on expensive video-level annotations to attain high-accuracy . In contrast , our method achieves competitive results without any video-level supervision . 3 Cross-Input Consistency . In our novel cross-input consistency method , we derive an learning signal for training an RNN tracker model through a three-step process . We assume that a corpus of unlabeled video is available , along with an object detector for the object category of interest . During pre-processing , we apply the detector on each frame of unlabeled video to compute object detections . Then , during training , we first repeatedly randomly sample a sequence of contiguous frames from the video , 〈I0 , . . . , In〉 , where each Ik is a video frame . Let Dk by the detections automatically computed in Ik by the detector , and let dki = ( im , x , y , w , h ) be a detection in Dk , where ( x , y , w , h ) are the 4D spatial coordinates ( center point and lengths ) of the detection bounding box , and im is the window of Ik corresponding to that box . We apply an input-hiding scheme to select two input variations A ( D ) , B ( D ) for the video segment , where each variation is a modified sequence of detections in the frames , A ( D ) = 〈DA0 , . . . , DAn 〉 , B ( D ) = 〈DB0 , . . . , DBn 〉 . For example , some detections may be removed entirely , while others may be partially hidden . Second , we apply the tracker model independently on each input variation to derive two probabilistic tracking outputs ( one per input ) , represented as transition matrices . Third , we compare the transition matrices with dot-product similarity to update the RNN parameters . Figure 1 summarizes our approach . Below , we first introduce the model architecture that we adapt from prior work in Section 3.1 . We then detail our novel training procedure , including the computation of the transition matrices and dot-product loss , in Section 3.2 . Finally , we propose two input-hiding schemes for selecting the input variations required by our approach in Section 4 . 3.1 Background : Tracker Model . We adopt a tracker model that is similar to prior work [ 12 ] . We summarize the architecture in Figure 2 . Given a video sequence 〈I0 , . . . , In〉 , and sets of detections Dk = { dk1 , . . . , dkmk } detected in each frame Ik , to initialize the tracking process , we create a length-1 track ti = 〈d0i 〉 for each detection d0i in the first video frame I0 . When processing subsequent frames , we will match the new detections with existing tracks , extending existing tracks if there is a match and initializing new tracks otherwise . Specifically , on each subsequent frame Ik , the model outputs a probability pi , j that each track ti corresponds to each detection dkj ∈ Dk . At inference time , we formulate the problem of matching tracks with detections in Ik as a bipartite matching problem , where the cost of matching ti with dkj is 1− pi , j . We solve this problem and compute a minimum-cost matching using the Hungarian algorithm ; for each pair ( ti , dkj ) in the matching , we append d k j to ti . For each detection in Ik that no track matches to , we create a new track for that detection . The model consists of a CNN , RNN , and matcher network . Together , these components score the likelihood that the ith track , ti = 〈d1 , . . . , dm〉 , matches with the jth detection in Ik , dkj . We first apply the CNN to derive detection-level features . Given a detection d = ( im , x , y , w , h ) , the CNN inputs im resized to 64× 64 , and consists of 6 strided convolutional layers , with ReLU activation in the first 5 layers and linear activation in the last layer . It outputs a 64-vector , which we concatenate with the 4D spatial coordinates to derive a 68-vector detection representation f ( d ) . Then , we compute track-level features f ( ti ) by applying the RNN ( an LSTM with 64 hidden states ) over the sequence of detection-level features of detections in the track , 〈f ( d1 ) , . . . , f ( dm ) 〉 . We use the output of the RNN on the last timestep as the track-level features f ( ti ) . Finally , we apply a matching network to score the likelihood that ti matches dkj . The matching network inputs the concatenation of f ( ti ) and f ( dkj ) , applies four fully-connected layers , and outputs a match score .

Summary: This paper presents a novel framework to train the multi-object tracker in an unsupervised scheme. The framework utilizes the cross-input consistency assuming that the two clips manipulated by one video must have the same tracking trajectory. The two manipulated clips are randomly generated by one of the occlusion-based hiding and visual-spatial hiding. By using the baseline of the RNN-based tracker, the proposed framework shows comparable performance even with the supervised MOT.