Self-Supervised Multi-Object Tracking with Cross-input Consistency
Authors: Favyen Bastani
Songtao He
Samuel Madden
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-Object Tracking
Self-Supervised Learning

Abstract: In this paper, we propose a self-supervised learning procedure for training a robust multi-object tracking (MOT) model given only unlabeled video. While several self-supervisory learning signals have been proposed in prior work on single-object tracking, such as color propagation and cycle-consistency, these signals are not effective for training RNN models, which are needed to achieve accurate MOT: they yield degenerate models that, for instance, always match new detections to tracks with the closest initial detections. We propose a novel self-supervisory signal that we call cross-input consistency: we construct two distinct inputs for the same sequence of video, by hiding different information about the sequence in each input. We then compute tracks in that sequence by applying an RNN model independently on each input, and train the model to produce consistent tracks across the two inputs. We evaluate our unsupervised method on MOT17 and KITTI --- remarkably, we find that, despite training only on unlabeled video, our unsupervised approach outperforms four supervised methods published in the last 1--2 years, including Tracktor++, FAMNet, GSM, and mmMOT.

Self-Supervised Multi-Object Tracking with Cross-Input Consistency
Favyen Bastani
Songtao He MIT CSAIL
Sam Madden MIT CSAIL madden@csail. mit. edu
MIT CSAIL favyen@csail mit. edu
songtao@csail mit. edu
Abstract
In this paper, we propose self-supervised learning procedure for training robust multi-object tracking (MOT) model given only unlabeled video. While several self-supervisory learning signals have been proposed prior work on single-object tracking, such as color propagation and cycle-consistency, these signals cannot be directly applied for training RNN models, which are needed to achieve accurate MOT; they yield degenerate models that, for instance, always match new detections tracks with the closest initial detections. We propose a novel self-supervisory signal that we call cross cross-input consistency: we construct two distinct inputs for the same sequence of video, by hiding different information about the sequence in each input. We then compute tracks in that sequence by applying an RNN model independently on each input, and train the model produce consistent tracks across the two inputs. We evaluate our unsupervised method on MOT17 and KITTI remarkably, we find that, despite training only on unlabeled video, our unsupervised approach outperforms four supervised methods published in the last 1-2 years, including Tracktor++ [1, FAMNet [5), GSM [8] and mmMOT (9].
Introduction
Multi-object trackers identify all instances of particular object type in video, and track each instance through the segment of video in which it is visible in the camera frame. Annotating training data for multi-object tracking tedious and costly; for example, annotation of pedestrian tracks in just six minutes of video in the training set of the MOTI5 Challenge [14] requires estimated 22 hours (00 of human labeling time using LabelMe (28). While unsupervised, heuristic detect-to-trace
methods 2 [ have been proposed that group detections into tracks by estimating motion using combination spatial and visual cues, these methods suffer low-accuracy in scenarios with frequent occlusion where heuristics insufficient.
Recent work has proposed applying self-supervised learning for training single-object tracking models unlabeled video [24] 255 These approaches train model to propagate instance labels from a reference frame through the rest video sequence. In contrast to work on self-supervised representation learning from video, these fully unsupervised approaches do not require fine-tuning to apply the model for single-object tracking.
However, significant limitation in prior work is that the model independently compares pairs of frames time. multi-object tracking, a key challenge is robustly re-localizing tracks across potentially long occlusions, especially when object instance is occluded by other instances of the same object type. Pairwise frame comparisons are thus insufficient for high-accuracy multi-object tracking; instead, learning recurrent features that encode the history of track crucial for enabling robust re-localization. However, extending prior work to learn RNN parameters is challenging, For example, Wang et al. (25 propose training using dorward-backwadd consistency: from patch in an initial frame, after tracking forwards through video and then backwards to return to the initial frame,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the final patch should align with the original patch. Training an RNN in this way would be ineffective as the RNN could simply memorize the features of the original patch.
To address this challenge, we propose a novel self-supervised learning method, cross-inpul con- sistency. We first compute object detections in each frame of unlabeled video (like unsupervised, heuristic detect-to-track methods, we assume that a robust detector is available). Then, we derive a learning signal from the unlabeled video by sampling short sequence of contiguous frames from the video, constructing two input variations of that sequence that each hide different information about objects detected in the sequence, and training the tracker to produce consistent tracking outputs when applied independently on each of the two inputs. We propose two alternative input-hiding schemes for computing the imput variations: visual-spatial hiding and occlusion- based hiding. Visual- souual-ppttall hiding applies the tracker once when only observing spatial inputs (bounding box coordinates in the video frame), and once when only observing visual inputs (pixel values inside detection boxes). Occlusion-based hiding eliminates information about object detections in random intermediate sub- sequences of frames to simulate occlusion incidents; thus, it constructs two inputs by eliminating different subsequences of detections in each input. After sampling a sequence of video and com- puting the two input variations under the chosen input-hiding scheme, we apply the tracker model independently each input, and back-propagate a learning signal that measures the consistency between tracks computed across the two inputs. To attain high consistency, the model must accurately group detections that correspond to the same object: if the model were to instead arbitrarily group detections into tracks, then variations in the inputs would cause the tracker to produce inconsistent outputs.
To implement cross-input consistency, we adapt a now standard RNN model and tracker architecture from prior work [12] the tracker processes each frame in sequence by matching detections in the current frame with tracks computed up to the previous frame. In prior work, this model is trained under a supervised procedure: they sample video sequence (Jo...., In.) and a track & in that sequence, and apply the tracker on & over the sequence. On each frame 1j. the RNN outputs a probability distribution indicating the likelihood that the prefix of track / up to I matches with each detection in Ij. Prior work back propagatsss the label (i.e., the correct detection of t in Iÿ) under cross entropy loss.
In contrast, under our method, on each training iteration, we propose to sample a sequence (To...., In)) from corpus of unlabeled video, and apply the RNN model to compute transi- tion matrix that specifies the probability that each detection in To (rows) matches with each detection in In (columns). We select the sequence length n so that most objects in to are still visible in In- Then, when applying the tracker on two input variations extracted from the sequence, we obtain two transition matrices (one for each input). We compute the dot-pre not-product similarity to measure the consistency between these matrices, and paok-proe the negative similarity as a loss function.
We evaluate our approach on the MOT17 and KITTI benchmarks against 9 baselines, including both unsupervised and supervised methods. We train our tracker model using cross-input consistency over corpus of unlabeled video, which can be cheaply obtained. Like other unsupervised methods, we use an object detector trained on image-level bounding box annotations in COCO [17, but do not use any expensive video-level annotations. We find that, on MOT17, our approach improves both IDF1 and MOTA accuracy over the unsupervised baselines by 14% to 18% Moreover, remarkably, our fully unsupervised approach outperforms five of the seven supervised methods we compared, even though these methods train on expensive video-level bounding box and track annotations.
Our code is available at httts:///fayeen com/uns20
2 Related Work
Self-supervised learning over video has been studied extensively in many contexts. Most work focuses on learning representations of video that can be applied through fine-tuning for tasks such as activity recognition, image classification, and object detection 6 our work, several recent approaches have proposed leveraging widely available unlabeled video to
7 9 [S|
23 26]. More closely related to
directly train single-object tracking models, without needing fine-tuning [3] 16). Vondrick et al. [24] train a model to colorize gray-scale video by propagating colors from a colored reference frame. The model is then applied to track objects at inference time by propagating instance IDs instead of colors. Wang et al. [25] train model to capture correspondence by applying a cycle-consistent loss: from

---

Video Sequence
Input A
Matrix
Matrix
Input
Derecfoons in
Detecionss In
Desection:
Tracker Model
0.1
Dat Product Similarity +1007
01 0.0
Tracker
02 01
0. 0.5
Model
0.9 0.0
Figure I: Overview of cross-input consistency. An input-hiding scheme produces two inputs, A and B, from video sequence; these inputs contain identical information about objects detected in the first and last frames of the sequence, but vary in intermediate frames. We apply the tracker model on each input to derive transition matrices that match detections between the first and last frames to represent tracker outputs. We then pack-propagate a similarity score between the matrices that encourages the model produce consistent outputs across both inputs.
Active Tracks (1)
((d)), Mdy, Kdy)
CNN
Track
Matching Network
RNN
Score: 0.13
Detections in 10
kdP
Figure 2: Tracker model architecture. The model scores the likelihood that each detection in the current frame matches with each track.
a patch in an initial frame, after tracking forwards through video and then backwards to return to the initial frame, the final patch should align with the original patch. As discussed in Section [ self-supervisory signals used in prior work such as color propagation and cycle-consistency are not effective for training RNN models, which are needed to achieve accurate MOT.
Our work is also related to unsupervised, heuristic detect-tor to-cct--o-trakk multi-object tracking methods such as SORT [2] and V-IOU 4 These methods group detections across different frames using combination of heuristic spatial cues (e.g., Kalman filter over bounding box coordinates) and visual cues (e.g., optical flow) to track objects. Like our approach, these methods assume that robust detector is available; however, because they rely on heuristics to group detections into tracks, they suffer low-accuracy in scenarios with frequent occlusion.
Multi-object tracking has been studied extensively in supervised settings, where methods are trained on video-level bounding box and track annotations 5 expensive to hand-label, and so these methods costly to extend new types of video.
8 12
18] 30]. However, such annotations
Other work explores using unsupervised and self sulf-supereieed learning to further improve the perfor- mance of fully supervised methods. SimpleRelD [II] proposes improving the performance of one supervised method, CenterTrrcch [30), by training re-identification model through unsupervised learning. However, while the model can in principle be trained only image-level annotations through hallucinated motion techniques, their SimpleRelD+Center expensive video-level annotations attain high-accuracy. In contrast, our method achieves competitive results without any video-level supervision.
tracking method depends
Cross-Input Consistency
In our novel cross-input consistency method, we derive an learning signal for training an RNN tracker model through three-step process. We assume that corpus of unlabeled video is available, along with an object detector for the object category interest. During pre-processing, we apply the detector on each frame of unlabeled video to compute object detections. Then, during training,

---

we first repeatedly randomly sample a sequence of contiguous frames from the video, (fo.... In). where each L video frame. Let D; by the detections automatically computed in 17 by the detector, and let di = (im, x, y, we h)) be detection in Da. where (a,, y, w, h) are the 4D spatial coordinates (center point and lengths) of the detection bounding box, and im is the window of Ta corresponding to that box. We apply an input-hiding scheme to select two input variations ((D) for the video segment, where each variation is a modified sequence of detections in the frames, A(D) (Dg) ...., D B(D) = (Df ..... D For example, some detections may be removed entirely, while others may be partially hidden. Second, we apply the tracker model independently each input variation to derive two probabilistic tracking outputs (one per input), represented as transition matrices. Third, we compare the transition matrices with dot-product similarity to update the RNN parameters.
Figure D summarizes our approach.
Below, we first introduce the model architecture that we adapt from prior work in Section 3.1 We then detail our novel training procedure, including the computation of the transition matrices and dot-product loss, in Section Finally, we propose two input-hiding schemes for selecting the input variations required by our approach in Section 4
3.1 Background: Tracker Model
We adopt a tracker model that is similar to prior work 12 We summarize the architecture in Figure Given a video sequence (Jo...., In), and sets of detections D; = ={df.... di detected in each frame J2, to initialize the tracking process, we create a length-1 1 track 4 = (d)) for each detection di in the first video frame To- When processing subsequent frames, we will match the new detections with existing tracks, extending existing tracks if there is a match and initializing new tracks otherwise. Specifically, on each subsequent frame It. the model outputs a probability pt, that each track 6; corresponds to each detection dt E Dr. At inference time, we formulate the problem of matching tracks with detections in 1 as bipartite matching problem, where the cost of matching 4 with de is - Pt.j. We solve this problem and compute a minimum-cost matching using the Hungarian algorithm; for each pair (t,, d() ) in the matching, we append d to tt. For each detection in Ig that no track matches to, we create a new track for that detection.
2
The model consists of a CNN, RNN, and matcher network. Together, these components score the likelihood that the ith track, 4 (di,.... dm),, matches with the jth detection in TR. dz. We first apply the CNN to derive detection-level features. Given detection d = (im, I. y; wr h), the CNN inputs im resized to 64 x 64, and consists of 6 strided convolutional layers, with ReLU activation in the first 5 layers and linear activation in the last layer. It outputs 64-vector, which we concatenate with the 4D spatial coordinates to derive a 68-vector detection representation ((d). Then, we compute track-level features f(44) by applying the RNN (an LSTM with 64 hidden states) over the sequence of detection- features of detections in the track, (f(d)...... (dmm) We use the output of the RNN on the last timestep as the track-level features ft.)). Finally, we apply a matching network to score the likelihood that i, matches dz. The matching network inputs the concatenation of f (t:) and [(d)), applies four fully-connected layers, and outputs a match score.
3.2 Training Procedure
We develop a novel self-supervised learning method for training the model parameters on unlabeled video. During training. we repeatedly sample sequences of video "h....,,,), We apply one of two input-hiding schemes, which we will detail in the following section, to extract two distinct input variations A(D) and B(D) from sampled video sequence, where each input is sequence of detections. We then apply the tracker independently on A(D) and (DD to derive two tracking outputs for the same video sequence. In cross-input consistency, we train the model by enforcing similarity between these two outputs.
To represent tracker outputs, we compute an DDol x |D,] + transition matrix M where M[ .j < DDal is the probability that the track t; matches dj. We use the last column to represent tracks that are no longer visible in I,, i.e., (( is the probability that the track t, has exited the camera frame. When applying the model over video sequences during training, we update tracks with

---

new detections based on the scores output by the model on intermediate frames, but do not create additional tracks on frames after lg: thus, each track t, corresponds directly to a detection of in fo (i.e., 1, (d' ....,,,)) .)). Thus, we can also think of M()0x) as the probability that detection the first frame de matches a detection in the last frame d".
Applying the tracker on both input variations yields two transition matrices A(0m)) and ((0.n) that match objects detected in To with those in 7,. We train the model (CNN, RNN, and matching network) to maximize the dot-product similarity between these matrices. In addition to pushing the model to produce consistent outputs across both inputs, we also design our training method so that the model cannot attain a high similarity score by, for example, saying that all objects visible in Io are no longer visible in n,-
In our method, it is important that the training sequence length be chosen that, in most sequences, most (but not all) objects in Io are still visible in L, but that objects nevertheless move non-trive -oiiiiivillly during the sequence (so that the tracking task is not too easy). In general, we find that setting n to one-half of the average time that objects linger in the camera frame works well; this value can be quickly estimated by hand-labeling the duration of a few (e.g., 10-20) objects randomly sampled from the video.
Below, we detail our method to compute transition matrices, and discuss dot-product similarity loss.
Transition Matrix. We propose computing a transition matrix M (0.k) on each frame Is to represent the tracker outputs, where 1. is the probability that the track t; matches the detection dý. On intermediate frames, we apply the Hungarian method on M/ match detections in D₁ with tracks, updating each track with the matched detection (if any). On the last frame 4n, use the M (0.n) matrix produced under different inputs (denoted (00m)) and ((0.))) to compute and pack-propagate a consistency score. Because we do not create new tracks after do during training, M, (0.n) is the likelihood that d. and d' match (since 4; begins with d.).
We first construct a score matrix S(U,k), by computing s(0.k) as the score (any real number) output by the tracker model given the track 4 and detection dy We then transform the score matrix into probability matrix derive (0.A), We could simply compute (0.K) by taking softmax along rows in ((0.k), However, computing the transition matrix this way would allow the tracker to cheat and maximize similarity between (00,n) and (00m)) by simply matching all detections in To to a single detection d' E D.. Indeed, we find that in practice this yields degenerate models.
Thus, instead, we compute (0ww)),row and (oll) by applying softmax along rows and columns, respectively, and compute ( = min (U)),,,, (U,)),.oo)
exp(Sin) (Xp(S,.))
exp(S...)
M
M
M., min Mg9h)
E. exp(Sk.))
This produces transition matrix (0.A) that is almost doubly stochastic: rows and columns to at most 1, but not necessarily exactly The operation ensures that the model must match each detection in lo to unique detections in In to maximize the consistency score between (0..n) and B(0.)); if two detections in Io are matched to the same detection in In, then the columnar softmax would reduce those probabilities in the corresponding matrix to at most 0.5, thereby reducing any dot dot-products involving those rows.
Dot-Product Similarity. We train the RNN tracker by computing two transition matrices A (0,n) and (0..)) over different input variations, and then back- propagating loss that measures the inconsistency between the matrices. In particular, we use the dot-product to measure the similarity of corresponding rows in the matrices. We define the loss as:
L=
Σ log Σ (0.m) (0.a)
Here, L is computed by taking the logarithm of the dot product of corresponding rows in A (U, and ((0.a), averaged across rows. Note that this is equivalent to the cross-entropy loss between the diagonal matrix and the matrix product of A (0,n) and the transpose of g00.s)
5

---

%
Tracker A (Visual)
(93, 681, 176, 103)
(490, 429, 211, 140)
(847, 325, 53, 37)
(609, 282, 75, 48)
Tracker B (Spatial)
(950, 459, 143. 75)
(740. 575, 285, 127)
(458. 313, 62, 49)
(899. 337, 62, 48)
(652, 333, 78, 87)
(821. 335, 63, 47)
[199 336, 62, 47)
(848, 325, 51, 38)
Figure 3: Visual-spatial hiding. One input includes only visual information about the detections, while the other includes only spatial bounding box coordinates.
This loss function has several desirable properties. First, the dot-product is maximized when the matrices computed based on different inputs are most similar. This pushes the matching model to learn reasonable visual and spatial tracking constraints, because arbitrarily matching tracks to detections will lead to dissimilarity. Second, the dot-product pushes each matrix to be almost doubly stochastic rather than leaving some rows and columns summing to much less than 1. The model can only produce doubly stochastic A (0.n) and (0,m) matrices by finding unique detections in In for each detection in Lo.
Spatial Mask. In some cases, training as described above may converge at local minimum where the model outputs uniform probabilities for entries M. mitigate this issue, we add spatial constraint to the loss that penalizes the tracker when it matches detections that are highly improbable to correspond to the same object based on bounding box positions. We first compute a mask matrix ((0..) so that (0,,) 0 if it is "improbable" that de matches d"j, and Cx(O.x) otherwise. Then, we compute L as:
L L= log Σ ((
(0.m) B
((0.n)
(0,)) 1. 1.j
We determine whether matches in C are improbable by applying simple floodfill-like algorithm that propagates sets of labels from the first frame In to the last frame In. the label from a detection de in Io does not propagate that could form a track between di and dy. In Jo. we label each detection di with set containing only that detection, i.e., {. In 1g. we label each detection dł with the union of sets of labels of detections d-d in preceding frames 1x-1 (1 < < 10) such that the bounding boxes of dł and de intersect. Note that consider several preceding frames since the detector may occasionally fail to localize an object in an intermediate frame. Then, (. 1 only if the label set for dlt includes d!
detection d'.. then it implies there no sequence of intermediate detections
Artificial Detections. To improve the model's robustness in learning visual features, artificially construct additional detections in I by pairing the spatial coordinates of detections in I, with object images selected randomly from frames in the underlying video data that temporally far from (Jo,..., In). Thus, these artificial detections added to D,, have correct spatial coordinates, but include visual cues that do not correspond to any object in Iu. and the tracker model must learn leverage visual cues that it does not assign high probabilities in M n) to artificial detections.
We exclude artificial detections in the mask C. Then, to perform well under dot-product similarity, the model must learn to leverage visual features to distinguish the correct detections in I, from artificially constructed ones tracker that only considers spatial features would assign half of its probability mass along each row to artificial detections, and thus would be penalized by the loss.
4 Input-Hiding Schemes
In this section, we detail two alternative input-hiding schemes for selecting the two input variations, denoted A(D) = (D&.... DA) and B(D) = {Df D Recall that D is the original set of all objects detected in a video sequence (Jo.... 7.). Although we only introduce two schemes, our cross-input consistency framework is general-purposss and there may be other input-hiding schemes that offer comparable or better performance.

---

Tracker
Compare Outputs
Tracker B %
Figure 4: Ocelusion-based hiding produces imput variations with different subsequences of occluded frames where all detections are hidden from the tracker. It also independently applies the tracker before and after hand-off frame (74 and I2), and merges the outputs through the matrix product.
4.1 Visual-Spatial Hiding
Under visual-spatial hiding, we apply one tracker instance that observes only visual features and one tracker instance that observes only spatial features: in A(D), we set x = 0. 0, = 0, = 0 for all detections (hide all spatial information), and (D), we im 0 (hide the image content).
Training with cross-input consistency forces the model to produce similar outputs between the visual and spatial inputs. We find that, practice, the model naturally learns robustly track objects, because doing anything else would not lead to high consistency. For example, when given visual features, the model must learn match detections based visual similarity in order to consistent with matching based on spatial proximity. Similarly, when given spatial features, the model must learn to estimate motion across occlusion, since the visual-only instance would not have difficulty re-localizing a visually distinctive object following frames where it was occluded.
key issue with visual-spatial hiding is that the visual-only instance estimate the change in spatial coordinates of an object between two frames by comparing the background of the detection bounding boxes of that object across the frames, similar optical flow. This reduces performance because the visual-only instance then learns only to process the background rather than learn a robust embedding for contrasting distinct objects. To mitigate the issue, we make adjustments to the training and inference procedures.
Training. We prevent the visual-only instance from focusing on background features when matching detections in In with those in Io by encumbering its ability to aggregate estimates of changes in spatial coordinates across sequences of consecutive frames: although processing the background is sufficient to compute spatial movement between detections that are in close proximity (e.g., detections of the same object between consecutive frames), this strategy fails for detections that share no overlap. Thus, for sufficiently large where objects move substantially during the sampled video sequence, background processing is only an issue because the tracker can add up changes that it computes between each pair of consecutive frames.
In particular, we eliminate the recurrent unit for the visual-only instance: its matching network inputs visual features for detections in Iu and for detections in In. and scores each pair of detections in (0,n) without observing intermediate frames. On the other hand, we make no changes to the spatial-only instance: it computes B(0.n) by processing 4D spatial coordinates for each detection in every frame in the sequence, and employs both the recurrent unit and the matching network. Figure illustrates the training procedure.
The separation of visual and spatial inputs, and the specialized training procedure that we employ, imposes two challenges during inference. First, because the visual-only and spatial- only instances observe very different inputs, cannot expect the model to perform well when provide both inputs-in effect, we have trained two separate models. Second, since we eliminated recurrent features for the visual-only instance during training, the visual-only instance is essentially re-identification model, and we must decide how to apply it during inference to take advantage of multiple prior observations track in previous frames.
Inference.
To address the first challenge, during inference, for an input video sequence (Jo...., Ix), we indepen- dently compute the visual-only tracker output (0.8) and the spatial-only tracker output ((O.)) We then compute ((0k) as the minimum of (0.k) and B(0,k), and use (((k)) to update tracks before processing the next frame. Taking the minimum for matching between track f; and detection d

---

ensures that the final transition probability reflects whichever of the visual features or spatial features that make d less likely to match with t, This is desirable-for example, two red sedans visible in the same segment of video will have high visual similarity, and we may have to leverage spatial features to distinguish them.
The second challenge is that the visual-only tracker lacks recurrent features, and thus is only able to compare pairs of frames. To address this, when using the visual-only tracker, for each track-detection pair (ti, d,),, we compute 5 scores by applying the model on dÿ and 5 randomly selected images associated with the track t; in previous frames. We then average these scores to derive (0.k), This enables the model use context from multiple preceding frames when localizing an object in new frame.
4.2 Occlusion-based Hiding
We also experimented with an occlusion- based hiding scheme. Since we found that visual-spatial hiding performs better, we introduce occlusion-baac based hiding only a high level here, but include details in the supplementary material.
Occlusion-based hiding produces the variations A(D and B(D) by simulating random occlusion incidents where all detections in occluded frames are eliminated from the input, i.e., if /s is occluded for A(D), then D is empty. We only occlude intermediate frames (i.e., frame I& is only considered for occlusion if 0 < k < n)) so that the transition matrices still compare detections in To with those in occluded /A- the tracker is forced to match all tracks the absent column
n-- When processing in that frame, and re-localize the tracks after the occlusion. In occlusion-based hiding, we also incorporate an RNN hand-off method that cuts off the propagation of RNN features from Io In by employing two separate RNN executions: for some handoff index 1 < h < we apply the model from lo to In, and separately apply the model from J, to I,,, and combine the transition matrices by taking their product. We summarize the scheme in Figure
5 Evaluation
We compare our method and nine baselines on the MOT17 and KITTI 10 benchmarks.
Baselines. We compare with two unsupervised methods (SORT 2 and V-IOU 4 and seven fully supervised methods ( D. MHT-BLSTM [12). FAMNet (5) LSST g
GSM [8],
mmMOT 29] and CenterTrack (30)) Like our approach, SORT and V-IOU require object detector, but do not train on any video-level bounding box and track annotations in the MOT17 and KITTI training sets. The fully supervised methods train on video-level annotations; Tracktor+ incorporates core component that uses only the detector regression network, but requires video-level annotations for training a re-identification network. Results for 8 baselines are available on MOT17, and results for 4 baselines are available on KITTI.
Dataset. MOT17 21 consists of 14 video sequences of pedestrians in a wide range of contexts, including moving camera inside shopping mall and a fixed, elevated view of an outdoor plaza. The dataset is split into 7 training sequences and 7 test sequences; each split includes approximately 11 minutes of video. KITTI [10] consists of 48 video sequences captured from vehicle-mounted cameras, split into 20 for training and 28 for testing, and the objective is to track cars.
Training. The supervised baselines train on video-level bounding box and track annotations provided by MOT17 and KITTL. In contrast, method trains only corpus of unlabeled video. Because video-level annotations are expensive to label, our method requires substantially less annotation time, and thus greatly reduces the effort needed to apply multi-object tracking on new datasets.
For MOTI7, we collect unlabeled video from two sources: we use five hours of video from seven YouTube walking tours, and all train and test sequences from the PathTrack dataset (20] (we do not use the PathTrack ground truth annotations). For KITTI, we use both the 46 minutes of video the KITTI dataset together with 7 hours of video from Berkeley DeepDrive 227. We train our tracker model on an NVIDIA Tesla V100 GPU; training time varies berween and hours depending on the input-hiding scheme. During training, randomly select sequence lengths n between 4 and 16
8

---

Method
IDF1
MOTA
Occlusion (ours)
52.4
56.7
Visual-Spatial (ours)
57.3
60.2
Spatial-Only (ours)
56.5
57.8
Table Ablation study on the MOT17 training set.
Method
IDFT
MOTA
MT
MI.
FP
FN
Idsw
Frag
Unsupervised Methods
Visual-Spatial (ours)
58.3
56.8
538
880
12K
231K
1K
2K
SORT
39.8
43.1
295
997
28K
288K
5K
7K
IOU
3
39.4
45.5
369
953
20K
282K
6K
7K
Tracktor++
52.3
53.5
459
861
12K
248K
2K
5K
MHT-BLSTM
51.9
47.5
429
981
26K
268K
2K
3K
Supervised Methods
FAMNet 5
48.7
52.0
450
787
14K
254K
3K
5K
LSST 8
62.3
54.7
480
944
26K
228K
1K
4K
GSM
18
57.8
56.4
523
813
14K
230K
1K
3K
CenterTrack 30
59.6
61.5
621
752
14K
201K
3K
5K
Table Performance on the MOTI7 test set. compare methods terms of IDFI and MOTA, but include other non-comprehensive metrics from MOT17 as well for completeness.
Method
HOTA
DetA
AssA
DetRe
DetPr
AssRe
AssPr
LocA
SssutalSpatial (ours)
62.5
61.1
65.3
67.7
73.8
69.1
83.1
80.3
SORT
42.5
44.0
41.3
47.3
73.9
42.8
83.0
80.8
FAMNet 5
52.6
61.0
45.5
64.4
78.7
48.7
77.4
81.5
imMOT 29
62.1
72.3
54.0
76.2
84.9
59.0
82,4
86.6
CenterTrack 30
73.0
75.6
71.2
80.1
84.6
73.8
89.0
86.5
Table 3: Performance on the KITTI test set (tracking cars). We show unsupervised methods, including approach, the top, and methods that require video-level annotations at the bottom. We use HOTA to compare methods, but include other non-comprehensive metrics from KITTI for completeness.
frames, and apply stochastic gradient descent one sequence at a time. We apply the Adam optimizer with learning rate 0.0001, decaying to 0.00001 after plateau.
In contrast MOT17, KITTI does not provide object detections for use by tracking methods. We extract detections from video using YOLOv5 model trained on COCO. On MOT17, we pre-process the provided Deformable Parts Model, Faster R-CNN, and Scale-Dependent Pooling detections with classification and regression following the pre- pre-processing method in Tracktor++
Metrics. We Multi-Object Tracking Accuracy (MOTA) 21 and ID Fl Score (IDFI) 222 MOT17, and Higher Order Tracking Accuracy (HOTA) 19] for KITTI Broadly, these comprehensive metrics measure the accuracy of inferred tracks against ground truth tracks, and penalize both when an inferred track contains a detection that doesn' match to some ground truth detection (or vice versa), and when ground truth track is split into and KITTI employ several other non non-comprehensive metrics, many of which are used to compute MOTA, IDF1, and HOTA; we report these for completeness.
more inferred tracks (or vice versa). MOT17
Ablation Study. We first compare occlusion-bate based hiding and visual-spatial hiding on the MOT17 training set in Table Visual-spatial hiding yields higher performance on both MOTA and IDFI because objects are often visible in the video for only a short duration, when training under occlusion- based hiding, the model is unable to learn to re-localize objects over simulated occlusions since the simulated occlusion must then also be short. Under Spatial-Only we show results visual-spatial hiding when inputting only the spatial coordinates of detections during inference (no image features).

---

Figure 5: Output of Visual- -paaial on portion of an MOT17 sequence. Our method tracks objects through several instances of occlusion.
Quantitative Results. Table 2 shows results on the MOT17 test se and Table 3 shows results on the KITTI test se Metries are automatically computed by the challenge websites. Per the challenge policies, we only submit the best method, and thus show Visual-Spatial performance.
On MOT17, our approach substantially outperforms both of the unsupervised baselines. Moreover, despite training only on unlabeled video, our method outperforms Tracktor++ M. MHT-BLSTM, [22. FAMNet [5] and GSM [8], even though these baselines (all of which except MHT-BLSTM, were published the last 1-2 years) are supervised methods that train on expensive video-level annotations in the MOT17 7 training set. Our approach is also competitive with LSST ( yielding higher MOTA but lower IDF1. Nevertheless, CenterTrack [30 yields slightly higher accuracy on both metrics.
Similarly, on KITTI, our approach outperforms SORT (2). FAMNet [5], and mmMOT [29]. but yields lower performance than CenterTrack (30].
Qualitative Results. We show qualitative results in Figure 5
Additional Experiments. In the supplementary material, we report results for five additional experiments, where we compare MOTA on the MOT17 training when various experimental parameters are changed, including detector accuracy, unlabeled video corpus size, and the training example sequence length n.
Conclusion
In this paper, have shown that robust, fully unsupervised multi-object tracker can be trained through a novel self-supervisory learning signal, cross-input consistency, that enforces consistency in the tracking outputs across different input variations of one video sequence. Despite training only unlabeled video, our approach outperforms four supervised trackers published in the last 1-2 years (Tracktor++ [1, FAMNet [5), GSM (18), and mmMOT (2)). ) which train on expensive video-level bounding box and track annotations.
Social Impact. By enabling robust multi-object tracker to be trained given only unlabeled video, our work promises to greatly reduce the effort for users to apply multi-object tracking on new datasets without sacrificing accuracy. Thus, we believe that our novel self-supervised MOT method can open up new video analyties tasks that were previously too costly. This impact may positive or negative depending the nature of these tasks however, in general, we believe that tasks with greater potential for negative impact such as surveillance and pedestrian tracking would not benefit from the reduction in annotation cost associated with our method.
Funding Transparency Statement. This research was supported in part by the Qatar Computing Research Institute (QCRI).
'These results taken from http:///occccabllee.e.eeeeeeeees where method is denoted UNS20regress. Baselines are denoted SORTI7, Trakkot MHT bLSTM, FAMNet, LSST17, GSM_ Tracktor, and CTTrackPuh.
These results are taken http:///ww///wooo...///////////t
10

---

References
[1]
Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In IEEE International Conference Computer Vision (ICCV), 2019.
[2] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple Online and Realtime Tracking. In IEEE International Conference Image Processing (ICIP), 2016.
[3] Erik Bochinski, Volker Fiselein, and Thomas Sikora. High-Speed Tracking-by Without Using Image Information. In IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017.
Frackingby-Detecttion Detection
[4] Erik Bochinski, Tobias Senst, and Thomas Sikora. Extending IOU Based Multi-Objec Tracking by Visual Information. IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2018.
[5] Peng Chu and Haibin Ling. FAMNet: Joint Learning of Feature, Affinity and Multi-diimensionl Assignment for Online Multiple Object Tracking. In IEEE International Conference on Com- puter Vision (ICCV), 2019.
[6] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised Visual Representation Learning by Context Prediction. In IEEE International Conference Computer Vision (ICCV), 2015.
[7] Carl Doersch and Andrew Zisserman. Multi-task Self- Superviie Visual Learning. In IEEE International Conference on Computer Vision (ICCV), 2017.
[8] Weitao Feng, Zhihao Hu, Wei Wu, Junjie Yan, and Wanli Ouyang. Multi-object tracking with multiple cues and switcher-aware classification. arXiv preprint arXiv: 000669 2019.
[9] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? The KITTI Vision Benchmark Suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[11] Shyamgopal Karthik, Ameya Prabhu, and Vineet Gandhi. Simple unsupervised multi-object tracking. CoRR, foss2006.02609, 2020.
[12] Chanho Kim, Fuxin Li, and James M. Rehg. Multi-Objeet Tracking with Neural Gating using Bilinear LSTM. In European Conference on Computer Vision (ECCV), 2018.
[13] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A nemory-augmenteed self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[14] Laura Leal-Taixé, Anton Milan, Ian Reid, Stefan Roth, and Konrad Schindler. MOTChallenge 2015: Towards Benchmark for Multi-Target Tracking. arXiv preprint 2015.
[15] Hsin- Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), Oct 2017.
[16] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuar Yang. Joint-task self- self-supervised learning for temporal correspondence. In Advances in Neural Infor- mation Processing Systems, volume 32, 2019.
[17] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects Context. In European Conference on Computer Vision (ECCV), 2014.
11

---

[18] Qiankun Liu, Qi Chu, Bin Liu, and Nenghai Yu. GSM: Graph Similarity Model for Multi Objeet Tracking. In International Joint Conference on Artificial Intelligence (LJCAI), 2020.
[19] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Tor, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking. International Journal of Computer Vision, 129(2):548-5)) 578, 2021.
[20] Santiago Manen, Michael Gygli, Dengxin Dai, and Luc Van Gool. PrthTrack: Fast Trajectory Annotation with Path Supervision. In IEEE International Conference Computer Vision (ICCV), 2017.
[21] Anton Milan, Laura Leal- Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. MOTI6: A Benchmark for Multi-Object Tracking, arXiv preprint 1Xiv:1603300033 2016.
[22] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance Measures and Data Set for Multi-Targee, Multi-Camera Tracking. In European Conference on Computer Vision (ECCV), 2016.
[23] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised Learning of Video Representations using LSTMs. In International Conference Machine Learning (ICML), 2015.
[24] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking Emerges by Colorizing Videos. In European Conference Computer Vision (ECCV), 2018.
[25] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning Correspondence from the Cycle- Consistency of Time. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[26] Donglai Wei, Joseph J. Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018
[27] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[28] Jenny Yuen, Bryan Russell, Ce Liu, and Antonio Torralba. LabelMe video: Building a Video Database with Human Annotations, In IEEE International Conference on Computer Vision (ICCV), 2009.
[29] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, and Chen Change Loy. Robust multi-modality multi-object tracking. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
[30] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking Objects as Points. In European Conference Computer Vision (ECCV), 2020.
12


Reviews:
[{"review_id": "mj01YzEWtE", "reviewer": "Reviewer_thph", "summary": "This paper presents a method for self-supervised multi-object tracking in videos. The authors propose a cross-input consistency loss to train their RNN-based tracking method to form correct associations between detected objects in the various video frames and also to relocate them when tracks are lost. The core idea of the cross-input consistency loss is to present two different (altered) versions of the same video sequence to the tracking algorithm and to ensure that despite differences in the exact sequences the algorithm is able to identically associate the detections in the first and last frames of the two sequences. The authors present various methods to alter the sequences so as to avoid degenerate solutions from being learned. Their method shows significant improvements over the existing unsupervised tracking algorithms and are competitive with several existing SoTA supervised approaches on two benchmark datasets -- MOTA17 and KITTI.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper does not propose a completely novel task, but rather the first learning-based algorithm for the task of self-supervised video multi-object tracking. Little research has been done for this task before so in that regard this work represents the first dedicated and systematic attempt at trying to solve this problem and hence is sufficiently novel.\n\nQuality: Overall the proposed approach is well-described and sufficiently motivated. However, I have some major concerns about some parts of the proposed approach. I describe them below.\n\n1. The biggest concern that I have is that from line 316 and Table 2 it is clear that in their final algorithm the authors employ the \"Visual-Spatial Hiding\" only. Why then is the description of occlusion-hiding (section 4.1) even included in the method section paper since it is never used in the final method? This largely undermines a significant portion of the proposed method and I feel that to some extent the authors have mis-represented their proposed method.\n\n2. The authors make the speculative claim several times that the existing colorization [24] and cycle-based [25] techniques for learning with RNNs \"would not work\" -- in the abstract and lines 38-39. However, this claim is not supported by the experiments shown in the paper. I believe that the \"Combined Occlusion Scheme\" proposed by the authors along with a forward and backward cycle loss from [25] could be applied to train the RNN while ensuring that it does not just memorize the transitions between frame 0 and $n$ only and to force it to learn the transitions in between in the sequence as well. Anyhow, the authors should either verify this claim with experimental evidence or remove it entirely from the paper. \n\n3. In lines 170-179 the authors describe that the success of their algorithm depends on the assumption that most objects presents in frame $0$ should also be present in frame $n$. They further mention that this critical parameter $n$ can be set as \"one-half of the average time that objects linger in the camera frame\". However, this requires knowledge of labeled tracks in videos. Can the authors comment on this limitation of their algorithm? How critical is this parameter to the overall success of their unsupervised learning algorithm? Lines 308-312 seem to indicate that the proposed algorithm is indeed quite sensitive to this parameter. \n\nHave the authors systematically analyzed its sensitively for a particular dataset and across datasets? Related to this, have the authors studied if the availability of longer tracks in general results in better overall performance for their algorithm?\n\n4. Have the authors analysed if domain differences between the labeled and unlabelled datasets affect the accuracy of their approach and ways to mitigate these?\n\nClarity: The material presented is mostly clear. However, some parts of the method are not explained clearly:\n\n1. \"When processing an occluded $I_k$, the tracker is forced to match all tracks to the absent column in that frame, and re-localize the tracks after the occlusion.\" Is this something that is enforced by design or something that emerges automatically in the transition matrix probabilities during training? How does the subsequent re-localization happen, when all tracks get assigned to a single \"absent\" detection? Are all detections in a subsequence of frames hidden or just a few of them? This part is not very clear and it would be helpful to explain this more clearly by illustrating transition matrices and how they change in this case.\n\n2. line 295: When $n$ is set to 4 during training, there are only 3 frames in between the first and last frame. How to the authors set $k1$, $k2$, $k3$, $k4$, $k5$ and $k6$ in this case?\n\n3. In the tables it would be good to add (up/down) arrows to indicate the direction of improvement of the metric.\n\n4. Why are the results of the proposed algorithm (Visual-spatial) different in Tables 1 and Table 2?\n\n5. For the experiments on MOT17, how were the frame-wise detections obtained? \n\nSignificance: Obtaining reliable video-level annotations is a real practical problem while deploying video-based tracking algorithms, which limits their applicability currently. This paper addresses this existing open problem, which has much practical importance. The results of the proposed method also present a significant improvement over the existing unsupervised methods and hence this work advances the research in this field. However, I have several concerns about the presented method and how it was implemented, which I would like to hear more clarification about from the authors.\n\nPost rebuttal: The authors have adequately addressed my concerns. The paper addresses an important problem. I recommend acceptance and have raised my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the (positive and negative) societal impact of their work.\n\nThey should include more discussion of the $n$ parameter used in their algorithm and how it should be selected in real-world settings without video annotations.\n\nThey should remove the discussion of the occlusion-hiding section from the proposed method as it is never used in their final method and is mis-leading.\n\nThey should also remove speculative and unsupported claims about [24] and [25] not being able to work with RNNs.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DqZ9QdZCJiM", "reviewer": "Reviewer_4Dyg", "summary": "This paper proposes a self-supervised learning scheme for training Multi-Object Tracking (MOT) models using only unlabeled video. The proposed scheme constructs two distinct inputs by hiding different information about the sequence in each input and trains the model to produce consistent tracks across the two inputs. The proposed algorithm is evaluated on the MOT17 and KITTI datasets with comparisons against several supervised methods. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty and Contributions:\nThe idea of using the cross-input consistency, which trains the model to produce consistent tracks on two inputs with different hiding information, to train MOT models on unlabeled video is novel.\nIn addition, the authors provide two input-hiding schemes (Occlusion-based hiding and visual-spatial hiding) for generating input variations. \nExploiting self-supervised learning signals to train MOT models on videos without annotations is meaningful.\n\nHowever, I have the following concerns.\n-\tLimitations. Although the proposed method can be trained on unlabeled videos, it requires an object detector for the object category of interest, to provide object bounding boxes. The proposed algorithm only exploits the self-supervised learning scheme to learn the data association module for matching tracks and detections. It can be only applied in tracking-by-detection MOT methods, which cannot be trained end-to-end. In addition, since the detection part also belongs to the MOT framework, it may be inappropriate to call the proposed MOT tracker a fully unsupervised one.  \n-\tLine 178 writes ‘in general, we find that setting n to one-half of the average time that objects linger in the camera frame works well.’. But how to find the average time on large-scale training videos is not given. This assumption also limits the selection of training data and the diversity of training samples.\n\nExperimental results:\nThe experimental results are good. However, I have the following concerns.\n-\tLine 275 writes ‘Like our approach, SORT and V-IOU require an object 276 detector,’. Since the authors do not state clearly the test settings on the MOT17 dataset, I am wondering whether the public detections or the private detections (generated by the yolov5 detector) are used?\n-\tFor the ablation study, adding a baseline that directly performs tracking without training may be helpful, since the features of the same object are usually similar. The comparison against this baseline will better show how much the unsupervised learning scheme contributes to the final performance.\n\nClarity:\nThis paper is not well organized. \nIt takes more than 1 page (from Line 216 to Line 259) to describe the Occlusion-based hiding and uses a 1/4 page to introduce the visual-spatial hiding. The results (in Table 1) show that the later scheme achieves better performance. Line 310 writes that ‘when training under occlusion based hiding, the model is unable to learn to re-localize objects over simulated occlusions since the simulated occlusion must then also be short.’, which indicates that the former occlusion-based hiding scheme does not work well.\nFinally, the authors use the visual-spatial (later) scheme to compare with other methods.\nThis is really confusing. If the authors know the occlusion-based hiding scheme does not work well, then why spend more than 1 page to describe it. In addition, some details about the visual-spatial scheme are missing. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BzeBzUTXWl", "reviewer": "Reviewer_fDwk", "summary": "This paper presents a novel framework to train the multi-object tracker in an unsupervised scheme. The framework utilizes the cross-input consistency assuming that the two clips manipulated by one video must have the same tracking trajectory. The two manipulated clips are randomly generated by one of the occlusion-based hiding and visual-spatial hiding. By using the baseline of the RNN-based tracker, the proposed framework shows comparable performance even with the supervised MOT.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The framework seems novel and its effectiveness is also impressive. In addition, a detailed description is given as well sufficiently to reproduce the overall framework. The definition and notations were clear to understand the implementation.\n\nI have several questions and comments.\n\n- The relation between the number of data and the accuracy\n  Since the proposed framework can utilize infinite data to train the tracking model, the performance can be improved when more variety of videos are trained. I hope to know when the performance improvement converges and why.\n\n- Full clip instead of the manipulated clip?\n  The proposed framework utilizes two manipulated clips. However, when one of the two clips is replaced by a full video clip that suffers from no manipulation, the training would be more stable. Since the tracking performance in the full video clip would be the upper bound of the performance in the manipulated video clip, we may be able to acquire the increased performance. What is the reason why both two video clips should be manipulated? \n\n- Sensitivity to the noisy detector\n  This framework basically assumes that the detection box is reliable. Then, what would be happened when the noisy detection is given in the self-training phase? Since all the experimental results utilized one specific detector, we cannot find the sensitivity to the noisy detection. I recommend the authors show some results with the different detectors or the noisy detection boxes.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please read the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "B8upm6d0QPV", "reviewer": "Reviewer_rjLZ", "summary": "This paper proposes a self-supervised learning procedure for training a robust MOT model given only unlabeled video. This unsupervised method outperforms several recent supervised methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The submission proposes a new self-supervised architecture for multi-object tracking, which is novel.\n\nQuality: The submission is technically sound. The author submits the code, which is convenient to reproduce.\n\nClarity: The written is not so clear.\n\nSignificance: The performance improvement is general compared to the existing methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The two input is the objects detected in the first and last frames of the sequence. What if the object disappears or is occluded before the last frame？\n2. The written need be improved and the figures should be more clear to explain the method. The related work should be described by different classes. It is best not to put Figure 1 and Figure 2 in the related work. In Figure 1, I suggest using real pictures instead of synthetic pictures to display the cross-input consistency. Input A and Input B is the different input while using the same image here.\n3. The performance of Visual-Spatial is much better than SORT and IOU. But these two methods is too old (published in 2016 and 2019). Several more recent methods (unsupervised/semi supervised/supervised) should be added and discussed, such as Chained-Tracker [1] and QDTrack [2].\n4. This paper focues on the training process, which is lack of the detailed description of the test process. Besides, the time-consuming (fps) of each module should be analyzed and compared with other methods.\n\n[1] Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking. ECCV, 2020.\n[2] Quasi-Dense Similarity Learning for Multiple Object Tracking. CVPR, 2021.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received mixed reviews, two for (6 & 7) and two against (4 & 5).  The reviewers appreciated the motivation and novel self-supervised learning method proposed for MOT, as well as the superior results against previous unsupervised methods, as well as many supervised methods.  The main concerns from the reviewers were: 

1. missing discussion about choice/effect of hyperparameters (n).
2. "occlusion-based hiding" is proposed (and a lot of space is dedicated to describing it), but performs worse than the proposed "visual-spatial hiding".
3. some claims not supported
4. still requires a trained detector to work, which can be considered as not fully unsupervised MOT.
5. missing ablation studies and baselines.
6. use more recent comparison methods for MOT, Chained-Tracker and QDTrack (published after NeurIPS deadline).

The author provided a response, and attempted to address these points through further explanations and promises to rewrite/expand parts of the paper. Some ablation studies were already presented in the supplemental. After the discussion one positive reviewer raised from 6 to 7, while the other positive reviewer maintained 7. One negative reviewer was concerned that a large amount of changes are required to address the review comments, and thus maintained rating 4.  The other negative reviewer did not participate in the discussion. The AC checked the response to this reviewer's comments, and thinks that the responses to their comments were satisfactory.  In the end the AC has a positive impression on the paper, and thus recommends acceptance, under the condition that the presentation is improved. 

The authors should revise the paper according to the reviews and discussion, in particular: reduce the discussion on occlusion-based hiding, increase discussion on visual-spatial hiding, include the ablation studies from the supplemental, add baselines (tracking w/o training), expand discussion about parameter n. 
Decision: Accept (Poster)