Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for self-supervised multi-object tracking in videos. The authors propose a cross-input consistency loss to train their RNN-based tracking method to form correct associations between detected objects in the various video frames and also to relocate them when tracks are lost. The core idea of the cross-input consistency loss is to present two different (altered) versions of the same video sequence to the tracking algorithm and to ensure that despite differences in the exact sequences the algorithm is able to identically associate the detections in the first and last frames of the two sequences. The authors present various methods to alter the sequences so as to avoid degenerate solutions from being learned. Their method shows significant improvements over the existing unsupervised tracking algorithms and are competitive with several existing SoTA supervised approaches on two benchmark datasets -- MOTA17 and KITTI.
main_review: Originality: The paper does not propose a completely novel task, but rather the first learning-based algorithm for the task of self-supervised video multi-object tracking. Little research has been done for this task before so in that regard this work represents the first dedicated and systematic attempt at trying to solve this problem and hence is sufficiently novel.

Quality: Overall the proposed approach is well-described and sufficiently motivated. However, I have some major concerns about some parts of the proposed approach. I describe them below.

1. The biggest concern that I have is that from line 316 and Table 2 it is clear that in their final algorithm the authors employ the "Visual-Spatial Hiding" only. Why then is the description of occlusion-hiding (section 4.1) even included in the method section paper since it is never used in the final method? This largely undermines a significant portion of the proposed method and I feel that to some extent the authors have mis-represented their proposed method.

2. The authors make the speculative claim several times that the existing colorization [24] and cycle-based [25] techniques for learning with RNNs "would not work" -- in the abstract and lines 38-39. However, this claim is not supported by the experiments shown in the paper. I believe that the "Combined Occlusion Scheme" proposed by the authors along with a forward and backward cycle loss from [25] could be applied to train the RNN while ensuring that it does not just memorize the transitions between frame 0 and $n$ only and to force it to learn the transitions in between in the sequence as well. Anyhow, the authors should either verify this claim with experimental evidence or remove it entirely from the paper. 

3. In lines 170-179 the authors describe that the success of their algorithm depends on the assumption that most objects presents in frame $0$ should also be present in frame $n$. They further mention that this critical parameter $n$ can be set as "one-half of the average time that objects linger in the camera frame". However, this requires knowledge of labeled tracks in videos. Can the authors comment on this limitation of their algorithm? How critical is this parameter to the overall success of their unsupervised learning algorithm? Lines 308-312 seem to indicate that the proposed algorithm is indeed quite sensitive to this parameter. 

Have the authors systematically analyzed its sensitively for a particular dataset and across datasets? Related to this, have the authors studied if the availability of longer tracks in general results in better overall performance for their algorithm?

4. Have the authors analysed if domain differences between the labeled and unlabelled datasets affect the accuracy of their approach and ways to mitigate these?

Clarity: The material presented is mostly clear. However, some parts of the method are not explained clearly:

1. "When processing an occluded $I_k$, the tracker is forced to match all tracks to the absent column in that frame, and re-localize the tracks after the occlusion." Is this something that is enforced by design or something that emerges automatically in the transition matrix probabilities during training? How does the subsequent re-localization happen, when all tracks get assigned to a single "absent" detection? Are all detections in a subsequence of frames hidden or just a few of them? This part is not very clear and it would be helpful to explain this more clearly by illustrating transition matrices and how they change in this case.

2. line 295: When $n$ is set to 4 during training, there are only 3 frames in between the first and last frame. How to the authors set $k1$, $k2$, $k3$, $k4$, $k5$ and $k6$ in this case?

3. In the tables it would be good to add (up/down) arrows to indicate the direction of improvement of the metric.

4. Why are the results of the proposed algorithm (Visual-spatial) different in Tables 1 and Table 2?

5. For the experiments on MOT17, how were the frame-wise detections obtained? 

Significance: Obtaining reliable video-level annotations is a real practical problem while deploying video-based tracking algorithms, which limits their applicability currently. This paper addresses this existing open problem, which has much practical importance. The results of the proposed method also present a significant improvement over the existing unsupervised methods and hence this work advances the research in this field. However, I have several concerns about the presented method and how it was implemented, which I would like to hear more clarification about from the authors.

Post rebuttal: The authors have adequately addressed my concerns. The paper addresses an important problem. I recommend acceptance and have raised my score.
limitations_and_societal_impact: The authors have addressed the (positive and negative) societal impact of their work.

They should include more discussion of the $n$ parameter used in their algorithm and how it should be selected in real-world settings without video annotations.

They should remove the discussion of the occlusion-hiding section from the proposed method as it is never used in their final method and is mis-leading.

They should also remove speculative and unsupported claims about [24] and [25] not being able to work with RNNs.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel framework to train the multi-object tracker in an unsupervised scheme. The framework utilizes the cross-input consistency assuming that the two clips manipulated by one video must have the same tracking trajectory. The two manipulated clips are randomly generated by one of the occlusion-based hiding and visual-spatial hiding. By using the baseline of the RNN-based tracker, the proposed framework shows comparable performance even with the supervised MOT.
main_review: The framework seems novel and its effectiveness is also impressive. In addition, a detailed description is given as well sufficiently to reproduce the overall framework. The definition and notations were clear to understand the implementation.

I have several questions and comments.

- The relation between the number of data and the accuracy
  Since the proposed framework can utilize infinite data to train the tracking model, the performance can be improved when more variety of videos are trained. I hope to know when the performance improvement converges and why.

- Full clip instead of the manipulated clip?
  The proposed framework utilizes two manipulated clips. However, when one of the two clips is replaced by a full video clip that suffers from no manipulation, the training would be more stable. Since the tracking performance in the full video clip would be the upper bound of the performance in the manipulated video clip, we may be able to acquire the increased performance. What is the reason why both two video clips should be manipulated? 

- Sensitivity to the noisy detector
  This framework basically assumes that the detection box is reliable. Then, what would be happened when the noisy detection is given in the self-training phase? Since all the experimental results utilized one specific detector, we cannot find the sensitivity to the noisy detection. I recommend the authors show some results with the different detectors or the noisy detection boxes.
limitations_and_societal_impact: Please read the main review.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a self-supervised learning scheme for training Multi-Object Tracking (MOT) models using only unlabeled video. The proposed scheme constructs two distinct inputs by hiding different information about the sequence in each input and trains the model to produce consistent tracks across the two inputs. The proposed algorithm is evaluated on the MOT17 and KITTI datasets with comparisons against several supervised methods. 
main_review: Novelty and Contributions:
The idea of using the cross-input consistency, which trains the model to produce consistent tracks on two inputs with different hiding information, to train MOT models on unlabeled video is novel.
In addition, the authors provide two input-hiding schemes (Occlusion-based hiding and visual-spatial hiding) for generating input variations. 
Exploiting self-supervised learning signals to train MOT models on videos without annotations is meaningful.

However, I have the following concerns.
-	Limitations. Although the proposed method can be trained on unlabeled videos, it requires an object detector for the object category of interest, to provide object bounding boxes. The proposed algorithm only exploits the self-supervised learning scheme to learn the data association module for matching tracks and detections. It can be only applied in tracking-by-detection MOT methods, which cannot be trained end-to-end. In addition, since the detection part also belongs to the MOT framework, it may be inappropriate to call the proposed MOT tracker a fully unsupervised one.  
-	Line 178 writes ‘in general, we find that setting n to one-half of the average time that objects linger in the camera frame works well.’. But how to find the average time on large-scale training videos is not given. This assumption also limits the selection of training data and the diversity of training samples.

Experimental results:
The experimental results are good. However, I have the following concerns.
-	Line 275 writes ‘Like our approach, SORT and V-IOU require an object 276 detector,’. Since the authors do not state clearly the test settings on the MOT17 dataset, I am wondering whether the public detections or the private detections (generated by the yolov5 detector) are used?
-	For the ablation study, adding a baseline that directly performs tracking without training may be helpful, since the features of the same object are usually similar. The comparison against this baseline will better show how much the unsupervised learning scheme contributes to the final performance.

Clarity:
This paper is not well organized. 
It takes more than 1 page (from Line 216 to Line 259) to describe the Occlusion-based hiding and uses a 1/4 page to introduce the visual-spatial hiding. The results (in Table 1) show that the later scheme achieves better performance. Line 310 writes that ‘when training under occlusion based hiding, the model is unable to learn to re-localize objects over simulated occlusions since the simulated occlusion must then also be short.’, which indicates that the former occlusion-based hiding scheme does not work well.
Finally, the authors use the visual-spatial (later) scheme to compare with other methods.
This is really confusing. If the authors know the occlusion-based hiding scheme does not work well, then why spend more than 1 page to describe it. In addition, some details about the visual-spatial scheme are missing. 




limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a self-supervised learning procedure for training a robust MOT model given only unlabeled video. This unsupervised method outperforms several recent supervised methods.
main_review: Originality: The submission proposes a new self-supervised architecture for multi-object tracking, which is novel.

Quality: The submission is technically sound. The author submits the code, which is convenient to reproduce.

Clarity: The written is not so clear.

Significance: The performance improvement is general compared to the existing methods.
limitations_and_societal_impact: 1. The two input is the objects detected in the first and last frames of the sequence. What if the object disappears or is occluded before the last frame？
2. The written need be improved and the figures should be more clear to explain the method. The related work should be described by different classes. It is best not to put Figure 1 and Figure 2 in the related work. In Figure 1, I suggest using real pictures instead of synthetic pictures to display the cross-input consistency. Input A and Input B is the different input while using the same image here.
3. The performance of Visual-Spatial is much better than SORT and IOU. But these two methods is too old (published in 2016 and 2019). Several more recent methods (unsupervised/semi supervised/supervised) should be added and discussed, such as Chained-Tracker [1] and QDTrack [2].
4. This paper focues on the training process, which is lack of the detailed description of the test process. Besides, the time-consuming (fps) of each module should be analyzed and compared with other methods.

[1] Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking. ECCV, 2020.
[2] Quasi-Dense Similarity Learning for Multiple Object Tracking. CVPR, 2021.
review_title: null

Meta-review: The paper initially received mixed reviews, two for (6 & 7) and two against (4 & 5).  The reviewers appreciated the motivation and novel self-supervised learning method proposed for MOT, as well as the superior results against previous unsupervised methods, as well as many supervised methods.  The main concerns from the reviewers were: 

1. missing discussion about choice/effect of hyperparameters (n).
2. "occlusion-based hiding" is proposed (and a lot of space is dedicated to describing it), but performs worse than the proposed "visual-spatial hiding".
3. some claims not supported
4. still requires a trained detector to work, which can be considered as not fully unsupervised MOT.
5. missing ablation studies and baselines.
6. use more recent comparison methods for MOT, Chained-Tracker and QDTrack (published after NeurIPS deadline).

The author provided a response, and attempted to address these points through further explanations and promises to rewrite/expand parts of the paper. Some ablation studies were already presented in the supplemental. After the discussion one positive reviewer raised from 6 to 7, while the other positive reviewer maintained 7. One negative reviewer was concerned that a large amount of changes are required to address the review comments, and thus maintained rating 4.  The other negative reviewer did not participate in the discussion. The AC checked the response to this reviewer's comments, and thinks that the responses to their comments were satisfactory.  In the end the AC has a positive impression on the paper, and thus recommends acceptance, under the condition that the presentation is improved. 

The authors should revise the paper according to the reviews and discussion, in particular: reduce the discussion on occlusion-based hiding, increase discussion on visual-spatial hiding, include the ablation studies from the supplemental, add baselines (tracking w/o training), expand discussion about parameter n. 
Decision: Accept (Poster)