Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes using a unified self-supervised framework for various video-related tracking tasks including single-object bounding box (SOT) or mask propagation (VOS) or multiple object bounding box (MOT), mask (MOTS) or keypoints (PoseTrack) association of detections across video frames. The idea proposed in this paper is inspired by the finding that mid-level representations of networks trained either with full-supervision (e.g., for object classification) or via self-supervised learning for a pre-text task are believed to contain information for establishing dense correspondence between objects and instances. The authors propose to use a single backbone pre-trained network to extract fc3/fc4 features for these various tasks and further employ methods based on dense affinity/ discriminative correlation filters/Hungarian matrix, etc to perform the various propagation or association tasks. The main finding that the authors share is that a common task-agnostic mid-level representation learned using ImageNet is competitive with many supervised and unsupervised task-specific algorithms designed for individual tasks.
main_review: Originality: The paper is original in several aspects: -- (a) in investigating a large number (5) of different but closely related tracking tasks together in a unified manner, (b) in highlighting the competitiveness of a single representation learned using ImageNet for many video-related tasks and proposing ways to use them effectively for the different tasks, (c) in analyzing for the first time as to how well self-supervised representations work at these tasks and what their shortcomings are. In summary, this paper proposes a new task, a new research paradigm and a new algorithm.

Quality: Overall the method is technically sound, well-motivated and the experiments are thorough. However, I note some concerns below.

1. With regards to unsupervised learning approaches for association-type tasks there exists the following recently published work, where the authors train a unified network for MOTS and PoseTrack: "Fu et al., Learning to Track Instances without Video Annotations, CVPR 2021". It is not cited. It would be good to cite this work in the reference section and compare against it in Table 1(e) of the main paper and Table 5 of the supplementary material.

2. How critical is the modification of the stride of the original ImageNet18 to the performance of UniTrack? Did the authors also apply this modification to all the networks of the competing methods that they compared against? If not, for a fairer comparison, could the authors also provide the equivalent performance numbers for UniTrack (in Table 1), but without this stride modification?

Clarity: The paper is quite clear and easy to understand. The supplementary document is extensive and provides detailed implementation steps. A few typos in the main document are noted below:

- ln 26: "form" should be "forming"
- ln 27: "differ" should be "differing"
- ln 207: "calculate" should be "calculating"
- ln 263: "/" should be "."

Significance: This work is likely to have a broad impact across the computer vision research community and is likely to be of interest to a wide audience. The fact that it proposes to unify networks for many different, but related video tracking tasks is also quite nice and influential from the point of view of practitioners of computer vision technology as well. It encourages simplification of network design and hence reduction in inference compute cycles, which is an important consideration in real-world applications.
limitations_and_societal_impact: The authors addressed the limitations of their work in the broader societal context.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a unified visual tracking method using a single and task-agnostic appearance model, in which this model can be learned via n a supervised or self-supervised strategy. For this purpose, they examine appearance model performance on five tracking tasks and describe it using radar charts in fig1. Experimental results show that  most tracking tasks can be solved within the proposed framework.

main_review: The paper considered a unified framework for the appearance model. However it is unclear why the method can contribute improve the visual tracking performance. While visual tracking environments considerably vary over time, visual tracking method needs to be specialized to a current visual tracking environment, rather than using a unified appearance model. 
limitations_and_societal_impact: All compared methods were not pre-traineded using ImageNet, thus comparisons can be unfair. 
The experimental evaluations are insufficient to demonstrate the effectiveness of the proposed unified appearance model under various visual tracking environments. The method needs to be evaluated using single target tracking datasets including OTB, VOT, LaSOT, OxUvA, TLP, TrackingNet, UAV123, TC, and GOT-10k. The compared methods are not recent state-of-the-art methods. More recent methods need be compared. It needs to be proven theoretically that the proposed appearance modeling strategies can cover most changes in target appearance and adapt to change the appearance under a current visual tracking environment.  It would be better to discuss and compare the proposed method against the fusion method of multiple appearance models. The paper also needs to explain why box, mask, and pose representations are considered.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper investigates the possibility of using a single appearance model for all tracking-related tasks/subtasks (the paper proposes a precise classification of these tasks). The authors propose an elegant unified framework with 3 levels: basic appearance model (level 1), propagation and association blocks (level 2), and task-specific heads (level 3). The only level that needs a training phase is level 1, while the other levels are training-free. Another contribution is the introduction of the RSM metric, which is used to measure the similarity between features within the proposed association algorithm. Following a rich experimental section, the authors conclude by giving a negative answer to the question contained in the title of the paper itself.
main_review: -**Originality**: The authors propose a method with some innovative elements and an interesting combination of well-known techniques to tackle the problem of object tracking in videos. The main innovative contribution, in my opinion, is the RSM metric, which is used to measure the similarity between features within the proposed association algorithm. The presented framework, as a whole, is interesting, although it takes many techniques and ideas from pre-existing works (which are correctly cited by the authors). Overall, I believe that the originality level of the paper is not high, but still acceptable.  
-**Quality**: Technically, the paper is of good quality. The claims of the paper are well supported by the theoretical treatment, but they lose some points due to the experimental results that, in my opinion, are not always amazing. However, the authors highlight the weaknesses of their approach and explain the reasons behind the most underwhelming results. The paper therefore shows how the use of a different appearance model for each tracking subtask is not strictly necessary to have competitive results; but in my opinion it also highlights how a specific approach may in some cases be necessary to obtain SoTA results.  
-**Clarity**: The paper is written in a clear and well-organized way. From reading the paper, you have all the information you need to re-implement the method. Furthermore, the supplementary material further enriches the discussion by making the paper even clearer. My only note in this sense is related to the figures: some of them, in particular Figure 3, would in my opinion have benefited from more space and a greater amount of detail; relegating the overall scheme of the framework to a small figure with few details can be a bit limiting (the details are however contained in the paper body).  
-**Significance**: Some elements that the paper proposes could be reused by the community, such as the RSM metric and the categorization of the various tracking-related subtasks. The framework as a whole can also have some relevance when used as a baseline for comparison with other methods, particularly for methods that use a specific approach to a given subtask. However, it does not seem to me that the conclusions of the paper are particularly significant, because the experimental results (that are not always excellent) make the authors' main claim a little bit weak.  
-**Overall Opinion**: The paper is well done and contains some interesting elements. Not particularly groundbreaking, but acceptable. Given the experimental results, I would propose the framework just as a baseline for task-specific methods, because I don't think the answer to the question that gives the paper its title is a definitive “no”.  
limitations_and_societal_impact: Not applicable, as the paper does not have a significant social impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper categorizes tracking object tasks into five lines and divides tracking algorithms into two classes: propagation and association. It indicates no necessity to fragment the problem of tracking in so many different specifications and separate trained models for each one. A unified tracking solution is proposed to address five different tasks using the same appearance model, which is task-agnostic and can be learned in a supervised or self-supervised way. The proposed model contains multiple heads to address individual tasks without training, and a reconstruction-based similarity metric is designed for association that preserves fine-grained visual features and supports multiple observation formats (box, mask and pose). I feel this work is interesting and has strong contributions to the tracking community, but experimental results are somewhat weak.
main_review: Strengths

-- The method is novel. It should be the first simple and universal framework that simultaneously supports five tracking tasks (SOT, VOS, MOT, MOTS, PoseTrack), and can also be easily extended to new ones. 

-- Compared with traditional cosine similarity methods, which do not support varying spatial sizes, this work proposes a reconstruction-based similarity metric that is able to effectively deal with different observation formats while preserving fine-grained information.

-- The framework has achieved competitive performance against dedicated methods on the five tasks of SOT, VOS, MOT, MOTS and PoseTrack. Moreover, unlike existing self-supervised learning methods that require additional training, the method proposed in this paper does not require additional training for different tasks.

Weaknesses

-- This framework does not train an appearance model on a single target task, nor does it apply multi-task learning on multiple tasks. The complementary benefits of different tracking tasks are not explored and exploited, since these tracking tasks have some common properties that can be utilized for the enhancement to each other. It would be interesting if authors perform shared representation learning across different tasks and fine-tune task heads to consider their specialty, although domain difference and optimization object conflict might exist.

--UniTrack performs comparable with SOTA methods on MOT, MOTS and poseTrack, but greatly worse on SOT and VOS. Moreover, some settings make comparison unconvincing. For example, in the task of SOT, comparison is only made on a small-scale and somewhat out-of-date dataset and recent SOTA methods are missing. 

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: Initially the paper received mixed positive review: three accept (7) and 1 reject (3).  The reviewers appreciated the originality (unified framework for 5 tracking tasks, new similarity measurements and association algorithm, evaluation of supervised and self-supervised representations on 5 tracking tasks).  The reviewers' main concerns were: 
1. additional results with new unsupervised methods.
2. some writing/clarity issues
3. didn't use multi-task learning to train the framework.
4. comparison on SOT uses out-of-date dataset and older methods. need further evaluation on more recent datasets.
5. for tracking model, appearance model needs to be specialized for the current environment, rather than a unified appearance model.
6. missing theoretical analysis that appearance models can cover most changes in target appearance.
7. possible unfair comparisons since some methods not pre-trained on imagenet.

In the response, authors provided more experiment results (point 4: SOT results on 5 larger datasets; point 1: another SSL method) and further clarifications and explanations (points 2, 3, 5, 6, 7). The 3 positive reviewers were satisfied with the response, while the negative reviewer was only partially satisfied and raised their rating to "4".  The remaining concern of the negative reviewer was mainly about the theoretical analysis (point 6).  The AC tends to discount this concern, since theoretical analysis seems outside of the scope of this paper, given the large amount of empirical results. The AC agrees with the 3 positive reviewers that the paper provides an interesting new way to unify tracking tasks, and to evaluate representation learning methods on tracking tasks. Thus the AC recommends accept. The authors should update the paper according to the review comments and discussion.
Decision: Accept (Poster)