Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a tracking algorithm by designing a ModaMixer and asymmetrical networks to learn unified-adaptive VL representation. The combination of the two modules gives very good results. Applying language features to tracking tasks is a very reasonable and innovative multimodal learning algorithm.
strengths_and_weaknesses: Strengths
•	The paper is clearly written.
•	Results greatly improve the performance and seems achieve SOTA.
•	The paper has adequate ablation experiments.

Weaknesses
•	I noticed that the results in Table 2 used different settings(0-Tensor vs Template). The author just lists the best results, the results of same algorithms should come from the exact same network structure, otherwise, they should be shown separately. For the same reason, the result of TNL2K in table 3(48.3/46.6) is different from it in Table 2(49.8/51.0).

questions: This paper doesn't clearly state how to get the language description of a template during inference. If the language description is from annotation or manual setting, it is not reasonable enough to compare results with algorithms that do not use language features.
limitations: The article does not include a discussion of solving common tracking problems such as target deformation and occlusion. The introduction of the template update module may have better results on some samples.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an interesting solution to show how to achieve state-of-the-art (sota) tracking performance without relying on the complex Transformer architecture in visual tracking. Specifically, the authors explore a unified multimodal learning of vision and language under the simple ConvNets for tracking. The authors first give an in-depth analysis of the limitations of previous solutions and then introduce an innovative unified framework that learns better vision-language (VL) representation by proposing ModaMixer and asymmetrical searching strategy (ASS). The former well enhances the interaction inside vision-language for discriminative representation, while the later improves the adaption of the learned VL representation by ModaMixer. Extensive experiments on five challenging datasets demonstrates that, the proposed method improves a pure CNN-based baseline to be competitive and even better than recent Transformer-based counterparts. Moreover, the proposed method is general and can also be applied to improve Transformer tracking architecture, as shown by experiments. In addition, some theoretical analysis is given to explain the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths

Overall, I am rather positive on this paper. In particular, I really like the motivation of this work that aims at finding alternative path, instead of relying on Transformer architecture (although it is shown to be powerful), to achieve sota tracking by exploring the almost ignored multimodal learning, which I believe can inspire many other works on tracking and facilitates this field. The strengths in this work include:

(1) Novelty. This paper introduces an innovative framework of multimodal vision-language learning (VL) for object tracking. The proposed ModaMixer enables a good interaction between the two modalities throughout the whole network, which is completely different from other works that only interact two modalities at the result fusion stage. In addition, another novel point is the proposed ASS that adapts the representation of different branches and modalities in an asymmetrical way. Compared with the conventional symmetrical way, ASS is the first time to show the asymmetrical architecture may be more suitable for representation learning in tracking and may provide new insights to future research. I like these novel ideas in this work.

(2) Good performance. This paper demonstrates excellent performance in improving the simple pure CNN-based tracking baselines. In specific on the challenging LaSOT, the proposed approach improves the baseline from 50.7% to 65.2% in SUC with absolute 14.5% gains, which is competitive and even better than recent Transformer-based trackers. Compared to previous best VL method with 54.0% SUC, this paper shows 11.2% improvement. The performance on other benchmarks is also sota. This excellent performance clearly shows the effectiveness and advantages of the proposed method.

(3) Generality. The proposed method is general and not limited to CNN-based framework. The authors show this point by applying their methods on a Transformer-based tracking framework and show promising improvements on all the five challenging benchmarks, which is consistent with the improvements shown for CNN-based framework.

(4) Rich experiments and analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module and overall performance, and the theoretical analysis gives an explanation why the proposed method works well in improving tracking performance.

(5) Good writing and organization. This paper is well written and organized. Each section has a clear motivation. It’s easy to follow the ideas. I enjoy reading the paper.

Overall, I believe this paper is significant to the visual tracking community because it shows new insights and directions in designing simple but effective tracking framework with sota performance.

Weaknesses

Although this paper is technically sound and novel, I have some minor concerns or questions.
(1) In this work, language is crucial. I noticed that in the experiment the pseudo language description generated by an image caption model does not show significant improvements. What do you think are the reasons causing this?
(2) The proposed method is shown to be general, which is nice. However, it will be great if the authors can show more examples such as conducting more experiments on additional pure CNN-based frameworks like SiamRPN++ (Li et al, CVPR, 2019).
(3) The authors show many comparisons with other sota methods. But I don’t see qualitative comparison with other approaches. It will be nice to see these results and comparison to other sota methods.
(4) For experiments with partial language (50% and 75%) in the supplementary material, how do you determine the which 50% or 75% should be used? Randomly generating? If you randomly generate it, it will be better to do multiple (e.g., 3) times of experiments. 

questions: See Weaknesses.
limitations: I don’t see the discussion related to the limitations and broader impact. I hope the author can provide a brief discuss on this part in the response and are encouraged to detail this part in the revision.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a method to learn the vision-language representation for achieving SOTA tracking by using pure ConvNets. It proposes a modality mixer as the fusion module and uses it in both low-level ConvStages and high-level ConvStages to explore the complementarity of different modalities. It also uses NAS to search an asymmetrical siamese network to learn adaptive vision-language representations. The proposed method is also used to improve a Transformer-based tracker.
strengths_and_weaknesses: Strengths: 
+The proposed method can make a CNN-based tracker and a Transformer-based tracker achieve SOTA performance on five benchmarks. The performance of vision-language tracking is improved largely.

+This work uses NAS for tracking to search different model structures for template and search region respectively. This is a novel practice for siamese trackers.

 Weaknesses: 
-The proposed trackers and the baselines use different training data. The comparisons in this paper are unfair. For instance, both SiamCAR and TransT didn’t use TNL2K for model training, they shouldn’t be used as the baselines for comparison on TNL2K. SiamCAR only uses the LaSOT training set for model learning to achieve 50.7SUC. But the proposed trackers use lots of extra training sets. So the performance gains shown in Figure1 are debatable, and the current ablation study cannot show the effectiveness of the proposed method.

-Modality Mixer is too simple to be the main contribution. It’s cross-modal channel attention. The authors attempted to make the structure more complex by designing a residual connection. But this modification brings some doubts about the effectiveness of the cross-modal channel attention. We don’t know whether the final improvements can be attributed to the modal-fusion or the pure vision feature. This paper didn’t provide any ablation analysis about it. The current experiments are not sufficient.

-This paper proposes a VL representation learning method for object tracking. But half training pairs don’t have a language description, and the authors use the 0-tensor or the visual feature as the pseudo language description for training. It's doubtful that the learned representations are really multi-modal representations. The authors conducted an experiment by removing the description to prove the effectiveness of modal fusion. But how about using a 0-tensor or a visual pooling feature for inference? The current experiments are insufficient.
questions: -The proposed trackers and the baselines should be trained with the same training set.
-More quantitative studies about the modality mixer should be supplemented.
-During inference, using the 0-tensor or the visual pooling feature for further analysis.
-See weaknesses.
limitations: N/A
review_title: null

Meta-review: metareview: All three reviewers lean towards the acceptance of the paper. Reviewer YvUr was not 100% excited about the paper, pointing out the simplicity of the approach and lacking ablations. We encourage the authors to include the new materials they prepared for the rebuttal in the final version of the paper.
confidence: Certain
Decision: Accept