Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a simple yet strong baseline tracker, named SwinTrack, for single object tracking (SOT). SwinTrack adopts the classic siamese structure, and uses Swin Transformer for both feature extraction and fusion of template and search region. A special motion token which encodes the past trajectory is introduced to further boost the tracking performance. SwinTrack is evaluated on major SOT benchmarks and achieves SOTA performance. 
strengths_and_weaknesses: Single object tracking (SOT) is an important problem in computer vision and for video understanding in particular. The proposed method is technically sound and achieves SOTA performance on major SOT benchmarks. Although most components in SwinTrack are existing ones and cannot be said novel, I appreciate the system work which puts them together in a proper way and eventually achieves good results. In particular, the design choices to use concatenation-based fusion over cross-attention-based fusion and not to use query-based decoder are reasonable and well-justified by ablation studies. 
The motion token design is new, based on my knowledge. According to the ablation, it achieves non-negligible gain, making it a good added component to the tracker. 
questions: One main contribution of SwinTrack is to use Transformer network not only for feature fusion but for feature extraction. This is quite obvious since Transformer-based vision feature has shown its advantages over CNN-based feature in numerous scenarios and downstream tasks. One question I have been wondering is  why previous Transformer-based trackers adopt CNN-Transformer hybrid architecture? Are there any obstacles in using Transformer for feature extraction? Is there any secret weapon that the authors use to make it work?
limitations: This reviewer does not see potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the problem of general object tracking, where the target object is given during test time. The authors propose a simple but strong baseline approach based on a Swin Transformer image encoder and a standard transformer decoder. Further, the paper introduces a motion token, which encodes past bounding boxes in order to improve the final prediction. The authors also revise the losses employed to train the tracker. Very strong results are reported on 5 popular benchmarks.
strengths_and_weaknesses: Before dwelling into the strengths and weaknesses, I want to discuss the novelty aspect of this paper. On one hand, the novelty is not remarkable: the authors are adopting the Swin backbone, while concatenation fusion is also used in Stark. However, the novelty is not limited to these aspects. On the contrary, I believe that this paper brings substantial value to the tracking field by consolidating existing techniques, while investigating important details in order to achieve a simple, yet highly powerful tracking framework. Importantly the authors provide valuable insights when motivating their approach and comparing to other techniques (modifications of fusion, transformer architecture, losses, etc.). Lastly, I find the motion token a very interesting novelty that could reopen a long-forgotten direction in tracking, namely exploiting motion prediction and other dynamic information. Although seemingly incremental at the first glance, I consider the novelty to be significant based on how much this paper advances the useful knowledge in the field.

Other strong points of the paper are:

• Very strong results, clearly SOTA.

•	Simple and elegant architecture.

•	Insightful discussions.

•	Method and motivation are easy to follow.

•	Interesting ablative experiments on multiple datasets.

•	Relatively fast frame-rates.


Weaknesses:

1. Details regarding pre-training are missing. I assume that the authors use ImageNet-22k pre-training, while most trackers use ImageNet-1k. This has shown to give about 2-3% on LaSOT in previous papers. The authors should therefore analyze this in a separate experiment.

2. By looking into the more detailed results in the supplementary material, it seems that the improvements mostly stem from increased accuracy. That is, better bounding box regression. The robustness (low overlap scores in the success plot) seems to be on par with recent trackers. While accuracy is also important, the major challenge in tracking is to improve the robustness. It is important to discuss these aspects in the paper in order to understand where and how SwinTrack performs better compared to other trackers. Moreover, please add more high-performing trackers to the success plots in the supplementary material.

3. Unfortunately, the design of the motion token is motivated. For instance, why are the past box encodings concatenated in the channel dimension, and not processed in some other manner? Why add it as a single token in the transformer, instead of one per box?

4. Results on UAV and VOT should be added, even if the tracker does not beat SOTA.

5. There are quite a few language mistakes.
questions: See weaknesses.
limitations: I could not find discussion of negative social impact or limitations. It would be good to add these.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose an interesting Siamese architecture for single object tracking based on the Transformer model. The encoder is divided into the two feature transforms and a transformer encoding step where correlations of spatiotemporal features on the concatenated branches are made. Instead of CNNs the authors suggest SwinTransformers as branches with shared weights. In the encoding part of the Transformer motion information is added to the stream of features by assuming smooth motion of the object and by maintaining a limited history of the BBs of the object. BBs are regressed with the output embeddings. The experiments show state-of-the-art results on large-scale tracking datasets.
strengths_and_weaknesses: The idea to use SwinTransformer instead of a CNN in the branches seems to be novel and also seems to be fruitful at least in terms of algorithmic efficiency and surpassing the 70% limit in LASOT (Fig. 1). Concatenating the features of the branches and then applying a Transformer like attentional mechanism to extract embeddings for which BBs can be regressed is then straightforward [5, 36]. The use of non-learnable queries might be a novel change in the design and beneficial in terms of interpreting the tracking problem as classification problem.
Motion consistency restricts the method to track objects properly e.g. when it comes to abrupt motions. Although the common experimental procedure on standard datasets including ablation studies has been conducted, the experiments show little insight into the effects of assumptions and design decisions. While overall benchmark scores and leave-one-out results give an answer to the usefulness of (parts of) the method compared to others, the methodology gives very little insight into the challenges and the potential causes of low performance, e.g. in case of rapid motion, occlusion, illumination changes, etc. I very much miss a deeper qualitative analysis of the model and understanding of its behaviour when it comes to the challenges of tracking.
The paper is well structured, however I recommend to proof-read the document as there are typo errors in the text, e.g. line 120. The English needs to be improved, e.g. line 163: “chopped off” sounds flappy. Technically the paper needs also to be improved, e.g. line 145, for n=0 T_traj is not defined properly, for n=1 T_traj = {o_c-delta, o_c, o_c-delta} which makes no sense.

questions: Could you explain in more detail why it is beneficial to neglect learnable queries in the model design?
Why did you not test your method on the VOT challenges? They give experimental results for the different challenges / attributes of tracking.
How would your method work when an object abruptly changes the direction of motion as you assume motion consistency?
limitations: No.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a fully-attentional tracker where both the representation learning and the feature fusion modules are based on the Transformer. Besides, a motion token is introduced to embed the historical target trajectory, further improving the tracking performance. Extensive experiments and ablation study on several large-scale tracking benchmarks are performed to demonstrate the effectiveness and efficiency of the proposed approach.
strengths_and_weaknesses: For the strengths, 1) Compared with recent Transformer-based methods that use the Transformer to perform the feature fusion, the proposed tracking framework takes advantage of the Transformer on both the representation and feature fusion modules, which is more thorough and more reasonable.
2) The design of introducing the motion token is novel and effective for tracking.
3) Experiments, especially the ablation study are sufficient, the effectiveness of each designed module is proved.

For the weaknesses, 1) For the motion token, the template-search region pair is based on the single frame, while the object trajectory is sequential. To make the trajectory invariant to the cropping, do all bounding boxes in one trajectory perform the same transformation, i.e., based on the cropping of the current frame? This point is somewhat confusing. 
2) The fixed sampling interval of the trajectory is set to 15 and 8 for different datasets, is it the same setting for the training and inference phase? Besides, will it make a significant performance fluctuation when increasing or decreasing this value? I think the setting of this value needs to be explained. 
questions: Please explain the aforementioned two weakness issues.
limitations: N/A
review_title: null

Meta-review: metareview: Authors present a method for single object tracking (SOT) that is entirely comprised of transformers. The architecture is simple: 

1) Swin is used to generate embeddings for both template and search region
2) Embeddings are concatenated
3) An encoder transformer performs MHSA of the embeddings.
4) A decoder performs cross-attention from search tokens to template tokens and a special "motion token" which is constructed from a linear operation over the prior motion trajectory relative to the frame. 
5) Output token is fed to final layers that perform IoU aware classification and bounding box regression.

Evaluations are performed on 5 SOT datasets, achieving SOTA on all of them.


Pros:
- [AC/R] Important problem, technically sound, and new SOTA on this task.
- [AC/R] New motion token approach is novel and provides significant improvement.
- [AC/R] Simple and elegant architecture
- [AC/R] Insightful discussions
- [AC/R] Clearly written and easy to follow
- [AC/R] Interesting ablations
- [AC/R] High frame rate

Cons:
- [R] Low novelty of transformer approach, but this is negated by the novelty of the motion token.
- [R] Details regarding pretraining are missing. Authors provide in response.
- [R] Motivation of motion token design is not clear. Authors provided further ablations of different implementations of the motion token, showing the current form performs best.
- [R] Provide additional details regarding where and how SwinTrack outperforms other approaches on the benchmarks. Authors provided additional granularity of performance stratifications within the LaSOT benchmark.
- [R] Add more recent high performing trackers. Authors added several methods published in 2022.
- [R] Some additional questions about various details were posed by reviewers, which will all answered by the authors. 

The single reviewer with reject recommendation changed to accept in their comments after the discussion period but did not update their score. Given unanimous agreement on accept, AC recommendation is accept. 

AC Rating: Strong Accept
confidence: Certain
Decision: Accept