Title: LoRATv2: Enabling Low-Cost Temporal Modeling in One-Stream Trackers
Venue: NeurIPS 2025
Accepted: yes
Keywords: visual object tracking
Abstract: Transformer-based algorithms, such as LoRAT, have significantly enhanced object-tracking performance. However, these approaches rely on a standard attention mechanism, which incurs quadratic token complexity, making real-time inference computationally expensive. In this paper, we introduce LoRATv2, a novel tracking framework that addresses these limitations with three main contributions. 
First, LoRATv2 integrates frame-wise causal attention, which ensures full self-attention within each frame while enabling causal dependencies across frames, significantly reducing computational overhead. Moreover, key-value (KV) caching is employed to efficiently reuse past embeddings for further speedup.
Second, building on LoRAT's parameter-efficient fine-tuning, we propose Stream-Specific LoRA Adapters (SSLA). As frame-wise causal attention introduces asymmetry in how streams access temporal information, SSLA assigns dedicated LoRA modules to the template and each search stream, with the main ViT backbone remaining frozen. This allows specialized adaptation for each stream's role in temporal tracking.
Third, we introduce a two-phase progressive training strategy, which first trains a single-search-frame tracker and then gradually extends it to multi-search-frame inputs by introducing additional LoRA modules. This curriculum-based learning paradigm improves long-term tracking while maintaining training efficiency.
In extensive experiments on multiple benchmarks, LoRATv2 achieves state-of-the-art performance, substantially improved efficiency, and a superior performance-to-FLOPs ratio over state-of-the-art trackers.
The code is available at https://github.com/LitingLin/LoRATv2.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LoRATv2, a novel framework for efficient multi-frame temporal modeling in Transformer-based trackers. The key contributions are Frame-Wise Causal Attention (FWCA), Stream-Specific LoRA Adapters (SSLA), and two-phase progressive training. Extensive experiments demonstrate state-of-the-art accuracy–FLOPs trade-offs across multiple tracking benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.Integration of causal attention into one-stream Transformer trackers with KV caching, greatly reducing temporal modeling cost.\n2.SSLA provides stream-specific adaptation without unfreezing the ViT backbone, incurring zero extra inference cost.\n3.The two-phase progressive curriculum reduces memory/time overhead in multi-frame training while improving long-term tracking.\n\nWeaknesses:\n1.The choice of LoRA rank is only reported in Section 4.1 without any ablation study to justify the selected rank size.\n2.The discussion of the KV cache is insufficient—for example, what is the maximum cache capacity and how much memory does it consume?\n3.The Phase 1 models ablation experiments (1 and 3) do not adequately demonstrate the effectiveness of the FWCA and SSLA modules. Could you please supplement with additional experiments involving self-attention + SSLA?\n\nQUESTIONS:\n1.What level of performance can be achieved using only Phase 1 in low-compute scenarios?\n2.Does caching multiple frames in the KV cache lead to GPU memory or efficiency bottlenecks, or cause performance degradation?\n3.How do different LoRA rank values impact performance?\n\nLIMITATIONS:\n1.Due to the causal attention mechanism, reliance on historical frames in occlusion and fast-motion scenarios can lead to tracking errors and degrade tracking accuracy.\n2.As the number of historical frames increases, the KV cache size also grows, which may cause GPU memory pressure in long-sequence scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
The paper proposes LoRATv2, an efficient multi-frame Transformer tracker that combines three ideas: (i) frame-wise causal attention plus KV-caching to cut the quadratic cost of vanilla self-attention, (ii) stream-specific LoRA adapters that let template and search streams specialize while keeping the ViT backbone frozen, and (iii) a two-phase curriculum that first trains on single-frame inputs and then extends to multi-frame ones.

STRENGTHS AND WEAKNESSES:
Strengths
1. Well-targeted efficiency innovation. The frame-wise causal mask plus KV-cache gives a clean O(N) path length without redesigning the backbone.
2. Stream-specific LoRA keeps training cost low and lets the authors reuse large off-the-shelf ViTs.
3. The paper is well-written and easy to follow, with clear visualizations.  

Weaknesses
1. Table 1 needs fresher baselines.
2. OSTrack speed anomaly in Table 2. Although OSTrack shows lower MACs than LoRATv2, its FPS is lower. 
3. Minor English issues. “state-of-the-art” appears both hyphenated and as “state of the art”; consider unifying.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFirst, this paper addresses the computational complexity of standard attention operations in LoRAT v1. Specifically, when processing high-resolution or multi-frame inputs, the standard attention mechanism incurs substantial computational overhead during inference. To resolve this, the paper reframe tracking as an autoregressive sequence prediction task and propose a frame-wise causal attention mechanism, where each frame exclusively attends to preceding frames. This approach is further accelerated through key/value caching, significantly reducing inference latency.\n\nSecond, while causal attention mitigates computational costs, it introduces feature inconsistency across frames due to varying receptive fields. The paper counteract this by designing Stream-Specific LoRA Adapters, which allocate dedicated LoRA modules to distinct input frames. This ensures parameter-efficient feature alignment while maintaining temporal coherence.\n\nThird, to scale the model for multi-frame inputs, a two-phase progressive training strategy (inspired by curriculum learning) is introduced. Compared to direct end-to-end training on multi-frame data, this phased approach reduces training complexity.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality: The paper presents technically sound methodologies with substantial experimental validation. However, additional experiments are required to strengthen the conclusions:\n\n    • Comparative analysis of direct multi-frame training: Quantitative results (accuracy/FPS) and resource consumption (GPU hours/memory) versus the proposed two-stage strategy, demonstrating the necessity of phased training.\n\n    • Scalability validation of SSLA: Performance evaluation with more frame inputs to substantiate Stream-Specific LoRA Adapters' efficacy in large-scale scenarios.\n\n2. Clarity While generally well-structured, the manuscript requires little revisions:\n\n- The type emb in Figure 2 comes from the LoRAT v1 paper, but it is not mentioned in the revisiting section of this paper, so it needs to be briefly explained again.\n    \n\n- Table 3 conducts ablation experiments on model design. First, as a personal suggestion, if the final model design is used as the baseline for the experiment, the Variants column should use “-” to indicate changes in model design instead of “+”. For example, “+ w/o SSLA” should be “- SSLA”. Second, the ablation designs for Variants in phases 1 and 2 should be as consistent as possible.\n    \n\n3. Significance: The work significantly addresses the computational bottleneck of ViT-based trackers during inference, particularly for high-resolution/multi-frame use cases.\n\n4. Originality: The core contributions are primarily engineering-driven enhancements rather than theoretical innovations:\n\n- Frame-wise causal attention: While effective, this mirrors prior autoregressive designs and shares operational similarities with S-MAM in MixFormer.\n    \n- Key-value caching: Positioned as an implementation optimization (also seen in MixFormer's public codebase) rather than a novel methodological contribution.\n\nQUESTIONS:\n1. The Model Variants section in 4.1 is unclear. The model's input settings for different phases and resolutions should be clearly specified.\n\n2. Table 2 shows the parameters of different models in terms of efficiency. The parameters of LoRAT are consistent across different resolutions, but those of LoRAT v2 are not. Could you please explain the reason for this?\n\n3. Could you please explain the reasons for the failure of ② in Table 3 to converge as much as possible?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}