LoRATv2: Enabling Low-Cost Temporal Modeling in One-Stream Trackers
Authors: Liting Lin
Heng Fan
Zhipeng Zhang
Yuqing Huang
Yaowei Wang
Yong Xu
Haibin Ling
Venue: NeurIPS 2025 spotlight ()
Keywords: visual object tracking
Primary area: applications

Abstract: Transformer-based algorithms, such as LoRAT, have significantly enhanced object-tracking performance. However, these approaches rely on a standard attention mechanism, which incurs quadratic token complexity, making real-time inference computationally expensive. In this paper, we introduce LoRATv2, a novel tracking framework that addresses these limitations with three main contributions. 
First, LoRATv2 integrates frame-wise causal attention, which ensures full self-attention within each frame while enabling causal dependencies across frames, significantly reducing computational overhead. Moreover, key-value (KV) caching is employed to efficiently reuse past embeddings for further speedup.
Second, building on LoRAT's parameter-efficient fine-tuning, we propose Stream-Specific LoRA Adapters (SSLA). As frame-wise causal attention introduces asymmetry in how streams access temporal information, SSLA assigns dedicated LoRA modules to the template and each search stream, with the main ViT backbone remaining frozen. This allows specialized adaptation for each stream's role in temporal tracking.
Third, we introduce a two-phase progressive training strategy, which first trains a single-search-frame tracker and then gradually extends it to multi-search-frame inputs by introducing additional LoRA modules. This curriculum-based learning paradigm improves long-term tracking while maintaining training efficiency.
In extensive experiments on multiple benchmarks, LoRATv2 achieves state-of-the-art performance, substantially improved efficiency, and a superior performance-to-FLOPs ratio over state-of-the-art trackers.
The code is available at https://github.com/LitingLin/LoRATv2.

LoRATv2: Enabling Low-Cost Temporal Modeling in One-Stream Trackers
Liting Lin Pengcheng Laboratory 1t. ww..ionootloo.... com
Heng Fan University of North Texas heng fan@unt..... edu
Zhipeng Zhang
Shanghai Jiao Tong University
Anyverse Intelligence zhipeng. zhang cv@outlook.com com
Yuging Huang
Yaowei Wang' Harbin Institute Technology, Shenzhen Pengcheng Laboratory wangyaovei@hit. edu, cn
Pengcheng Laboratory
domaingreenn2gmail.ccmm
Yong Xu
Haibin Ling" Westlake University linghaibinnkesstlake edu. cn
South China University of Technology yxu@scut.edus edu. cn
Abstract
Transformer-baseee algorithms, such LoRAT, have significantly enhanced object- tracking performance. However, these approaches rely on a standard attention mechanism, which incurs quadratic token complexity, making real-time infer- ence computationally expensive. this paper, we introduce LoRATv2, a novel tracking framework that addresses these limitations with three main contributions. First, LoRATv2 integrates frame-wise causal attention, which ensures full self- attention within each frame while enabling causal dependencies across frames, significantly reducing computational overhead. Moreover, key-value (KV) caching is employed efficiently reuse past embeddings for further speedup. Second, building on LoRAT's parameter-efficien fine-tuning, we propose Spream-Specific LoRA Adapters (SSLA). As frame-wise causal attention introduces asymmetry how streams access temporal information, SSLA assigns dedicated LoRA modules the template and each search stream, with the main ViT backbone remaining frozen. This allows specialized adaptation for each stream's 's role in temporal track- ing. Third, we introduce a two-phase progressive training strategy, which first trains single-search-frang tracker and then gradually extends it to multi-search- frame inputs by introducing additional LoRA modules. This curriculum-bast based learning paradigm improves long-term tracking while maintaining training effi- ciency. In extensive experiments on multiple benchmarks, LoRATv2 achieves state-of the-art performance, substantially improved efficiency, and superior performaance-t--FLOO ratio over ofate-of-thh-al -het trackers. The code available at htttps://githiii gitgggtthbbb... httpa//cmm/mnnnnnnnnnnooo.o
Introduction
Visual object tracking a fundamental task in computer vision with broad applications in surveillance. autonomous driving, robotics, augmented reality, and human-computer interaction. Accurate and continuous object localization across frames crucial for the reliability of these applications.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Visual Object Trackers: Efficiency vs. Performance
76
LaSOT SUC score
74
73
72
71
OSTrack
SegTrack
ROMTrack
ARTracky2
LeRATv2 (Ours)
20
30
50
70
100
200
300
500
MAC' (G)
Figure I: Comparison of our tracker (LoRATv2) with other Transformer trackers on computational complexity (MACs) and performance (SUC scores) on the LaSOT dataset.
Recent advances in Vision Transformers (ViT) []4] have substantially improved tracking performance, primarily due to the powerful representation capabilities of self-attention. In particular, one-stream trackers, such as MixFormer()) OSTrack [43], ARTrack (35). SeqTrack 77 and LoRAT (24), unify feature extraction and relation modeling within a single Transformer backbone. Although this design has demonstrated excellent accuracy and efficiency, these methods rely on standard self-attention. which has quadratic complexity with respect to the number of tokens. Consequently, inference can become prohibitively expensive when dealing with high-resolution input frames or extended temporal sequences, constraining real-time deployment on resource-limien systems.
recently proposed state-of- the-art tracker, employs Parameter-Efficient Fine-Tuning (PEFT) via Low-Ram Adaptation (LoRA) [19 to significantly reduce the training overhead for large- scale Transformer trackers. However, it retains the standard self-attention mechanism inference leading to substantial computational costs as input resolution or frame count increases.
LoRAT 24
excel with single template- seauth pairs, simply extending them to multi-frame inputs using standard bidirectional attention often yields only marginal performance gains in preliminary experiments. We hypothesize this may be partly due to potential rank collapse issues in deep Transformers with bidirectional attention, which can limit their expressive power for complex temporal sequences [33]
Furthermore, while one-stream trackers 43 24
To overcome these limitations and better model the sequential nature of tracking, we propose LoRATv2. Visual object tracking can naturally be framed an autoregressive sequence prediction task, where the target's s state with causal attention mechanisms. Therefore, LoRATv2 introduces the frame-wise causal atlention mechanism within the ViT backbone. In this design, tokens within the current frame maintain full self-attention for rich fitra--raam representation, while attending causally only to tokens from previous frames, enforcing an autoregressive structure. This approach, coupled with key-value (KV) caching 33
current frame depends on its history. This perspective aligns well
| 31 to reuse past embeddings, not only significantly enhances inference efficiency by reducing MACs and improving FPS, but as our experiments demonstrate (shown in Fig. [. also leads to improved tracking performance by effectively modeling temporal dependencies.
The adoption of frame-wise causal attention, while beneficial for temporal modeling and efficiency, introduces an inherent asymmetry in how different input streams (e.g.. the static template, the first search frame, subsequent search frames) process information. Unlike traditional Siamese trackers one-stream trackers with full attention where all tokens potentially interact symmetrically, in our

---

causal setup each stream has distinct view of historical context, as it can only attend to itself and preceding information. To manage this asymmetry effectively. while preserving LoRAT's parameter- efficient fine-tuning paradigm, we propose Stream-Specific LoRA Adapters (SSLA). SSLA allocates dedicated LoRA [19] adapters to each input stream, and the main ViT backbone is kept frozen and shared. This allows each stream to develop specialized adaptations. For instance, the template's s LoRA can focus on robust initial feature extraction, the first search frame's 's LoRA might learn to enhance target features while identifying distractors, and LoRAs for subsequent search frames can specialize in precise localization based on the evolving temporal context. This modification maintains the benefits of shared backbone yet allows minimal, dreendddepeneett adaptation critical for effective multi-frame tracking.
Finally, we present a two-phase progressive training strategy to incrementally expand from single- search-frame to multi-search-framm tracking. This approach significantly reduces the computational resources required for training. as at any given time, only the LoRA modules associated with one or two input streams are trainable, while previously trained LoRA modules and the main backbone remain frozen. This easy-to-hard, curriculum learning-style [2] training paradigm, as demonstrated in our experiments, leads to better tracking performance compared to training all LoRA modules from scratch. Furthermore, this strategy naturally yields family of trackers capable of processing different numbers of input frames (e.g., a single-search-frame tracker from the initial stage and a multi-search-- frame tracker from the extension stage), providing additional flexibility for deployment scenarios with varying computational budgets or accuracy requirements.
Our contributions can be summarized as follows:
1. Frame-wise Causal Attention: We integrate frame-wise causal attention into one-stream visual tracking, where tokens attend fully within their frame and causally to past frames. Combined with KV caching reuse past embeddings and prevent re-encoding, this mech- anism enables computationally efficient long-term temporal modeling for trackers and improves tracking accuracy by robustly capturing temporal context.
2. Stream-Specifie LoRA Adapters (SSLA): To handle the asymmetry introduced by causal connections, we equip each input stream (template or search) with its own low-rank LoRA modules. This preserves shared backbone for all streams while allowing minimal, stream- specific adaptation for improved tracking accuracy, with zero additional inference cost.
3. Two-Phase Progressive Training: We first train single-frame tracker (template - one search frame) and then incrementally extend it to multi-frame inputs by adding new LoRA adapters only for the additional search frames. This curriculum-like approach improves long-term performance while greatly reducing memory and training time compared to direct multi framm training from scratch.
4. Extensive experimental evaluations across multiple benchmark datasets demonstrate that our proposed approach delivers superior tracking performance and significantly improves performance-to-FIee ratio compared to existing state-of ofate-of-the-arr the-art self-attention-bet based meth- ods.
2 Related Work
Temporal modeling in tracking. Temporal modeling in visual tracking typically involves leverag- ing historical information and explicit temporal dependencies to enhance robustness and accuracy. Some methods employ historical prompts improve tracking stability, such as AQATrack [40] and HipTrack [3]. Approaches like STARK [41] and TATrack [18) dynamically update tracking templates based on prior tracking outcomes to maintain effectiveness over time. Furthermore, au- toregressive prediction strategies have been explored extensively in methods such as ARTrack [35) ARTrackV2 [ and SeqTrack 77, which explicitly model temporal dependencies across frames. Recent advancements also include multi-frame modeling methods. For example, TCTrack [5] in- corporates video-level contextual information through adaptive convolutional techniques, whereas VideoTrack [37] uses transformer-basel architectures to integrate broader temporal context. Ad- ditionally, ODTrack [45] explicitly propagates token sequences between frames maintain dense contextual associations.
3

---

MLP head
Lx
LoRA
Lok
LoRA
MLP
Frozen
Traimblee
Norm
pos. emb.
yype emb.
LoRA
LoR
LoRA
patch emb.
Multi-head Causal Attention
LayerNorm
(cached)
(cached)
Linear Projection
Figure 2: Overall architecture of LoRATv2. We tokenize the template and search regions, append them into a single token sequence with shared positional and stream-specifie token type embeddings, and feed them a frozen ViT encoder. Frame-wiss wise causal attention ensures that each frame only attends to itself and preceding frames. Mcanwhile, KV caching reuses previously computed key/value embeddings to avoid redundant computation across frames. We attach lightweight LoRA modules (one per stream) to each linear projection layer and keep the main ViT weights frozen. The final embeddings are passed through an MI. head for target classification and bounding-bow regression, yielding an efficient and scalable solution for multi-frame visual tracking.
Causal attention in vision. Causal attention in vision addresses spurious correlations by explicitly modeling causal relationships. Methods like CATT [42] introduce causal interventions within attention modules to mitigate contextual bias, enforeing object-context separation through sample- wise masking. Building upon these foundations, subsequent approaches extend causal attention to autoregressive frameworks. Rision-RWKV [5], for example, adapts causal attention mechanisms for bidirectional global interactions within vision transformers. To enhance scalability and efficiency, some models incorporate hierarchical or sequential structures. VAR (32 employs causal masking strategies for efficient coarse-to-fine image generation. Similarly, Causal Vision Transformers [22 apply sequential causal attention to efficiently handle large-scale images. Moreover, Show-O unifies multimodal generation tasks under single causal transformer framework. Collectively, these advancements underscore the versatility and efficacy of causal attention methods across diverse visual tasks, spanning recognition to generation, while enabling scalable and controllable visual modeling.
3 Method
Our proposed LoRATv2 builds upon LoRAT (24 and introduces additional components for efficient multi-frame, temporal-aware visual object tracking. First, we revisit LoRAT to recap its core design prineiples (Sec. 3.T. Next, we present our Frame- Wise Causal Attention (Sec. 3.2) and Key- Value Caching (Sec. 3.3 which jointly enable efficient multi-frame modeling without re-encoding past frames. To address the asymmetric dependency introduced by causal attention, we propose Stream- Specific LoRA Adapters (S4c.3.44) which preserve a unified embedding space while enabling minimal, per-stream adaptation. Finally, Sec. (3.5 details our Two-Phase Progressive Training pipeline for scaling from single-frame to multi-frame scenarios with minimal overhead. Unless otherwise noted, all other design and training settings (e.g., shared positional embeddings, token-type embeddings. anchor-free MLP heads) follow LoRAT (24]
Krame-Wise

---

T
T
T
T
T
T₂
S
S
T
T2 S
T
T2
S
Tj T
S
(a) Full Self-Attention
(b) Causal Attention
(c) Frame-wise Causal Attention
Figure 3: Visualization of three attention masks. (a) Full self-attention, where each query position (b) Causal attention, where each token can only attend to itself and tokens to its left. (c) Trame-wise causal attention, where each token can only attend to the tokens within current frame and tokens in the past frames. Dark squares indicate valid (unmasked) attention, while white squares indicate invalid (masked) attention.
can attend to all key positions.
3.1 Revisiting LoRAT
One-Stream Transformer Tracker. LoRAT 24 follows the one-stream Transformer tracker paradigm, where the template and the search region are processed together by single ViT backbone. First, the images are divided into non-overlapping patches. These patches are then flattened and mapped to patch embeddings via a linear projection, yielding the template tokens {zi } and search region tokens {x,) To distinguish these tokens after concatenation, LoRAT incorporates learnable token type embeddings that are added to the patch embeddings of each stream (template or search). Additionally, uses shared positional embeddings for both streams. The final input sequence is formed by concatenating the enriched tokens:
X [Zi..... Zm X1. XI.
Transformer encoder then applies multi-head self-attention and feed-forward layers capturing intra-frame and inter-frame relationships. Finally, the output embeddings corresponding to the search region are fed into MLP--based head network for target classification and bounding box regression, producing the tracker's output.
X,
LoRAT employs standard full self-attention in its VIT backbone. Let X e IR (md denote the token embeddings. The standard attention mechanism first computes:
Self-Attention Complexity.
Q = X K K=XW
- XWW
and then:
QK
Attention SettiiinnQ,,K Q, = softmax
V
(3)
Vd
This incurs O(mm + 2) operations. Although LoRAT uses Low-Rank Adaptation (LoRA) [19 (0
reduce training overhead, the inference cost remains governed by full self-attention.
3.2 Frame-Wiis Causal Attention
Standard full self-attention (Eq. 3 grows quadratically with the number of tokens and does not incorporate temporal order. To efficiently model longer sequences in tracking, introduce causal mask that enforces an autoregressive dependency across frames while preserving full attention within each frame.
Assume we have 7' frames, each producing n, tokens (xf...., } for I,...,2 T. Concatenate them into single sequence:
X
x
(4)
frame
5

---

In wrim causal attention, queries from frame t can attend to tokens in frames 1,.... è but not to frames > t. We implement this via causal mask M IRNN,, where N = > Te defined by
0, if fg) I frame(p),
(
-G0 otherwise.
Let = Xw Xw and V XWV be the usual linear projections. Our frame-wise causal attention modifies Eq. by adding M inside the softmax:
Attention Q,, V) softmax OK'
Vd + +M) V.
(6)
(p.r) = 0 since frame(p) frame (q), thus preserving full (unmasked) attention. Across frames, any future-to-par past connection is disallowed by setting M = -oo if frame(p) < f4) enforcing auto-regressive structure. The visualization of frame-wise causal attention mask is shown in Fig. 3c
Within each frame, (p.r)
3.3 Key- Value Caching
When performing online tracking processing long sequences, naively recomputing key/value embeddings for past frames is computationally expensive. Key- Value (KV) caching [33 provides a simple yet powerful mechanism amortize this cost over time.
31
At frame 1, we first compute queries for the current frame:
X'
Here, X E gu,xd xd denotes the tokens frame t. The key/value pairs for frame are
K = X' V = X'W.
(8)
We then cuche past keys and values (frames through 1):
K lit-1
K' V 1:t-1 = [V]
V'-I].
99)
For the frame-wise causal attention at step t, we form
K'
K 1 K' V V lit-1 v
(10)
Hence, the attention computation at frame t becomes
Attention CaauutionnQ."r QQ,K'. K' V*),
(11)
where the mask M (Eq. 5) ensures that Q' can only attend K 1=-1 (past frames) and K' (the current frame), but not beyond. Crucially, we never re-encode past frames; we simply reuse their cached Kt V
3.4 Stream-Specific LoRA Adapters (SSLA)
While frame-wise causal attention preserves within-frame symmetry, the Siamese template and search region streams across frames might still diverge in how they attend to historical context. Specifically, the template in frame remains mostly static (or slowly updated multiple templates are used), whereas the search region tokens in frame t accumulate knowledge from frames {1...., t 1}. This discrepancy can degrade matching accuracy if not handled carefully.
We therefore introduce Stream-Specific LoRA Adapters. Each input stream s-template, search,, search,, and forth-is assigned its LoRA offset:
E ttemplate, search,, search,, }.
(12)
Concretely, for the query projection,
Q X. We + AW?).
(13)
where X, is the set of tokens from stream s, W is the frozen backbone weight, and A AW? is the learnable low-rank offset for that stream. Analogous offsets AWK AW! attach to WK WV.
Such design brings the following benefits:
It preserves shared representation space via the unmodified backbone.
It applies minimal, stream-dependent adaptations for each set of tokens.
It allows easy extension by adding new LoRA modules for additional frames.

---

Table I: Benchmarking tracker four large-scale challenging datasets. For GOT- 10k evaluation, all the methods follow the one-shot protocol, training only on the train split GOT-1Ok. Bold indicates the best results and underline indicates the second-best.
LaSOT 16
TNL2K
34
GOT-1Ok 20
VastTrack
30
Tracker
SUC
P Norm
P
SUC
AO
SRo.5
SRo.75
SUC
TransT2se
g
64.9
73.8
69.0
50.7
51.7
67.1
76.8
60.9
29.9
25. ,4
AutoMatchess
44
58.3
59.9
47.2
43.5
65.2
76.6
54.3
28.8
26.6
S2ARK320
41
67.1
77.0
78.1
64.1
33.4
KeepTrackaas
27
67.1
77.2
70.2
MixFormeras4
9
70.1
79.9
76.3
2BT224
38
66.7
71.1
70.4
80.8
64.7
AiATracK320
17
69.0
73.8
69.6
80.0
SimTrackası
6
70.5
79.7
55.6
55.7
69.8
78.8
66.0
34.4
30.3
OSTrack384
43
71.1
81.1
77.6
55.9
56.7
73.7
83.2
70.8
33.6
31.5
Swin Tracksa3
25
71.3
76.5
55.9
57.1
72.4
80.5
67.8
33.0
30.3
DropTrack2s
36
71.8
81.8
78.1
56.9
57.9
75.9
86.8
72.0
37.0
36.5
SeqTrack-B256
7
69.9
79.7
76.3
56.4
74.7
84.7
SeqTrack-Log4
7
72.5
81.5
79.3
57.8
74.8
81.9
72.2
39.6
40.2
ARTrack-B2se
35
70.4
79.5
76.6
59.8
73.5
82.2
70.9
ARTrack-Lass
35
73.1
82.2
80.3
60.3
78.5
87.4
77.8
35.6
32.4
ARTrackV2-B..
35
71.6
80.2
77.2
59.2
75.9
85.4
72.7
ARTrackV2-Los4
35
73.6
82.8
81.1
61.6
79.5
87.8
79.6
CiteTrackerasa
[23
69.7
78.6
75.7
57.7
59.6
74.7
84.3
73.0
ROMTrackse1
4
71.4
81.4
78.2
74.2
84.3
72.4
37.0
36.1
MixViT-L-as4
10
72.4
82.2
80.1
75.7
85.3
75.1
39.5
39.8
ODTrack-
45
74.0
84.2
82.3
61.7
87.2
77.3
LoRAT-B B 224
24
71.7
80.9
77.3
58.8
61.3
72.1
81.8
70.7
38.7
37.8
LoRAT- L378
24
75.1
84.1
82.0
62.3
67.0
77.5
86.2
78.1
43.9
45.8
LoRATv2-B2aa
72.0
81.3
77.9
59.6
62.7
74.7
84.7
72.7
39.1
38.7
LoRATv2-B357
74.3
83.6
80.9
60.9
64.9
75.8
85.7
75.4
40.6
40.8
LoRATv2-Laza
74.4
83.8
81.2
61.8
66.7
76.9
86.3
76.4
42.0
43.3
LoRATv2-L97a
76.1
83.1
62.4
67.7
78.2
86.8
79.1
44.2
3.5 Two-Phase Progressive Training
Inspired by curriculum learning (2). adopt two-phase approach to gradually introduce multi- frame complexity. This strategy not only reduces memory requirements compared to training on multiple frames from scratch but also consistently yields better final accuracy in practice.
Phase 1: Single Template Single Search Region. We follow LoRAT 24 by training only two LoRA adapters, one for the template and one for a single search region, while keeping the ViT backbone frozen. This phase converges quickly and requires minimal memory.
Phase 2: Extending to an Additional Search Region, Next, we fix all previously learned parameters and attach new LoRA adapter for the second search region. Only this newly added adapter is updated, enabling efficient multi-frame modeling without large-scale retraining. The final model supports both single-frame (for speed) and multi-frame inference (for improved robustness), with additional frames accommodated via further adapters as needed.
4 Experiments
4.1
Implementation Details
All models are trained on 4XNVIDIA GeForce RTX 4090 GPUs and evaluated NVIDIA GeForce RTX 5090 GPU. Basically, we follow LoRAT 24 for fundamental settings (e.g. training datasets, optimization hyperparameters).
Model Variants. We develop four LoRATv2 variants using ViT-Base (B) and ViT-Large (L.) back- bones, trained progressively described in Sec. 3.5 The specific configurations are:

---

LLRRATv-BB/-2224 (Phase Models);
Backbone: ViT-Base/ViT-Large
Template (z): 224 x 224
Search Region (x)) 224 224
LORATTV-BBL-3378 (Phase 2 Models):
Backbone: WVT-Base/ViT-Larg
Template (z): 224 224
Past Search Region (x*): 224 224
Current Search Region (x²): 378 x 378
Training. We use LaSOT [6]. TrackingNet 28 GOT-1OK 20) (excluding 1k sequences as (2I). and COCO (26] for training. For the GOT-1OK evaluation, models are trained exclusively on the GOT-10k training split.
Phase 1 (-224 variants): Models are trained for 170 epochs (131,072 iterations/epoch) on (z, x pairs. The template (z) and search region x' are sampled from the same video (up to a 100-frame gap), with strong crop jitter applied x². The ViT backbone (DINOv2 pre-trained (2 12] is frozen; two sets of LoRA modules (rank 64), one for the template stream and one for the search region stream, are trained.
Phase 2 (-378 variants): Training continues for an additional 170 epochs on (z, (2,x², ,x)x) triplets. The template randomly sampled from video; x x² are sampled from the same video (up to 100-frame gap) with strong crop jitter. jiter. The backbone and previously trained LoRA modules remain frozen. A new set of extra LoRA modules (rank 64) and a corresponding prediction head are introduced exclusively for the x² stream.
Inference. During inference, LoRATv2 variants leverage frame-wise causal attention with KV caching. The initial template is encoded only once, and its key/value embeddings are cached to prevent re-comuutaiion for past frames. Subsequent search regions are cropped around the prior bounding box (area factor 4 for x 5 for x²).
Phase / (-224 variants): Tracking performed using only the x stream.
Phase 2 (-378 variants): By default, predictions are derived from the high-resolution x² stream. To maintain temporal context on the past frames, the Key-Value (KV) caches for the x stream are conditionally updated. This update is triggered when a prediction's 's classification score exceeds confidence threshold of 0.9. Upon this condition, new, smaller search region, with dimensions identical to the Phase 1 input, is cropped around the confident prediction and processed to refresh the x' stream's ' cached embeddings.
4.2 State-of-the-Art Comparison
We compare our LoRATv2 with recent Transformer-bases trackers on four challenging benchmarks, following their official evaluation protocols. Tab. reports the results.
LaSOT [66] is large-scale benchmark containing 280 long-tern test videos. From Tab, and Tab. 2 our smallest variant, LoRATv2-BB achieves Success (SUC) of 72.0% at 713 fps. Meanwhile, LoRATv2-L-377 sets new state the art at 76. 1% SUC, outperforming the previous best LoRAT-
L378
24 (75.1%).
TNL2K is a recently introduced tracking dataset comprising 700 test videos. Our LoRATv2-L- 378 attains 62.4% SUC, improving upon LoRAT-L-378 (62.3%) and confirming the advantage of our temporal modeling capability.
GOT-10k 20 consists of 180 test videos and enforces strict protocol requiring training exclusively on its designated training split. shows that LoRA7v2-L-377 achieves high AO of 78.2%. approach- ing ARTrackV2-L (79.5%), thus demonstrating our method's strong generalization when limited to a single dataset.
VastTTrak (30 is large-scale benchmark featuring extensive object categories (2,115 classes) to facilitate the development of more general and robust trackers. As shown in Tab. [ LoRATv2-L-33

---

Table Comparison on efficiency with titte-of-the-eeee art Transformer trackers. The values in parenthe- ses for the params of our trackers represent LoRA and extra components (token type embedding and head), respectively. The speed of all trackers was re-evaluated on our machine.
Tracker
Speed (fps)
MACs (G)
#Params (M)
OSTrack256
43
244
21.5
0STrack384
4
165
48.3
SeqTrack- B256
124
66
89
LeqTrack-Laap
-384
22
524
309
ROMTrack2se
224
35
92
ROMTracK3s4
141
78
92
LoRAT-BE
24
546
30
99 (11, 2)
LoRAT-BE B 378
24
401
97
99 (11, 2)
LoRAT-L22
24
255
103
336 (28, 4) 336 (28, 4)
LoRAT-La7s
24
167
325
LoRATv2-B222
713
25
1 110 (22, 2)
LoRATv2-B378
425
81
123 (11, 2)
LoRATv2-L224
288
85
364 (56, 4)
LoRATv2-L378
202
268
396 (28, 4)
Table 3: Ablation study on Frame-Wise Causal Attention (FWCA) vs. Fully Self-Autention (FSA), and Stream-Specifie LoRA Adapters (SSLA) vs. Shared LoRA. Performance metries are SUC
(Success) and P (Precision) on LaSOT [6] and VastTrack [30]. FPS and MACs denote efficiency.
LaSOT
VastTrack
Variants
Attention
LoRA Sctup
FPS
MACs (G)
SUC (%)
P (%)
SUC (%)
P (
Based LoRATv2-Baz (Single- Search Region):
Baseline
FSA
Shared LoRA
71.9
77.6
39.0
38.3
467
FSA
SSLA
72.0
77.3
38.8
38.0
452
FWCA
Shared LoRA
71.7
77.6
38.7
38.0
713
25
@ Our Model
FWCA
SSLA
72.0
77.9
39.1
38.7
713
25
Based LoRATv2-Bas B378 (Two-Search--eeee Region):
@ Bascline
FSA
Shared LoRA
73.7
79.9
39.7
39.6
250
137
FSA
SSLA
73.2
78.7
39.3
39.0
237
137
FWCA
Shared LoRA
425
Our Model
FWCA
SSLA
74.3
80.9
40.6
40.8
425
81
achieves 44.2% SUC, surpassing the previous best LoRAT-L97s (43.9%) and demonstrating robust performance across diverse categories.
Efficiency Comparison. Tab. 2 compares the efficiency of LoRATv2 and leading Transformer trackers. VoRATv2 demonstrates significant improvements in both MACs and practical inference speed (FPS) compared to its predecessor, LoRAT, and other contemporary trackers. For instance, LoRATv2-B-22 achieves 713 fp3 with only 25G MACs, outperforming LoRAT-B- ---2 (546 30G MACs) in both speed and computational load, while also achieving higher accuracy on LaSOT (Tab. and Tab. 2) This trend holds for larger models well: LoRATv2-L-378 runs 202 202 with 268G MACs, compared to AORAT-L-378 at 167 with 325G MACs. The improved FPS, despite the multi-frame processing and KV cache management, highlights the effectiveness of architectural optimizations and frame-wir wise causal attention in reducing redundant computations.
4.3
Ablation Studies
We conduct comprehensive ablation studies (Tab. 3) on key components of LoRATv2 using the ViT-Base backbone. All results are reported on LaSOT and VastTrack datasets, along with FPS and MACs. For more ablation experiments, please touch the appendix.

---

Table Ablation study on the training strategy. We compare our proposed Two-Phass Progressive Training against standard One-Phase
approach. Performance is evaluated on LaSOT and VastTrack.. Training resource consumption (Peak Memory, Time) is also reported.
Training Strategy
Schedule
LaSOT SUC (%)
VasiTrack SUC (%)
Peak Mem. (GB)
Time (h)
One-Phase
1x
69.7
36.1
17.5
16
One-Phase
2x
71.3
39.0
17.5
32
Two-Phase (Ours)
Ix + 1x
74.3
40.6
15.6
19
Frame-Wise Causal Attention (FWCA) and SSLA. Table 3 analyzes the interplay between the attention mechanism (FWCA vs. standard Fully Self-Attention, FSA) and the LoRA configuration (Stream- Speccffc vs. Shared). The results reveal strong synergy between FWCA and SSLA. When paired with SSLA, FWCA delivers substantial efficiency gains without compromising and often improving-accuree accuracy. For instance, our final single-frame model ( FWCA+SSLA) nearly doubles the speed of its FSA counterpart (2, 713 vs. 452 FPS) while halving the MACs (25 vs. 49G) and achieving superior performance on VastTrack. This advantage becomes more critical in the multi-frame setting, where the FWCA+SSLA model (@) surpasses its FSA equivalent (@) in both accuracy (+1.1% SUC on LaSOT) and efficiency.
The necessity of SSLA is most evident when combined with FWCA. While a shared LoRA configura- tion converges for a single search frame (3), suffers from training instability and fails to converge in the more complex two-frame setup (Z) In contrast, SSLA provides the necessary adaptability to manage the informational asymmetry introduced by the causal structure, ensuring robust training and optimal performance. Conversely, with FSA, the benefits of SSLA are less clear and can be slightly detrimental (6 vs. (5,, as the symmetric nature of full attention does not require such stream-specific adaptation.
Two-Phase Progressive Training. To validate curriculum- training strategy, compare it against a standard "one-phase" approach where the multi-frame tracker is trained from scratch. As detailed in Table our two-phase method superior in both effectiveness and efficiency. The model trained progressively achieves a LaSOT SUC of 74.3%, significantly outperforming the one-phase model even when the latter is trained for twice as long (71.3%). This result highlights the benefit of mastering simpler task before progressing to a more complex one. Furthermore, our approach more essurce-conscious. reducing training time by 40% (19 vs. 32 hours) and lowering peak GPU memory consumption compared to the extended one-phase baseline. This analysis confirms that progressive training not merely heuristic but more effective and efficient paradigm for developing complex temporal trackers.
Conclusion
LoRATv2 significantly advances cillt--frame object tracking by introducing frame-wise causal at- tention with KV caching, Stream-Specine LoRA Adapters, and progressive two-phase training strategy. These innovations lead to state-of-the-art performance with substantially improved computa- tional efficiency and a superior accuracy-FL.OP trade-off, as demonstrated multiple benchmarks. LoRATv2 offers a powerful and practical solution for real-time tracking with Transformers.
Acknowledgment
This work supported in part by the Natural Science Foundation of China (Grant No. No.32503323). This work was conducted while Liting Lin was a postdoctoral researcher at Pengcheng Laboratory. mentored by Yaowei Wang. Heng Fan participated in this research prior Yaowei Wang's affiliation with Harbin Institute of Technology and was not supported by any funds for this work.
References
[1] Bai, Y., Zhao, Z. Gong. Y., Wei, X., 2024. Artrackv2: Prompting autoregressive tracker where to look and how describe, in: CVPR.
10

---

[2] Bengio, Y., Louradour, J., Collobert, R.. Weston, J., 2009. Curriculum learning, in: ICML.
[3] Cai, W., Liu, Q. Wang, Y., 2024. Hiptrack: Visual tracking with historical prompts, in: CVPR.
[4] Cai, Y., Liu, J., Tang, J., Wu, G., 2023. Robust object modeling for visual tracking, in: ICCV.
[5] Cao, Z., Huang, Z., Pan, L.,, Zhang, S., Liu, Z., Fu, C., 2022. Tetrack: Temporal contexts for aerial tracking. CVPR.
Chen, B., Ii, P., Bai, I.., Qiao, I.., Shen, Li, B., Gan, W., Wu, W., Ouyang, W. 2022. Backbone is all your need: A simplified architecture for visual object tracking. ECCV.
[7] Chen, X., Peng. H., Wang, D., Lu, H., Hu, H., 2023. SeqTrack: Sequence to sequence learning for visual object tracking, in: CVPR.
[8] Chen, X., Yan, B., Zhu, J., Wang, D., Yang. X., Lu, H., 2021. Transformer tracking, in: CVPR.
[9] Cui, Y., Jiang, C., Wang. L., Wu, G., 2022. MixFormer: End-to-end tracking with iterative mixed attention, in: CVPR.
[10] Cui, Y., Jiang, C., Wu, G., Wang, L., 2024. MixFormer: End-to-end tracking with iterative mixed attention. IEEE Transactions Pattern Analysis and Machine Intelligence 1-18.
[11] Dai, Z., Yang. Z.. Yang. Y., Carbonell, J., Le, Q.V., Salakhutdinov, R., 2019. Transformer-xl: Attentive language models bcyond a fixed-length context, ACL.
[12] Dareet, Oquab, M., Mairal, J., Bojanowski, 2024. Vision transformers need registers, in; ICLR.
[13] Dong, Cordonnier. J.B., Loukas, A., 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, in: ICML.
[14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2021. An image is worth 16x16 words: Transformers for image recognition at scale, in: ICLR.
[15] Duan, Y., Wang, W., Chen, Z., Zhu, X., Lu, L., Lu, T., Qiao, Y., Li, H., Dai, J., Wang, W., 2025. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures, in: ICLR.
[16] Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., Ling, H., 2019. LaSOT: high-quality benchmark for large-scale single object tracking, in: CVPR.
[17] Gao, S., Zhou, C., Ma, C., Wang, X., Yuan, J., 2022. AiATrack: Attention attention for transformer visual tracking, in: ECCV.
[18] He, K., Zhang, C., Xie, S., Li, Z., Wang. Z., 2023. Target-aware tracking with long-term context attention, in: AAAI.
[19] Hu, E.J., Shen, Y.. Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, 2022. LoRA: Low-rank adaptation of large language models, in: ICLR.
[20] Huang, L., Zhao, X., Huang. K., 2021. Got-10k: large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1562 1577.
[21] Kristan, M. Matas, J., Leonardis, A., Vojif. T., Pfugfelder, R., Fernández. G., Nebehay. G., Porikli, F.. Čehovin, L.. 2016. A novel performance evaluation methodology for single-target trackers. IEEE Transactions Pattern Analysis and Machine Intelligence 38, 2137-2155.
[22] Li, W., Li, Z., Yang. X., Ma. H. 2023a. Causal-vit: Robust vision transformer by causal intervention. Engineering Applications Artificial Intelligence 126, 107123.
[23] Huang, Y. He, Z., Wang, Y., Lu, H., Yang. M.H., 2023b. CiteTracker: Correlating image and text for visual tracking, in: ICCV.
[24] Lin, L., Fan, H., Zhang, Z., Wang, Y., Xu, Y., Ling, H., 2024, Tracking meets lora: Faster training, larger model, stronger performance, in: ECCV.
[25] Lin, L., Fan, H. Zhang, Z., Xu. Y., Ling, H., 2022. SwinTrack: A simple and strong baseline for transformer tracking, in: NeurIPS.

---

[26] Lin. T.Y.. Maire. M.. Belongie, S., Hays, J., Perona, P. Ramanan, D., Dollár, P., Zitnick, C.L., 2014. Microsoft COCO: Common objects in context, in: ECCV.
[27] Mayer, C., Danelljan, M., Paudel, D.P., Van Gool, L.,, 2021. Learning target candidate association to keep track of what not track, in: ICCV.
[28] Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., Ghanem, B., 2018. TrackingNet: A large-scale dataset and benchmark for object tracking in the wild, in: ECCV.
[29] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al., 2024. DINOv2: Learning robust visual features without supervision, in: TMLR.
[30] Peng, I.., Gao, Liu, X., Li, W. Dong, S., Zhang, Z., Fan, H., Zhang, L., 2024. Vasttrack: Vast category visual object tracking, in: NeurIPS.
[31] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., al., 2018. Improving language understanding by generative pre-training.
[32] Tian, K., Jiang, Y., Yuan, Z., Peng. B., Wang, I.., 2025. Visual autoregressive modeling: Scalable image generation via nexl-scale prediction, in: NeurIPS.
[33] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.,, Polosukhin, 2017. Attention is all you need, in: NIPS.
[34] Wang, X., Shu, X., Zhang, Z., Jiang, B., Wang. Y., Tian, Y., Wu, 2021. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark, CVPR.
[35] Wei, X., Bai, Y., Zheng, Y. Shi, D., Gong, Y., 2023. Autoregressive visual tracking, in: CVPR.
[36] Wu, Q. Yang. T., Liu, Z., Wu, B., Shan, Y., Chan, A.B., 2023. DropMAE: Masked autoencoders with spatial-attention dropout for tracking tasks, in: CVPR.
[37] Xie, F. Chu, L., Li, Lu, Y., Ma. C., 2023. Videotrack: Learning to track objects via video transformer, CVPR.
[38] Xie, F., Wang, C., Wang, G., Cao, Y., Yang, W., Zeng, W., 2022. Conrelaioonawarce deep tracking, in: CVPR.
[39] Xie, J., Mao. W., Bai, Z.. Zhang, D.J., Wang, W. Lin, K.Q., Gu, Y., Chen, Z., Yang. Z., Shou. M.Z., 2025. Show-o: One single transformer to unify multimodal understanding and generation, in: ICLR.
[40] Xie, J.. Zhong, B., Mo. Z., Zhang, S.. Shi, L., Song. S., Ji. R., 2024. Autoregressive queries for adaptive tracking with spatio-tempol transformers, CVPR.
[41] Yan, B., Peng, H., Fu, J., Wang, D., Lu, H., 2021. Learinng seatio-temporal transformer for visual tracking, ICCV.
[42] Yang, X., Zhang, H., Qi, G., Cai, J., 2021. Causal attention for rision-language tasks, in: CVPR.
[43] Ye. B., Chang, H., Ma. B., Shan, S., Chen, X., 2022. Joint feature learning and relation modeling for tracking: A one-stream framework, in: ECCV.
[44] Zhang, Z., Liu, Y., Wang, X.. Li, B., Hu, w. 2021. Learn to match: Automatic matching network design for visual tracking, in: ICCV.
[45] Zheng, Y., Zhong, B., Liang, Q. Mo. Z.,, Zhang, S., Li, X., 2024. Odtrack: Online dense temporal token leamning for visual tracking, in: AAAI.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the training and evaluation details our method, including the
We also will release the code upon paper publication.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer:
[No]
Justification:
We will release the code upon paper publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpaiisssoooooocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training and test details in the experiment section, and provide full details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We follow the standard evaluation protocols of visual trackers. Empirically, the metrics like SUC of LaSOT lie range of +0.1 across different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We provide the details of compute resources in the main paper and the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neuripps.coo http:///wrrr..cccoaaaaccccccccaa
Answer: [Yes]
Justification:
We read and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work introduces new method of visual tracking, which a fundamental computer vision problem, thus, the societal impact is well-known.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We credited the source codes used in our code base.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide README for the to be released code base.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19


Reviews:
[{"review_id": "G5gZOfPWTk", "reviewer": "Reviewer_ZLNb", "summary": "First, this paper addresses the computational complexity of standard attention operations in LoRAT v1. Specifically, when processing high-resolution or multi-frame inputs, the standard attention mechanism incurs substantial computational overhead during inference. To resolve this, the paper reframe tracking as an autoregressive sequence prediction task and propose a frame-wise causal attention mechanism, where each frame exclusively attends to preceding frames. This approach is further accelerated through key/value caching, significantly reducing inference latency.\n\nSecond, while causal attention mitigates computational costs, it introduces feature inconsistency across frames due to varying receptive fields. The paper counteract this by designing Stream-Specific LoRA Adapters, which allocate dedicated LoRA modules to distinct input frames. This ensures parameter-efficient feature alignment while maintaining temporal coherence.\n\nThird, to scale the model for multi-frame inputs, a two-phase progressive training strategy (inspired by curriculum learning) is introduced. Compared to direct end-to-end training on multi-frame data, this phased approach reduces training complexity.", "questions": "1. The Model Variants section in 4.1 is unclear. The model's input settings for different phases and resolutions should be clearly specified.\n\n2. Table 2 shows the parameters of different models in terms of efficiency. The parameters of LoRAT are consistent across different resolutions, but those of LoRAT v2 are not. Could you please explain the reason for this?\n\n3. Could you please explain the reasons for the failure of ② in Table 3 to converge as much as possible?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "1. Quality: The paper presents technically sound methodologies with substantial experimental validation. However, additional experiments are required to strengthen the conclusions:\n\n    • Comparative analysis of direct multi-frame training: Quantitative results (accuracy/FPS) and resource consumption (GPU hours/memory) versus the proposed two-stage strategy, demonstrating the necessity of phased training.\n\n    • Scalability validation of SSLA: Performance evaluation with more frame inputs to substantiate Stream-Specific LoRA Adapters' efficacy in large-scale scenarios.\n\n2. Clarity While generally well-structured, the manuscript requires little revisions:\n\n- The type emb in Figure 2 comes from the LoRAT v1 paper, but it is not mentioned in the revisiting section of this paper, so it needs to be briefly explained again.\n    \n\n- Table 3 conducts ablation experiments on model design. First, as a personal suggestion, if the final model design is used as the baseline for the experiment, the Variants column should use “-” to indicate changes in model design instead of “+”. For example, “+ w/o SSLA” should be “- SSLA”. Second, the ablation designs for Variants in phases 1 and 2 should be as consistent as possible.\n    \n\n3. Significance: The work significantly addresses the computational bottleneck of ViT-based trackers during inference, particularly for high-resolution/multi-frame use cases.\n\n4. Originality: The core contributions are primarily engineering-driven enhancements rather than theoretical innovations:\n\n- Frame-wise causal attention: While effective, this mirrors prior autoregressive designs and shares operational similarities with S-MAM in MixFormer.\n    \n- Key-value caching: Positioned as an implementation optimization (also seen in MixFormer's public codebase) rather than a novel methodological contribution.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. Regarding the originality of the FWCA method, although the authors have further elaborated on the differences between FWCA and S-MAM, I personally find the distinction not sufficiently clear. However, considering its differentiated application in AR tasks and the inclusion of comparative experiments in the manuscript, I retain partial reservations. Therefore, I will maintain my original score.", "rebuttal": "Thank you for your detailed and constructive feedback. We address your questions and concerns below.\n\n**Q1: [Comparative analysis of training strategies]**\nProvide a comparative analysis of direct multi-frame training versus the proposed two-stage strategy, covering quantitative results (accuracy/FPS) and resource consumption (GPU hours/memory) to demonstrate the necessity of phased training.\n\n**A1:** \nWe appreciate the reviewer's valuable suggestion. To highlight the necessity and advantages of our progressive training strategy, we compare it with a standard \"from-scratch\" multi-frame training baseline. All methods are based on LoRATv2-B-378, the only difference lies in the training strategy.\n\n|Training Strategy|Schedule|LaSOT SUC (%)|VastTrack SUC (%)|Inference FPS|Peek Memory (GB)|Time (h)|\n| :- | :-| :-:| :-: |:-:|:-:|:-:|\n|One-Phase|1x|69.7|36.1|172|17.5|16|\n|One-Phase|2x|71.3|39.0|172|17.5|32|\n|**Two-Phase (Ours)**|**1x + 1x**|**74.3**|**40.6**|**172**|**15.6**|**19**|\n\nIn this experiment:\n*   **\"One-Phase (1x)\"**: Direct two-search-frame training for 170 epochs.\n*   **\"One-Phase (2x)\"**: Doubles to 340 epochs.\n*   **\"Two-Phase (1x+1x)\"** is our proposed method: 170 epochs for the single-search-frame model, followed by an additional 170 epochs for the two-search-frame model.\n\nThe results clearly validate the advantages of our approach:\n\n1.  **Higher Accuracy:** Our two-phase strategy achieves **74.3% SUC** on LaSOT, significantly outperforming the from-scratch model (71.3% SUC for \"One-Phase (2x)\"). This highlights the effectiveness of the curriculum-like learning process, which guides the model evolving from a simpler to a more complex task.\n\n2.  **Greater Efficiency:** Compared to the \"One-Phase (2x)\" baseline, our method requires substantially less training time (**19 hours vs. 32 hours**) and consumes less peak GPU memory (**15.6 GB vs. 17.5 GB**), primarily because fewer parameters are trainable in the second phase.\n\nWe will include this comparative analysis in the revision. Again, thanks!\n\n---\n**Q2: [Scalability of SSLA]**\nProvide a performance evaluation with more frame inputs to substantiate the scalability and efficacy of Stream-Specific LoRA Adapters in large-scale scenarios.\n\n**A2:** \nWe appreciate this insightful question regarding the scalability of our framework.\n\nTo empirically validate the scalability, we progressively extended our LoRATv2-B-378 model to incorporate up to three past search frames. Each new frame was managed by a new, progressively trained SSLA module.\n\nThe results are presented below:\n\n|# Cached Past Frames (Template + Search)|LaSOT SUC / P (%)|VastTrack SUC / P (%)|MACs (G)|FPS|\n|:-|:-:|:-:|:-:|:-:|\n|1 + 1 (LoRATv2-B-378)|74.3 / 80.9|40.6 / 40.8|81|172|\n|1 + 2|74.6 / 81.2|41.8 / 42.7|84|170|\n|1 + 3|75.0 / 81.5|42.1 / 42.9|88|169|\n\nThis experiment reveals two key advantages of our approach:\n\n1.  **Consistent Performance Gains:** Tracking accuracy improves consistently as more frames are added. This demonstrates that our model, guided by the specialized adaptations from SSLA, can effectively leverage longer temporal history for more robust predictions, with particularly notable gains on the challenging VastTrack dataset.\n\n2.  **Computational Efficiency:** The computational cost grows very slowly. Scaling from one to three past search frames increases MACs by only ~9% (from 81G to 88G), with a minimal impact on FPS. This high efficiency stems from our architecture: the most computationally intensive ViT layers (Attention and MLP linear projections) process only the newest search frame's tokens. The complexity of the frame-wise causal attention mechanism grows linearly with cached frames, and its contribution to total MACs is minor.\n\nWe will include the above analysis and results in reivions. Thanks again!\n\n---\n**Q3: [Clarity of “type emb.”]**\nThe \"type emb\" in Figure 2 is from LoRAT v1 but is not mentioned in the revisiting section.\n\n**A3:** Thanks for this careful comment. We will add a clear explanation of this component in the revision.\n\n---\n**Q4: [Clarity of Table 3]** Table 3 conducts ablation experiments on model design. First, as a personal suggestion, if the final model design is used as the baseline for the experiment, the Variants column should use “-” to indicate changes in model design instead of “+”. For example, “+ w/o SSLA” should be “- SSLA”. Second, the ablation designs for Variants in phases 1 and 2 should be as consistent as possible.\n\n**A4:**\nThanks for suggestions. We'll update Table 3 to \"-\" notation and ensure Phase 1/2 consistency.\n\n---\n**Q5: [Originality of FWCA and KV Cache]**\nThe core contributions are primarily engineering-driven enhancements rather than theoretical innovations: Frame-wise causal attention: While effective, this mirrors prior autoregressive designs and shares operational similarities with S-MAM in MixFormer. Key-value caching: Positioned as an implementation optimization (also seen in MixFormer's public codebase) rather than a novel methodological contribution.\n\n**A5:**\nThanks for this insightful comment.\n\nOur Frame-Wise Causal Attention (FWCA) builds on autoregressive principles to enable efficient temporal modeling without recomputing historical frames. \nHowever, autoregressive models are prone to error accumulation[1]: early prediction mistakes propagate and amplify over time, especially during early training phases. \nTo specifically mitigate this issue, we introduced novel Stream-Specific LoRA Adapters (SSLA) and the two-phase progressive training strategy, which together ensure stable convergence and robust temporal modeling.\n\nThis design sets our approach apart from methods like S-MAM in MixFormer. While S-MAM prunes target-to-search cross-attention and attention among target templates, effectively treating multiple templates as a reference bank for localization, our Frame-Wise Causal Attention establishes a genuine autoregressive history. By sequentially processing frames, it captures the target's evolving appearance and state, which strengthens tracking robustness and paves the way for future advancements in motions-aware or occlusions-aware trackers. As evidenced in our experiments, this design leads to superior performance compared to S-MAM.\n\nAlthough KV caching serves as an known optimization to accelerate autoregressive inference, akin to implementations in other codebases, our key innovation lies in integrating it with SSLA and progressive training to address error accumulation.\n\nAgain, we thank the reviewer for these helpful comments. We will include the above explanations and more discussions in revision to clarify the difference of our method with other methods. Thanks!\n\n[1] Yin, T., Zhang, Q., Zhang, R., Freeman, W. T., Durand, F., Shechtman, E., & Huang, X. (2025). From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. In CVPR.\n\n---\n**Q6: [Clarity of Model Variants]** The \"Model Variants\" section (4.1) is unclear. The model's input settings for different phases and resolutions should be clearly specified.\n\n**A6:**\nWe apologize for the lack of clarity. We will revise it as follows:\n*   **LoRATv2-B/L-224 (Phase 1 Models):**\n    *   Backbone: ViT-Base/ViT-Large\n    *   Template ($\\mathbf{z}$): $224 \\times 224$\n    *   Search Region ($\\mathbf{x}^1$): $224 \\times 224$\n*   **LoRATv2-B/L-378 (Phase 2 Models):**\n    *   Backbone: ViT-Base/ViT-Large\n    *   Template ($\\mathbf{z}$): $224 \\times 224$\n    *   Past Search Region ($\\mathbf{x}^1$): $224 \\times 224$\n    *   Current Search Region ($\\mathbf{x}^2$): $378 \\times 378$\n\n---\n**Q7: [Parameters in Efficiency Table]**\nIn Table 2, the parameters of LoRAT are consistent across different resolutions, but those of LoRATv2 are not. Explain the reason for this.\n\n**A7:**\nThanks for this careful comment. The difference stems from a core design choice:\n*   **LoRAT** uses a *single* set of LoRA adapters whose weights are shared between the template and search streams. Thus, changing input resolution does not alter the number of trainable parameters.\n*   **LoRATv2**, with our proposed SSLA, uses *dedicated* LoRA adapters for each stream. The \"-378\" models introduce an additional stream for the second search frame ($\\mathbf{x}^2$), which requires its own new LoRA adapter and a new prediction head. This is the source of the increased parameter count for our larger, multi-search-frame models.\n\nWe will add a clear explanation of this point in Section 4.2 when discussing the efficiency comparison. Again, thanks!\n\n---\n**Q8: [Convergence Failure]** Explain the reasons for the failure of variant ② in Table 3 (FWCA with Shared LoRA) to converge.\n\n**A8**: The convergence failure is caused by the functional asymmetry introduced by FWCA, which cannot adequately model with a simple shared LoRA. If a shared LoRA is tasked with learning two functionally distinct adaptations using a single set of weights, it woule lead to unstable optimization. Our SSLA adapt such asymmetry in network structure.\n\nTo validate this, we conducted a detailed ablation study:\n\n|Attention|Training Method|Converged?|LaSOT SUC / P (%) |VastTrack SUC / P (%)|\n|:-:|:-:|:-:|:-:|:-:|\n|Self-Attn | Full Finetune | Yes | 71.2 / 76.9 | 39.0 / 38.4 |\n| Self-Attn | Shared LoRA | Yes | 71.9 / 77.6 | 39.0 / 38.3 |\n|Self-Attn|SSLA|Yes|72.0 / 77.3|38.8 / 38.0|\n|FWCA|Full Finetune|Yes|69.4 / 73.9|36.0 / 34.0|\n|FWCA|Shared LoRA|**No**|-|-|\n|**FWCA**|**SSLA**|**Yes**|**72.0 / 77.9**|**39.1 / 38.7**|\n\nThe results are revealing:\n*   With symmetric Full Self-Attention, all methods converge, as Shared LoRA is sufficient.\n*   With asymmetric FWCA, Shared LoRA fails entirely. Full fine-tuning can adapt with the asymmetry but results in suboptimal performance.\n*   Only the combination of **FWCA and SSLA** both converges reliably and achieves the best performance, demonstrating that SSLA is an essential component of our framework.\n\nWe will include the above clarification and results in revision. Again, thanks for this comment!"}, {"review_id": "gmUy1Md865", "reviewer": "Reviewer_DMUb", "summary": "The paper proposes LoRATv2, an efficient multi-frame Transformer tracker that combines three ideas: (i) frame-wise causal attention plus KV-caching to cut the quadratic cost of vanilla self-attention, (ii) stream-specific LoRA adapters that let template and search streams specialize while keeping the ViT backbone frozen, and (iii) a two-phase curriculum that first trains on single-frame inputs and then extends to multi-frame ones.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Well-targeted efficiency innovation. The frame-wise causal mask plus KV-cache gives a clean O(N) path length without redesigning the backbone.\n2. Stream-specific LoRA keeps training cost low and lets the authors reuse large off-the-shelf ViTs.\n3. The paper is well-written and easy to follow, with clear visualizations.  \n\nWeaknesses\n1. Table 1 needs fresher baselines.\n2. OSTrack speed anomaly in Table 2. Although OSTrack shows lower MACs than LoRATv2, its FPS is lower. \n3. Minor English issues. “state-of-the-art” appears both hyphenated and as “state of the art”; consider unifying.", "was_revised": true, "final_justification": "The author response has addressed most of my concerns; therefore, I will maintain my original rating.", "rebuttal": "We appreciate your positive assessment and helpful suggestions. We address your points below.\n\n**Q1: [Fresher Baselines in Table 1]** Table 1 needs fresher baselines.\n\n**A1:**\nThank you for the suggestion. We will update main comparison table (Table 1) in the revision by including more recent state-of-the-art trackers, as shown below:\n\n| Model | Source |LaSOT SUC / P(%) | TNL2k SUC / P(%) | MACs (G) |\n|:------------ | :---:  |       :---:     |      :---:      | :---: |\n| LMTrack$_{384}$ [1] | AAAI 2025 | 73.2 / 81.0 | - / - | 69 |\n| MambaLCT-384 [2] | AAAI 2025 | 73.6 / 81.6 | 58.5 / - | 58 |\n| LoRATv2-B-378 | Ours | 74.3 / 80.9 | 60.9 / 64.9  | 81 |\n||\n| ARPTrack-L$_{384}$  [3] | CVPR 2025 | 74.2 / 81.7 |  -  | - | \n| SeqTrack-L$_{384}$  + MPT [4] | ICML 2025 | 73.9 / 81.0 | 60.4 / 64.2 | - |\n| LoRATv2-L-378 | Ours | 76.1 / 83.1 | 62.4 / 67.7 | 268 |\n\n[1] Xu, C., Zhong, B., Liang, Q., Zheng, Y., Li, G., & Song, S. (2025). Less is More: Token Context-aware Learning for Object Tracking. In: AAAI.\n\n[2] Li, X., Zhong, B., Liang, Q., Li, G., Mo, Z., & Song, S. (2025). MambaLCT: Boosting Tracking via Long-term Context State Space Model. In: AAAI.\n\n[3] Liang, S., Bai, Y., Gong, Y., & Wei, X. (2025). Autoregressive Sequential Pretraining for Visual Tracking. In: CVPR.\n\n[4] Zhao, J., Chen, X., Yuan, Y., Felsberg, M., Wang, D., & Lu, H. (2025). Efficient Motion Prompt Learning for Robust Visual Tracking. In: ICML.\n\n---\n**Q2: [OSTrack Speed Anomaly in Table 2]** Although OSTrack shows lower MACs than LoRATv2, its FPS is lower. Explain this anomaly.\n\n**A2:**\nThanks for this careful comment. The discrepancy between theoretical MACs and practical FPS arises primarily from architectural differences affecting hardware utilization. OSTrack's \"Early Candidate Elimination\" strategy, while reducing MACs, introduces sequential logic and conditional branching that are less amenable to the massive parallelization of modern GPUs, thus increasing latency. And in our testing, it requires higher-precision (float32) arithmetic for stable performance, which is inherently slower. Furthermore, its convolutional head introduce higher FLOPs compared with our MLP-based head.\n\nLoRATv2's design is highly parallelizable and optimized for fused GPU kernels, leading to higher throughput (FPS) despite the slightly higher MAC count.\n\nWe will include the above clarification in revision. Thanks again!\n\n---\n**Q3: [Minor English Issues]** The term “state-of-the-art” appears in different forms.\n\n**A3:** Thank you for pointing this out. We will carefully proofread the entire manuscript to ensure consistent terminology in the whole manuscript."}, {"review_id": "MAzvLnmijj", "reviewer": "Reviewer_A63V", "summary": "This paper proposes LoRATv2, a novel framework for efficient multi-frame temporal modeling in Transformer-based trackers. The key contributions are Frame-Wise Causal Attention (FWCA), Stream-Specific LoRA Adapters (SSLA), and two-phase progressive training. Extensive experiments demonstrate state-of-the-art accuracy–FLOPs trade-offs across multiple tracking benchmarks.", "questions": "1.What level of performance can be achieved using only Phase 1 in low-compute scenarios?\n2.Does caching multiple frames in the KV cache lead to GPU memory or efficiency bottlenecks, or cause performance degradation?\n3.How do different LoRA rank values impact performance?", "limitations": "1.Due to the causal attention mechanism, reliance on historical frames in occlusion and fast-motion scenarios can lead to tracking errors and degrade tracking accuracy.\n2.As the number of historical frames increases, the KV cache size also grows, which may cause GPU memory pressure in long-sequence scenarios.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.Integration of causal attention into one-stream Transformer trackers with KV caching, greatly reducing temporal modeling cost.\n2.SSLA provides stream-specific adaptation without unfreezing the ViT backbone, incurring zero extra inference cost.\n3.The two-phase progressive curriculum reduces memory/time overhead in multi-frame training while improving long-term tracking.\n\nWeaknesses:\n1.The choice of LoRA rank is only reported in Section 4.1 without any ablation study to justify the selected rank size.\n2.The discussion of the KV cache is insufficient—for example, what is the maximum cache capacity and how much memory does it consume?\n3.The Phase 1 models ablation experiments (1 and 3) do not adequately demonstrate the effectiveness of the FWCA and SSLA modules. Could you please supplement with additional experiments involving self-attention + SSLA?", "was_revised": true, "final_justification": "The author response has addressed most of my concerns. So I tend to raise the final rating.", "rebuttal": "Thank you for your valuable feedback and insightful questions. We provide our responses below.\n\n---\n**Q1: [Ablation on LoRA Rank]** The choice of LoRA rank is reported without an ablation study to justify the selected rank size. How do different LoRA rank values impact performance?\n\n**A1:** Thanks for the comment. Our choice of LoRA rank $r=64$ was inherited from the hyperparameters used in the original LoRAT.\n\nTo verify if this is the suitable choice for the LoRATv2 framework, we conducted an ablation study on both our phase-1 (**LoRATv2-B-224**) and phase-2 (**LoRATv2-B-378**) models with various LoRA ranks. The results are presented below:\n\n| Model | LoRA Rank ($r$) | LaSOT SUC / P (%) | TNL2k SUC / P (%) | Trainable Parameters |\n| :------------ | :---:  |       :---:     |      :---:      | :---: |\n| LoRATv2-B-224 |  16    |   70.1 / 74.8   |   56.4 / 57.5   |  ~7.7M  |\n|               |  32    |   70.6 / 75.5   |   56.6 / 57.8   |  ~13.0M |\n|               | **64** | **72.0 / 77.9** | **59.6 / 62.7** | **~23.6M** |\n|               |  128   |   71.8 / 77.6   |   59.4 / 62.6   |  ~44.8M |\n| LoRATv2-B-378 |  16    |   73.0 / 79.0   |   60.1 / 63.3   |  ~5.0M |\n|               |  32    |   73.1 / 79.6   |   59.9 / 63.2   |  ~7.7M |\n|               | **64** | **74.3 / 80.9** | **60.9 / 64.9** | **~13.0M** |\n|               |  128   |   73.7 / 80.3   |   59.9 / 63.5    |  ~23.6M |\n\nThe results show that $r=64$ offers the best trade-off between accuracy and parameter efficiency for our model.\n\nWe will include the above analysis and results in revision. Again, thanks!\n\n---\n**Q2: [Details on KV Cache]** The discussion of the KV cache is insufficient. What is the maximum cache capacity, how much memory does it consume, and does caching multiple frames lead to bottlenecks or performance degradation?\n\n**A2:** \nThanks and we agree that a thorough explanation of the KV cache's practical impact is essential.\n\n*   **Cache Capacity and Memory Consumption:** The memory footprint of the KV cache is modest. For instance, caching the template and one past search frame in our **LoRATv2-B-378** model consumes only **54.4MB**, and for the larger **LoRATv2-L-378** model, it is **145.6MB**. This constitutes a small and manageable fraction of total GPU memory, which is negligible for most tracking scenarios. The cache capacity is not a hard limit and can be extended to retain more historical frames.\n\n*   **Bottlenecks and Performance Degradation:** Caching multiple frames does not lead to significant bottlenecks or performance degradation. This is for two primary reasons:\n    1.  **Optimized Kernels:** We leverage modern fused attention kernels (e.g., FlashAttention), which are specifically designed to handle such memory access patterns efficiently. These kernels effectively overlap memory operations (i.e., fetching the KV cache from memory) with computation, thereby hiding most of the memory access latency.\n    2.  **Net Computational Gain:** The KV cache provides a substantial net computational gain. Because attention is causal, embeddings for past frames are immutable—they are not influenced by any subsequent frames. They can be computed once, cached, and reused without ever needing to be re-encoded. This massive saving in computation far outweighs the minor cost of memory access.\n\n*   **Performance with More Frames:** To empirically demonstrate that our framework scales gracefully, we progressively extended our LoRATv2-B model to incorporate up to three past search frames. As shown in the table below, adding more frames to the cache consistently **improves** tracking accuracy while only minimally impacting efficiency, confirming that caching does not cause performance degradation.\n\n| # Cached Past Frames (Template + Search) | LaSOT SUC / P (%) | VastTrack SUC / P (%) | MACs (G) | FPS |\n| :--- | :---: | :---: | :---: | :---: |\n| 1 + 1 (LoRATv2-B-378) | 74.3 / 80.9 | 40.6 / 40.8 | 81 | 172 |\n| 1 + 2 | 74.6 / 81.2 | 41.8 / 42.7 | 84 | 170 |\n| 1 + 3 | 75.0 / 81.5 | 42.1 / 42.9 | 88 | 169 |\n\nThese results confirm that not only does caching more frames avoid bottlenecks, it also actively improves performance with a very graceful degradation in FPS. We will provide these details about the KV cache by including the above analysis in revision. Again, thanks!\n\n---\n**Q3: [More Ablations for FWCA and SSLA]** The Phase 1 ablation experiments do not adequately demonstrate the effectiveness of FWCA and SSLA. Could you please supplement with an experiment involving standard self-attention + SSLA?\n\n**A3:** \nThanks for this thoughtful comment. This experiment was conducted and included in our appendix but omitted from the main text due to space limitations. To provide a comprehensive view, we present the full ablation table below and will ensure the designs for Phase 1 and 2 in Table 3 are consistent in the revision.\n\n| Model | Attention | LoRA Setup | LaSOT SUC / P (%) | VastTrack SUC / P (%) | FPS | MACs (G) |\n| :--- | :--- | :---:| :---: | :---: | :---: | :---: |\n| LoRATv2-B-224 | Full Self-Attn | Shared LoRA | 71.9 / 77.6 | 39.0 / 38.3 | 163 | 49 |\n|  | Full Self-Attn | SSLA | 72.0 / 77.3 | 38.8 / 38.0 | 158 | 49 |\n|  | **FWCA** | **SSLA** | **72.0 / 77.9** | **39.1 / 38.7** | **249** | **25** |\n|\n| LoRATv2-B-378 | Full Self-Attn | Shared LoRA | 73.7 / 79.9 | 39.7 / 39.6 | 101 | 137 |\n|  | Full Self-Attn | SSLA | 73.2 / 78.7 | 39.3 / 39.0 | 96 | 137 |\n|  | **FWCA**  | **SSLA** | **74.3 / 80.9** | **40.6 / 40.8** | **172** | **81** |\n\nWith standard Full Self-Attention, the bidirectional self-attention are symmetric, so moving from a Shared LoRA to SSLA provides only a marginal benefit. However, the combination of **FWCA and SSLA** not only yields comparable or better accuracy but also dramatically improves efficiency (e.g., 249 vs. 163 FPS for the -224 model). For the two-search-frame model, the benefits are even clearer, with our final design (FWCA+SSLA) substantially outperforming all other variants in both accuracy and efficiency.\n\nWe will include more detailed analysis in revision. Again, thanks!\n\n---\n**Q4: [Performance of Phase 1 Model]**\nWhat level of performance can be achieved using only the Phase 1 model in low-compute scenarios?\n\n**A4:**\nThanks for this insightful comment. As suggested, to further demonstrate its effectiveness in low-compute scenarios, we conducted an ablation of our method for CPU inference. Specifically we trained a **LoRATv2-S-224** variant by accordingly adopting a smaller ViT-Small backbone. For a fair and direct comparison against leading lightweight trackers, all models were re-evaluated on the same consumer-grade CPU (Intel Ultra 7 265k).\n\n| Model Name | LaSOT SUC / P (%) | VastTrack SUC / P (%) | UAV123 SUC / P (%) | MACs (G) | CPU FPS |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| HiT-Small [1]          | 60.5 / 61.5 | - / -| 63.3 / -  | 1.13 | 60.3 |\n| MixFormerV2-S [2] | 60.6 / 60.4 | -/ - | 65.8 / 86.8 | 4.4 | 61.2 |\n| LoRATv2-S-224 | 69.9 / 75.3 | 36.9 / 35.2 | 71.5 / 91.4 | 6.9 | 40 |\n\nThe results show that our architecture also can be served as a highly competitive option within the domain of efficient visual tracking in low-compute scenarios. We will include the analysis in revision. Again, thanks!\n\n[1] Kang, B., Chen, X., Wang, D., Peng, H., & Lu, H. (2023). Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking. In: ICCV.\n\n[2] Cui, Y., Song, T., Wu, G., & Wang, L. (2023). MixFormerV2: Efficient Fully Transformer Tracking. In: NeurIPS."}]

Author rebuttal:

Meta-review: This paper addresses the computational complexity of standard attention in LoRAT v1, which becomes costly for high-resolution or multi-frame inputs. To mitigate this, tracking is reformulated as an autoregressive sequence prediction task, and a frame-wise causal attention mechanism is introduced, where each frame attends only to preceding ones. With key/value caching, inference latency is further reduced. Since causal attention can cause feature inconsistencies across frames, the authors propose Stream-Specific LoRA Adapters, assigning dedicated LoRA modules to each frame for efficient feature alignment and temporal coherence. Finally, a two-phase progressive training strategy inspired by curriculum learning is presented to scale the model to multi-frame inputs, reducing training complexity compared to direct end-to-end training.
The paper has received two accepts and one borderline accept. All reviewers noted that the method is novel and the experimental evaluation is sufficiently strong to support acceptance. Therefore, I recommend accepting this paper.
Decision: Accept (spotlight)