Where Does It Exist from the Low-Altitude: Spatial Aerial Video Grounding
Authors: Yang Zhan
Yuan Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Aerial Video
Spatial Video Grounding
Multi modality Spatio-temporal Interaction
Hierarchical Progressive  Decoder
Primary area: applications

Abstract: The task of localizing an object's spatial tube based on language instructions and video, known as spatial video grounding (SVG), has attracted widespread interest. Existing SVG tasks have focused on ego-centric fixed front perspective and simple scenes, which only involved a very limited view and environment. However, UAV-based SVG remains underexplored, which neglects the inherent disparities in drone movement and the complexity of aerial object localization. To facilitate research in this field, we introduce the novel spatial aerial video grounding (SAVG) task. Specifically, we meticulously construct a large-scale benchmark, UAV-SVG, which contains over 2 million frames and offers 216 highly diverse target categories. To address the disparities and challenges posed by complex aerial environments, we propose a new end-to-end transformer architecture, coined SAVG-DETR. The innovations are three-fold. 1) To overcome the computational explosion of self-attention when introducing multi-scale features, our encoder efficiently decouples the multi-modality and multi-scale spatio-temporal modeling into intra-scale multi-modality interaction and cross-scale visual-only fusion. 2) To enhance small object grounding ability, we propose the language modulation module to integrate multi-scale information into language features and the multi-level progressive spatial decoder to decode from high to low level. The decoding stage for the lower-level vision-language features is gradually increased. 3) To improve the prediction consistency across frames, we design the decoding paradigm based on offset generation. At each decoding stage, we utilize reference anchors to constrict the grounding region, use context-rich object queries to predict offsets, and update reference anchors for the next stage. From coarse to fine, our SAVG-DETR gradually bridges the modality gap and iteratively refines reference anchors of the referred object, eventually grounding the spatial tube. Extensive experiments demonstrate that our SAVG-DETR significantly outperforms existing state-of-the-art methods. The dataset and code will be available at here.

Where Does It Exist from the Low-Altitude Spatial Aerial Video Grounding
Yang Zhan School of Artificial Intelligence, Optics and Electronics (iOPEN)
Yuan Yuan"
Northwestem Polytechnical University
zhanyangnspu@gmail. com, y. yuyun1 www.:iieeeeeml..oommmm com
Abstract
The task of localizing an object's spatial tube based on language instructions and video, known as spatial video grounding (SVG), has attracted widespread interest. Existing SVG tasks have focused ego-centric fixed front perspective and simple scenes, which only involved very limited view and environment. However, UAV-based SVG remains underexplored, which neglects the inherent disparities in drone movement and the complexity of aerial object localization. To facilitate research in this field, we introduce the novel spatial aerial video grounding (SAVG) task. Specifically, we meticulously construct a large-scale benchmark, UAV-SVG, which contains target categories. To address the disparities and challenges posed by complex aerial environments, propose
2 million frames and offers 216 highly diverse
end-to-end transformer architecture, coined SAVG-DETR. The innovations are three-fold. 1) To overcome the computational explosion of self-attention when introducing multi-scale features, our encoder efficiently decouples the multi-modality and multi-scale spatio-temporal modeling into intra-scale multi-modality interaction and cross-scale visual-only fusion. To enhance small object grounding ability, we propose the language modulation module to integrate multi-scale information into language features and the multi- level progressive spatial decoder to decode from high to low level. The decoding stage for the lower-level vision-language features gradually increased. 3) To improve the prediction consistency across frames, we design the decoding paradigm based on offset generation. each decoding stage, we utilize reference anchors constrict the grounding region, use context-rich object queries to predict offsets, and update reference anchors for the next stage. From coarse to fine, our SAVG- DETR gradually bridges the modality gap and iteratively refines reference anchors of the referred object, eventually grounding the spatial tube. Extensive experiments demonstrate that SAVG-DETR significantly outperforms existing state-of-the- art methods. The dataset and code will be available at here
Introduction
Grounding objects with natural language in visual contexts is fundamental and important task in multi-modal understanding I 3]. Recently, the spatial vidco grounding (SVG) has drawn significant attention [4, 5). However, the SVG task has predominantly focused the ego-centric fixed front perspective and simple scene 6 8.. As shown in Figure the supplementary material, they only provide a very limited view and environment. This means that existing methods can only perform target localization in simple scenes on the ground. This overlooks another important application scenario: moving aerial platforms in the sky 9 10 [T],
As the low-altitude economy
"Corresoondng author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DAV-SV
Figure I: An overview of the UAV-SVG dataset. SAVG grounds the referred object's ss spatial tube in the complex aerial scene by a natural language query. UAV-SVG presents distinctive challenges, including camera motion, low resolution, illumination variations, aspect ratio variations, viewpoint changes, rotation, etc. More detailed analysis is in Sec. Bf the supplementary material.
takes off, many tasks currently need to be performed in the sky such as UAV-based goods delivery, traffic/security patrol, and scenery tours [1 14 [5]. Spatial acrial video grounding (SAVG) emerges as a groundbreaking task. We can use UAVs to localize specific objects in various scenarios from the sky view and obtain much more holistic grounding.
To date, existing benchmarks (VID-sentence 6. VidSTG (7). HC-STVG (8) have predominantly
been confined to small-scale scenarios (Figure 6 The number of objects within the video is limited,
and the referred object covers a significant portion of the frame image. SVG in moving aerial platforms is quite different, presenting unique technical challenges: 1) Most objects only contain few pixels (1-200 pixels or so). The approaches in natural scenes without multi-scale feature learning cannot deal with them effectively. 2) Aerial videos usually have a wide field of view and a large scene scale. containing dense and numerous objects. The irrelevant or confusing information increases significantly. 3) Objects may be subject a range of environmental disturbances, such as occlusion from trees, shadows cast by sunlight, and low light at night. The discriminability of the referred object diminishes significantly. 4) UAV may move rapidly, causing viewpoint changes between adjacent frames, and the object may also move quickly. It is hard to accurately and consistently ground objects. To foster our proposed SAVG task, we contribute new sizeable benchmark, named UAV-SVG. UAV-SVG uses the million-sca scale tracking dataset [9] as the video source. To guarantee the high quality, referring expressions are generated using a combination of manual annotation and the advanced Gemini model [16). As shown in Table [ UAV-SVG contains over 2 million frames, 17,820 video-query pairs, and 216 highly diverse object categories. As shown Figure D unlike ground-fixed or hand-held shooting sources, the outdoor acrial view encompasses vast areas.
18 9 20 usually follow three steps: 1) transformer encoder for multi-modality 3D spatio-temporal modeling, 2) transformer decoder for mining infor- mation from encoded vision-language features by learnable queries, and 3) prediction head for regressing queries to obtain the spatial tube. In the field of aerial object localization, most works 21 23 use multi-scale visual features. If we introduce multi-scal visual features into the existing encoder, the self-attention of computing multi-modality multi-scale 3D spatio-temporal features will be computationally expensive and unaffordable. Meanwhile, it is difficult to capture the object-related region details efficiently in the complete and lengthy multi-modality multi-scale sequence during the decoding stage, especially the small objects. Moreover, due to the complicated movement of UAVs, the prediction consistency across frames of the grounding model is more demanding.
Existing advanced end-to-end methods 117
To address aforementioned problems, we introduce a new end-to-end transformer architecture, termed SAVG-DETR. Our core consists of a multi-modality multi-scale spatio-temporal encoder for cross- modal cross-scale scale feature fusion and alignment, and a hierarchical progressive decoder for efficient spatial tube prediction. The,fist key design is that the vanilla into two branches: intra-scale multi-modality interaction with a single high-level scale and cross-scale visual-only fusion with different scales. Our encoder can capture conceptual entities on high-level features with richer semantic concepts and integrate more object details from low-level features, which is convenient for the subsequent decoder to localize the object. In addition, this decoupling
encoder is decoupled

---

Table 1: Comparison of spatial video grounding datasets, where 'Exp.' indicates the expression.
Videos
Frame Num.
Object Classes
Mocicn Classes
Language Num.
Vocab
Exp. Length
Train/aareest Partitio
Target Scene
View
Num.
Duraton
ViD=sentenee
7,654
99K
1
30
X
7,654
1,523
13.15
Animul & Vihick
Fixed/ Handicll&& & Frout
VIaSTG Jaw 6,924
7.1M
69.1
79
X
99,943
1,581
10.12
908/1000/10
Hamui Aninal
Fixed/ Handheldd Fromt
HC STVG heevrra
3,600
3M
31,4h
1
X
5,600
2,289
17.27
80%%/00%/204
Human Wild
Movie Clips
216
17,820
79%/
Moving Bird's-Eye
strategy avoids an explosion the computation of multi-modality multi-scale 3D spatio-temporal features. The second key design is multi-level progressive spatial decoder, which decodes from high to low level and gradually increases the number of decoding layers for lower-level features. We devise the multi-level language modulation module to integrate multi-scale information into language features. The spatial decoder utilizes multi-level language-vision features to guide queries to decode more relevant spatial information. The third key design is the decoding paradigm based offset generation. Unlike existing methods, we utilize reference anchors as positional embedding to constrict the grounding region, queries to predict offsets, and update reference anchors at each decoding stage. We design the query and position generator to yield context-rich object queries and initial reference anchor boxes. This paradigm improves the consistency of the prediction. Furthermore, we adopt larger auxiliary bounding boxes to calculate losses, which is more effective for small objects. To demonstrate the effectiveness of our approach, we conduct comprehensive ablation studies and benchmark many state-of-the methods.
Contributions: (i) We highlight the significance of deploying spatial video grounding in acrial scenes and introduce a challenging benchmark, UAV-SVG, characterized by unique properties and challenges that set it apart from existing datasets. (ii) To overcome the computational explosion, our encoder efficiently decouples the multi- modalityy and multi-scale spatio-temporia modeling into multi-modalhe interaction and cross- scale visual-only fusion. (iii) To enhance small object grounding, we propose the modulation module to integrate multi-scale information into language features and the multi-level progressive spatial decoder to decode from high level to low level. (iv) To improve the prediction consistency, we design the decoding paradigm based on offset generation. At each decoding stage, we use context-rich queries to predict offsets and update reference anchors for the next stage. (v) Extensive experiments show that our method significantly outperforms all baselines. Comprehensive ablation studies and detailed analyses provide new ideas and useful insights.
2 Related work
Image-Based Visual Grounding in Aerial. The visual grounding in aerial (24 25 26 is mainly focused on the satellite remote sensing scenario, such MGVLF 277 QAMFN 28 LPVA 22). RMSIN 30 In aerial scenes, images with a large field of view and complex spatial scales are often encountered. Existing methods propose a multi-granulan granularity visual language fusion module [27] multi-level feature enhancement decoder [29], multi-scale moosl alignment module 226 or a
rotated multi-scale interaction network [30] to capture remote sensing rision-languse features and achieve improved performance effectively.
Video-Based Referring Expression Comprehension. This task aims detect the unique object or region in each video frame using a phrase or expression that describes the target attribute. The earlier tracking-bass based approaches generate phrase-relevane region proposals BI 32 and transform the visual tracking framework into natural language tracking [33 34 32). . Detection- based methods
38 39] do not rely on visual region proposals and directly localize objects in each frame. Recent one-stage frameworks, like Co-Grounding (35), DCNet (36), ConFormer (38), and MILCGF- Net (39], have focused on temporal correlation, inter-frame correlation, fine-grained patch-word alignment, phrase-region alignment, and image-language inter-modality dense associations.
35 36
37
Video Object Grounding. This task aims to localize all objects in the video referred in the natural language query. The number of target boxes output in each frame may vary and is not limited to only one. The VOGNet framework [40] adopts self-attention with relative position encoding to model object relations. Subsequently, the weakly supervised video object grounding (WSVOG) [4] introduces context-aware object stabilizer module and cross-modal alignment knowledge transfer modules to achieve stable context learning. The UMA framework 42 considers rich contextual
3

---

Million Scale Data
Expression Gencration
You powerful drone sectsl image and video objec captioner. The entire cur
imape from top buttum the firt, middle two and list framee of druw video reputivvly. Please croats unambigacus riforrnng exprosium deseriting the object repnoamted the red déject bounding bus the given video, The reterrngg expresion senutstion procedune follows the principles of
Object-Spedfiic Prompt with Humann Priors
Manual Data Cleaning and Spallo Temporal Tube Correction
from Google
Integrate keyffrmms into single compucite image
Gemii 15
Checking Error Fixing for Referring Expression
UAV-SVG (Large-scale High-quality Referring Expression Aerial Videos)
The white hurvester working in the field.
TTe white hurvesher that miving from left to right in the field.
TThe large white vahicle that is working in the fiedd.
The white vebide with rod accents that is harvesting the yellow rice.
The luevester thl traveling a row of the field.
Figure 2: The construction pipeline of our UAV-SVG benchmark: Step 1) manual data cleaning and spatio-temporal tube correction, Step 2) referring expression generation, and Step 3) manual checking and error fixing. OV indicates that the object is out view, and the other 12 characteristics are detailed in Sec B.3 of the supplementary material.
information of the same object explicitly learn textual and visual uni-modal associations. The adversarial contrastive learning and backdoor
unified causal framework [4] proposes adjustment for causal intervention learn object-relevant association.
Spatio-Tempor Video Grounding. This task aims to localize the spatio-temporal tube of the unique referred object or region in the untrimmed video by sentence query. The earlier two-stage employ a pre-trained detector to generate candidate region proposals. Inspired by recent works [7 20
methods 7 8 4
use the one-stage paradigm, which does not depend on the quality of the pre-trained detector. TubeDETR [17) and STCAT [8] design transformer based video-text encoders and space-time decoder for joint modeling of spatio-temporal and multi- modal interactions. To solve the heavy computational complexity and insufficient spatio-temporal interactions, SGFDN 45 decomposes 3D spatio-temporal features into 2D motion and ID object embedding. A cross-stream collaborative reasoning framework [46] decomposes static and dynamic langual flows to capture object appearance and motion cues, respectively. CG-STVG [19] contains instance context generation and refinement modules to capture instance visual context information and eliminate irrelevant or harmful information respectively. VideoGrounding-DINo
the DETR model
4
20 utilizes pre-trained Grounding DINO to achieve powerful vocabuaayyy performance.
3 UAV-SVG Benchmark
To the best of our knowledge, only aerial videos with natural language annotations are CapERA 47 and WebUAV-3M [9]. However, CapERA's annotations are video-level captions and cannot refer to objects or regions. WebUAV-3M is single-object tracking dataset with spatial tubes and language descriptions of first-frame objects. However, this dataset cannot be directly and perfectly adapted to the SAVG task due to many hard defects. To this end, we contribute a new datasct by annotating video objects in the W with the latest Gemini model [16), namely UAV-SVG.
Dataset Annotation. The construction pipeline of the newly proposed UAV-SVG is shown in Figure 2 We choose WebUAV-3M as our data source for two main reasons. First, it is the largest public UAV tracking dataset to date, containing videos with complex scenes and diverse categories. Second, object descriptions of the first frame and bounding boxes of the target sequence are provided to avoid labor-intensive annotation for spatial and textual labels of the dataset. The detailed process provided in Sec. B.I of the supplementary material.

---

Analyses of UAV-SVG. This section analyzes the salient differences between UAV-SVG and existing video grounding benchmarks, including video resolution, scales of bounding boxes, lengths of expressions, quantities of object classes, target position distribution, and data statistics. We provide detailed analyses in Sec. [f2 the supplementary material. UAV-SVG contains over 2.01 million frames across 3,564 videos and offers 216 highly diverse object categories. The total duration and average duration of videos are 18.7h and 18.86s, as shown in Table [ There are 17,820 video- sentence-tube triples in our constructed UAV-SVG dataset. The split of training, validation, and testing is shown in Table the supplementary material. Spatial video grounding in natural scenes is quite different from that aerial scenes. Specifically, 12 characteristics and challenges are detailed in Sec. [33 of the supplementary material. We believe that unique characteristics of UAV-SVG can open the door for the SAVG with practically useful and broader real-life applications.
4 Methodology
The framework of our SAVG-DETT is shown in Figure Problem definition is shown in Sec 4.1 We first extract multi-scale video and language features from pre-trained backbones (Sec. 4.2). Different from the original DETR [4]. our encoder (Sec. 4.3) decouples the multi moltti-modllityy mulū-ssale spatio- temporal modeling into intra-scale multi-modality interaction and cross-scale visual-only fusion. Learnable video tokens and frame tokens during multi-modality interaction are used to generate initial object queries and position embeddings for the decoder. Subsequently, our decoder (See. 4.4) utilizes fused multi-scale features to modulate language features and progressively decode multi-level language-viin -vieo features from high to low level. the decoder, we use object queries to generate position offsets to constantly update the object spatial tube. Finally, we introduce scaling factor to generate larger auxiliary bounding boxes and improve spatial grounding loss function (Sec. of the supplementary material).
4.1 Problem Definition
The spatial aerial video grounding task aims to localize the referred object sequence in an aerial video by integrating vision-language information. In contrast to spatial grounding, which focuses on localizing objects in a single frame image, this task extends the temporal dimension on this concept. This means understanding where the objects are in each frame and how they move over time from the acrial view. Given an acrial video V E 7 xx with T consecutive frames, C' channels, and H x W spatial resolution, respectively, and natural language description S depicting one object existing in V. The SAVG problem can be defined as localizing a spatial tube B = {br) of the specific object referred to by the description S, where b. (xt: Vt, we, he) represents bounding box in the t-th frame. (It, yt) are the coordinates of the center and (ws, h4) are the width and height of the bounding box.
4.2 Aerial Tixeo-Text Feature Extractor
Following the existing literature [1], we use ResNet as the visual backbone to extract the acrial visual features for each frame. The visual encoder is initialized with weights from MDETR [48] pre-trained on Flickr30k [49], MS COCO (50], and Visual Genome [51. We use multi-scale visual feature maps {S,, S4 Ss) extracted from the last three stages of the backbone as the input of the multi-modalle multi-scale opatio-tempont
encoder. Formally, we flatten multi-scale visual feature maps S, from R x to R xh,w; Three 1x1 convolutional layers are used to project them into the same channel dimension C = 256, thus we get multi-scale visual features F; E IR C' We take as input set of multi-scale aerial video features F.; E RR T visual encoder for all T frames of the input aerial video.
(Nur = hew:). 4, from the
For the language encoder, we leverage the pre-trained RoBERTa 55 to convert the natural language description into an output with N₁ tokens and Ct channel dimension. One linear layer is used to project it into the channel dimension C, thus we can get language features F; E RCCM
4.3 Multi-Modality Multi-Scale Spatio-Tempore Encoder
Inspired by 53 [4], we introduce multi-scale visual features into the traditional encoder. However, computing the self-attention between multi-scale visual features and language features for each frame
5

---

0
Igloked ealked
lecdee
Tredicted Tube
0
0-0
Visual Encoder
Croeescale Visual-only Fasion Branch
Mulli-level
Language
0
Modulation
The green Language
corner Encoder roundsbout
beading to spper right, from lower left
Linear
Multi-modality Interaction Branch
Query
Generator
Position
Multi-level Progressive
o 6... à 6-.8
Spatial Decoder
O aq-0
temporal interaction bayer
X
Interaction Ircanae
Intra-icaale Mustii 115--nedaaiiyy alcirrdaaity
Cinss-eral Visual-oni
Fusion Branch
video tolor
Prrt
Che
Fusion
Enaljaliineesttiie.... layer
Cass
spatial Internattoon layer
Trme
SAC
PM
Baptlack bd
00-0
1--00--- -
levrabbee Frune talken
Visual
bangange
Fusion
Figure 3: The framework of SAVG-DETR consists (a) aerial video-text feature extractor, (b)
multi-modality multi-scale spatio-temporal encoder, and (c) hierarchical progressive decoder.
computationally expensive and unaffordable. To overcome this problem, we decouple the encoder into intra-scale multi- multi-modality interaction and cross-scale visual-only fusion. In order intuitively understand, we show the detailed design idea in Sec. C.T of the supplementary material.
Intra-scale Multi-modality Interaction Branch. This branch aims to model multi multi-modality interactions between the language features F, and high-level acrial video features Fus
{Fess E CC 17 f=1- Specifically, the branch consists of a N layer encoder. Each layer starts with spatial interaction layer followed by temporal interaction layer, shown in Figure (b. We introduce learnable embedding F! CRx1 (namely frame token) t-th frame capture the spatial context of the referred object through intra-modality and inter-modality interactions. Frame tokens Ff {F(TT=
| fuse information across spatial dimensions of visual and textual modalities. The spatial interaction layer conducts local spatial modeling for each frame but lacks global temporal modeling. The temporal interaction layer applies the self-attention across temporal dimensions between frames tokens. Similarly, we introduce a learnable embedding F RRXC (namely video token) capture the global aerial video-text context.
Cross-scale Visual-only Fusion Branch. The objective of this branch to fully extract both high-level semantic information and detailed object information from multi-scale visual features for efficient localization of the grounded object. First, it transfers the rich contextual semantics embedded in high-level visual features after multimodal interaction to enhance low-level features. Moreover, it injects the object localization details contained in low-level features into high-level features via fusion. To address the challenges of extensive small objects and avoid a large computational cost generated by this branch, we derive solutions from the popular real-time detection transformer [2] 22 55 23
This branch mainly includes the two-scale fusion module, multi-scale feature fusion with frequency enhancement (MSFF-FE) module, frequency-focused down-sampling (FD), semantic alignment and calibration (SAC) module, illustrated in Figure 3 (b.2).
4.4 Hierarchical Progressive Decoder
We propose the hierarchical progressive decoder that utilizes multi-level language and visual features to guide object queries to decode more relevant spatial information. We show more technical detail in Sec. C.2 of the supplementary material.
Multi-level Language Modulation Module. This module is designed to enhance the visually contextualized language features by incorporating different levels of visual features. Given language
features F.
FN RBCM and multi-level visual features F. FF E C.C.N
t=1"

---

Multi-ievel Language Modulation Module
Multi-level Progressive Spatial Decoder
language
MatMal
SoftMax
High-level Decoder
Mid-level Decoder
Low-Jevel Decoder
Yep-ast
V Mathuu
Linenr
Mil-leve
Milleve
Tempueal
Allarionn
language
Viuua
Query Position Generator
Enooling
France
q
replication
MLP
Temporal Self-
Language
&C
Visual Cross-
4
Cross-
FEN
Attention
6f
Positional Embedding
Offsets Generator
x
MLP
A AA
Upaane
The structure the multi-level language modulation module, the query and position generator, and the multi-level progressive spatial decoder.
Figure 4:
we employ cross-attention to achieve cross-modality fusion:
proi, Jrroja F ?T
Attn:
(I)
Vd
Fi. W; (SoftMax(Attn) projo (F bi.
Fet))
where krojg.ke are the query, key, and value projections, W, and by are the learnable parameters, and Fi, RCM, Ni denotes the updated i-th level language features in the t-th frame. The resulting multi-level language features at different levels are:
Lewel, Livid Umiid-evely Lhigh = (Fi) (F (Fi)
Query and Position Generator. Existing advanced encoder-decoder based grounding methods either use learnable embeddings [7 19] or use language features 22 20 as decoder queries. Despite achieving advanced results in natural scenes, these methods in complex aerial scenes exac- erbate the problem of inconsistent localization results across frames. To alleviate this problem, we leverage video tokens F" and frame tokens F in the intra-scale multi molti-mmddllity interaction branch to generate queries and predict the initial bounding box tube as reference anchors. The structure is shown in Figure Firstly, we project the video tokens F as the initial queries Q E CRC The Q! temporally replicated 7 times for each frame resulting in object queries Q" = {QP) Unlike the previous approach, queries are mapped from video tokens. All frames associated and contain the same vision-languag contextualized semantics. Such a mechanism helps to make spatial positions of the decoding more temporally consistent.
Then, encode video tokens modulate the previous frame tokens F by scaling and shifting to generate the initial reference anchor per frame 8d = [b.. 1=1- Specifically, video tokens F" are projected scaling factor 7 E EC and shifting factor 3 C respectively:
tanh (W..F" - b,), 8 = tanh 3W 109+00 bb),
(4)
where W,, , by W 3. and by are the learnable parameters. Afterwards, frame tokens are then refined with the two modulation factors. Finally, generate the initial reference anchor box by E RR:
b Sigmoid Wp (F! (00+ bp
(5)
where @ represents hadamard product, and W, E Cx1 and by E RM. are learnable parameters.
Multi-level Progressive Spatial Decoder. Existing DETR decoders project object queries to bounding box coordinates via prediction head. It is difficult to accurately and consistently ground referred object across frames in aerial video with small objects and complex motion. improve the grounding consistency, we propose the multi-level progressive spatial decoder (MPSD). Previous object detectors [56) 57 amplify the importance of low-level features enhance small object detection. Therefore, MPSD decodes from high level to low level. and gradually increases the number
7

---

of decoding layers for lower-level features. As shown in Figure 4 our MPSD has M layers and is cascaded by High-level, Mid-level, and Low-level Decoders. First, reference anchor boxes are attached to object queries through positional embedding, guiding queries to capture the spatial information of the referred object:
P ".. )),
(6)
where SinEmbed means the sinusoidal position encoding to biT (xr) (x:]--1 U1
10-- w0 , 1im--_1__ hi hmm-1).
In addition to positional embedding P. sinusoidal temporal positional encoding is added to the positional part of object queries of the m-th layer. Then, object queries QT perform long temporal
dependency modeling along the temporal dimension and progressively decode the object position from high-level to low-level under the guidance of multi-level language and visual features:
QTT LN (Q
MHSA temporal ())-
Q =
LN
m + MHCA language
Q L, L))
(8)
QT
LN QT
+ MHCA m visual
(Q" V, v)). (QU
(9)
Q
LN
QT
+ FFN
(10)
where LN(.) denotes layer normalization. The multi-level language and visual feature inputs at different layers are:
Levgh V high -levet) (Lmid-level, level, V middleodd if m 3 if m 4, or 6.
if m
(L, (L, (Lioe-tevel, V Low (owwte
(11)
Finally, we fed Q into the offsets generator consisting of 3 fully connected layers with the ReLU activation function. The offsets generator directly regresses 4-dim bounding box offset coordinates (Ar?" Ave" Aw?" Aht"). The final reference anchor box 6," is updated by:
b." (xt + Axt" + Ayj"" w. + Awi", h4 + Ah!").
(12)
Our improved spatial grounding loss function is in Sec. C.3 of the supplementary material.
5
Experiments
In this section, we conduct extensive experiments to verify our SAVG-DETR, We first introduce implementation details and the evaluation protocol for the SAVG task in Sec. [J the supplementary material and Sec. 5.1 After this, we compare with the state-of-the-an methods in Sec. 5.2 We perform extensive ablation studies to investigate the effect cach component of SAVG-DETR in Sec. 5.3 Finally, we visualize some examples for an intuitive understanding of the approach in Sec. 5.4
5.1 Evaluation Metrics
We follow the literature 17 and define the video intersection over union as vloU I Mnn4 IoU (b, bz), where N₂ represents the total number of frames of the video. by and be are the predicted and ground-truth boxes at time t, respectively. To evaluate spatial acrial video grounding, we employ m m_ vloU and vloU@) R as evaluation criteria. m_ v_v is the average vlo of video samples. The prediction for video is considered "accurate" if vloU exceeds threshold. vloU@R is the proportion of video samples for which vIoU > R. The above metrics evaluate the model's performance based on the global spatial localization accuracy of the video. To assess the model's 's localization stability for each frame, we introduce novel metric, Acc. The prediction for a frame in the video is considered "accurate" if the frame IoU exceeds 0.5. Ace represents the proportion of frames in video that are predicted correctly. m fffcc denotes the average J Acc across all videos samples. fAce@ the proportion of video samples where f Aec > R. The threshold R is usually set to 0.3 and 0.5 during testing.
N,
8

---

Table 2: Performance comparisons of the state-of-the-art methods the UAV-SVG test set.
Methods
Visual Eacoder
Language Encoder
m__voU
+8.U
<JoU/60.5
m_
[Aco@0.3
Co-grunding
Darknet53
Bi-LSTM
10.24
6.11
11.17
16.20
DCNet [6 Joaees
Darknet53
11.65
8.79
13.10
TubeDETR 22
ResNett01
RoBERTa
22.60
32.91
20.49
23.84
29.69
22.00
STCAI 1 22
ResNett01
Ko
24.14
35.51
22.48
27.17
31.19
25.36
SGFDN
BX1 ecyoRee
ResNer101
RoBERTa
20.15
28.16
13:47
19.13
22.77
17.39
CG-STVG [0] 2opp-
RecNeriO1
RoBERTa
21.25
28.82
19.04
22.32
26.24
20,41
VideoGreumsinng DINO
N0
Swin-Trans.
BERT
23.83
33.84
19.92
25,80
51.72
23,00
SAVG-DETR
ResNetiOL
27.15
22.85
28.82
35.85
Table 3: Ablation study of key components of Table 4: Comparisons of the variants and baselines.
our SAVG-DETR framework.
Variants Methods
ngVloU (96)
m_.Axe
FLOPs (G)
Parzin (M4)
Mcs (G)
Eacoder
Decoder
m. m__loU
FAce
IMIB CVFB MLMM QPG MPSD
(%)
(%%
A
17.64
18.93
65.07
184.22
9.3
B
22.31
24,30
130.51
199.77
14.4
22.38
25,46
199.77
2-48
19.37
21.84
26.64
27.49
245.37
215.9
31.5
25.44
26.54
TibeL 1
E ((urs))
22.50
27.15
23.84
18.82
20008
144.04
209:25
185 17
11.5
26.7
27.92
STCAT
14.14
27.17
175.46
307 14
15.9
27.15
28.82
SGEDD
54
20.13
19.13
53.05
178.91
4,3
CG STVG
10
21.23
22.32
231.11
192.85
27.3
5.2 Comparison with the State-of-the-art Methods
To fully verify the superiority of our proposed SAVG-DETR, we compare it with all SOTA methods on Table Specifically, we provide two sets of comparison methods: SOTA video REC methods: Co-grounding 335 and DCNet (36). 2) SOTA spatio-temporal video grounding methods: TubeDETR
20 To date, CG- STVG and VideoGrounding DING have achieved the best performance in natural scenes. Our SAVG-DETR outperforms the state-of-the-ard consistently in all evaluation metrics. We provide more detailed baselines, result analyses, advantages and disadvantages of different methods in Sec.
177. STCAT I8. SGEDN 45 CG-STVG (1) and VideoGGrunding-DNNN
D.2 of the supplementary material.
5.3 Ablation Studies
Ablation study on key components of SAVG-DETR. In Table we conduct a thorough ablation study the proposed components. The first row performs SAVG with only intra-scale multi-modality interaction branch (IMIB), vanilla decoder, and prediction head. On this basis, we further introduce multi-scale visual features into the decoder. It can be found that the accuracy drops significantly by about 3 points, in the second row of Table IMIB only processes high-level single-scale interaction, while other lower-level visual features without multi-modal interactions cannot be accurately decoded. To solve this problem, in the third row, we further add the cross-scale visual-only fusion branch (CVFB) to achieve multi-scale fusion. Lower-level visual features gather contextual information from the high-level visual features of multi-modal interactions. We find the accuracy is improved by about points. The fourth row modulates language features through multi-scale visual features and guides object queries to capture spatial information more accurately in the decoder. We find the accuracy is again boosted by about point. The last row shows that after applying the query and position generator (QPG), our full-fledger model achieves the best performance.
Design of the multi-modality multi-scale spatio-temporal encoder and computation analysis. In T we evaluate the performance and complexity of the variants designed in Sec. C.I of the supplementary material and other baselines. Compared to variant A, variant B has approximately 5% increase in performance and an 85% increase in FLOPs. This proves that intra-scale multi-modal interaction is very important, but the Transformer encoder has high computational overhead. The computational cost of the self-attention is shown as a quadratic increase in the sequence length of the input. Variant C maintains the same parameter size as B, but inputs multi-scale multi-modal features of long sequences, resulting in a significant increase in FLOPs and an unaffordable memory demand (out of memory). Variant D reduces FLOPs compared with C and has a performance increase about 3.5% over B, indicating that our decoupling strategy not only reduces computational complexity but also increases performance. Our SAVG-DETR offers 0.9% performance improvement and 17% FLOPs reduction over D. Compared with other methods, SAVG-DETR has the largest size and memory usage due to the processing of multi-scale features. However, we efficiently decouple multi modii-to and multi-scale spatio-temporal modeling, which significantly improves performance.

---

Table More ablation studies. Detailed analysis is shown in Sec. D.3 of the supplementary material.
(a) Intra-scale multi multi-modality interaction branch
(b) Object query initialization strategies
Video tokens
SIL
TTL.
m_ m_ylou
m_fAcc
Query generation
aLviol
Average pooled frame tokens
25.21
25.58
Zero vector embrolingg
26.94
28.51
Max puoled frame tokens
25.92
27.14
Mas pooled language tokens
Average pooled language tokens
25. 37
24.91
25.11
25.96
Learaable tokens
27.15
28.82
and position
27.15
28.82
(c) Time encoding and temporal self-atter self-attention the MPSD
(d) Multi-level vision-language features the MPSD
(e) Positional embedding and offsets generator the MPSD
Time Encoding
Tomporal Self-AAtention
vloU (%)
Languge foateres
Visual features
vloU
m. mA.e
Positional
murece
(5)
(%6
Embeedinn
Glencs
(16)
25.5 25.56
25.30
leantshlic
25.3
26.06
25.90
leumable
22.99
26.49
26.21
27.70
26.79
reference eachors
23.57
20.15
27.15
28.82
27.15
28.83
reference smicooes
27.15
28,82
(I) Number of decoder layers on the MPSD.
(g) Different level decoder combinations.
Layer Index
M -4
M
M -
High-lovel
Mid-level Denskr
Low-lovel
m_vloU
m_fAce (9)
vloll FAce violi FAce
vloU FAce
vloll
FAce
23.12
25.79
n 5
m
25.94
27.35
27.15
26.93
28.82
28.45
25.65
25.82
28.16
27.34
2
2
26.21
24.38
27.17
26.23
21.98
23.01
25.12
26.04
25.82
27. 68
25.16
26.65
21.36
22.23
24.53
25.33
25.94
26.38
24.58
25.30
1
25.36
26.71
0
22.89
23.67
m
19.45
22.48
23.94
24.09
21.32
Expression 1:
The only white speedboat towards the upper left of the sea, sailing with rubber dinghy.
Expression
The only white coach that is driving near the building in the lower right cornet, towards the left.
Grand Truth
SAVGOETR
TObeDETR
STCAT
SGPON
CC-STVG
Video
Figure 5: Qualitative results of different methods on the UAV-SVG benchmark.
SAVG-DETR achieves a better trade-off between performance and complexity. We provide more detailed ablation studies in Sec. D33 of the supplementary material.
5.4 Visualization Analysis
In Figure 5 we present some qualitative examples, comparing with other methods. The challenge posed by small objects in aerial scenes is intuitively apparent. Additionally, in Expressions and multiple small, similar moving objects of the same class appear in frames, presenting a significant challenge. In Expressions 1 and it is necessary understand both the object semantic and the complex spatial motion, and to reason about the regional location of the speedboat and coach. In Expression 2, although STCAT (yellow) can initially detect the coach in the first frame, it gradually fail to locate it. Our proposed SAVG-DETR (red) performs well and achieves reasonable localization results. SGFDN (cyan) locates the blue coach as it pulls into the building in the lower right corner. As the drone moves, the blue coach disappears from view and SGFDN locates the parked white coach. We provide more qualitative results and failure analysis in Sec. D.4 of the supplementary material,
Conclusion
In this paper, we introduce novel SAVG task and contribute challenging large-scale benchmark UAV-SVG. To improve the grounding performance of aerial small objects and consistency across frames, we propose SAVG-DEL DEGR framework. The core design is a multi-modality multi-scall spatio-temporal encoder for cross-modal cross-scale feature fusion and alignment, and a hierarchical progressive decoder for efficient spatial tube prediction. From coarse to fine, SAVG-DETR gradually bridges the modality gap and iteratively refines reference anchors of the referred object. Extensive experiments validate the effectiveness and superiority of the proposed method.
10

---

Acknowledgments
This work is supported in part by grants from the National Key Research and Development (R&D) Program of China 20.222 and the Basic Research Project for Young Students of the National Natural Science Foundation China (No.624B2113).
References
[1] Jiajun Deng, Zhengyuan Yang. Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. TransVG++: End-to-end visual grounding with language conditioned vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
[2] Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. Dynamic MDETR: A dynamic multimodal transformer decoder for visual grounding. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI). 2023.
[3] Yang Zhan, Yuan Yuan, and Zhitong Xiong. Mono3DVG: 3d visual grounding in monocular images. In P Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). volume 38, pages 6988-6996, 2024.
[4] Wei Wang, Junyu Gao, and Changsheng Xu. Weakly-supervised video object grounding via causal intervention. IEEE Transactions on Pottern Analysis and Machine Intelligence (TPAMI). 45(3):3933 -348, 2022.
[5] Kun Liu, Mengxue Qu, Yang Liu, Yunchao Wei, Wenming Zhe. Yao Zhao, and Wu Liu. Single- frame supervision for spatio-temporal video grounding. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2024.
[6] Zhenfang Chen, Lin Ma. Wenhan Luo, and Kwan- Yee Kenneth Wong. Weakly-supervine opatio-temporally grounding natural sentence in vidco. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1884 1894, Association for Computational Linguistics, 2019.
[7] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, Lianli Gao. Where does exist: Spatio-temporal video grounding for multi-form sentences. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10668-10677, 2020.
[8] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human- centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 32(12):8238- 8249, 2021.
[9] Chunhui Zhang, Guanjie Huang, Li Liu, Shan Huang, Yinan Yang. Xiang Wan, Shiming Ge, and Dacheng Tao. WebUAV-3M: A benchmark for unveiling the power of million-scale deep uav tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 186)99918-92 2022.
[10] Zhichao Sun, Yepeng Liu, Huachao Zhu, Yuliang Gu, Yuda Zou, Zelong Liu, Gui-Song Xia, Bo Du, and Yongchao Xu. RefDrone: A challenging benchmark for referring expression comprehension in drone scenes. arXiv preprini arXiv:2502.000 00392, 2025.
[11] Chuang Yang, Bingxuan Zhao, Qing Zhou, and Wang. Mmo-ig: Multi-class and multi-scale object image generation for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[12] Yang Zhan, Zhitong Xiong, and Yuan Yuan. SkyEyeGPT: Unifying remote sensing vision-language tasks via instruction tuning with large language model. ISPRS Journal of Photogrammetry and Remote Sensing, 221:64 77, 2025.
[13] Chuang Yang. Xu Han, Tao Han, Yuejiao Su, Junyu Gao, Hongyuan Zhang. Yi Wang, and Lap-Put Pui Chau. Signeye: Traffic sign interpretation from vehicle first-person view. IEEE Transactions on Intelligent Transportation Systems, 2025.
[14] Chuang Yang, Xu Han, Tao Han, Han Han, Bingxuan Zhao, and Qi Wang. Edge approximation text detector. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
[15] Yan Li, Yang Zhan, Maohan Liang, Yu Zhang, and Jinhao Liang. UPTM-LLM: Large language models- powered urban pedestrian travel modes recognition for intelligent transportation system. Applied Soft Computing, page 1 113999, 2025.

---

[16] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- -ean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser. al. Gemini 155: Unlocking multimodal understanding across millions tokens of context. arXiv preprint arXiv:2403.0555 0553300 2024.
[17] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. TubeDETR: Spatio-temporal video grounding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). pages 16442-16453, 2022.
[18] Yang Jin, yongzhi li, Zehuan Yuan, and Yadong Mu. Embracing consistency: A one-stage approach for
video grounding. In Advances in Neural Information P Processing Systems (NeurIPS). volume 35. pages 192229904,,22 2022.
[19] Xin Gu, Heng Fan, Yan Huang. Tiejian Luo, and Libo Zhang, Context-guides spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 18330- 18339, 2024.
[20] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Ming-Hsuan Yang, and Fahad Shahbaz Khan. VideoGrrundingg DINO: Towards open vocabullyy apatio-tempora video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 18909 18918, 2024.
[21] Yian Zhao, Wenyu Lv, Shangliang Xu. Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16965 16974, 2024.
[22] Yi-Xin Huang, Hou-I Liu, Hong-Han Shuai, and Wen-Huang Cheng. DQ-DETR: for tiny object detection. In European Conference on Computer Vision (ECCV), pages 290-305, 2024.
Detr with dynamic query
[23] Huaxiang Zhang, Kai Liu, Zhongxue Gan, and Guo-Niu Zhu. UAV-DETR: Efficient end-to-end object detection for unmanned aerial vehicle imagery. arXiv preprint arXiv:2501.01885 01855, 2025.
[24] Zhenghang Yuan, Lichao Mou, Yuansheng Hua, and Xiao Xiang Zhu. RRSIS: Referring remote sensing image segmentation. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[25] Enyuan Zhao, Ziyi Wan, Ze Zhang, Jie Nie, Xinyue Liang, and Lei Huang. A spatial frequency fusion strategy based on linguistic query refinement for rsvg. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[26] Meng Lan, Fu Rong, Hongzan Jiao, Zhi Gao, and Lefei Zhang. Language query based transformer with multi-scale moosl-modal alignment for visual grounding on remote sensing images. IEEE Transuctions on Geoscience and Remote Sensing, 2024.
[27] Yang Zhan, Zhitong Xiong. and Yuan Yuan. RSVG: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 61:1-13, 2023.
[28] Chongyang Li, Wenkai Zhang, Hanbo Bi, Jihao Li, Shuoke Li, Haichen Yu, Xian Sun, and Hongqi Wang. Injecting linguistic into visual backbone: Query-awart multimodal fusion network for remote sensing visual grounding. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[29] Ke Li, Di Wang. Haojie Xu, Haodi Zhong, and Cong Wang. Language guided progressive attention for visual grounding in remote sensing images. IEEE Transactions Geoscience and Remote Sensing, 2024.
[30] Sihan Liu, Yiwei Ma, Xiaoqing Zhang, Haowei Wang, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. Rotated multi-scale interaction network for referring remote sensing image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 26658 26668, 2024.
[31] Qi Feng, Vitaly Ablavsky. Qinxun Bai, Guorong Li, and Stan Sclaroff. Real-time visual object tracking with natural language description. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 700-709, 2020.
[32] Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5851-5866 2021.
[33] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Amold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6495-6503, 2017.
[34] Zhengyuan Yang. Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. brounding-tracking integration. IEEE Transactions on Circuits and Systems for Video Technology (TCSV)). 3443, 2020.
12

---

[35] Sijie Song. Xudong Lin, Jiaying Liu, Zongming Guo, and Shih-Fu Chang. Co-grounding networks with semantic attention for referring expression comprehension in videos. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1346-1355, 2021.
[36] Meng Cao, Ji Jiang, Long Chen, and Yuexian Zou. Conrespondecce matters for video referring expression comprehension. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), pages 4967 4976, 2022.
[37] Linwei Ye, Mrigank Rochan, Zhi Liu, Xiaoqin Zhang. and Yang Wang. Referring segmentation in images and videos with tross-modal self-attention network. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI) 44(7):3719 3732, 2021.
[38] Jiang Ji. Meng Cao. Tengtao Song, Long Chen, Yi Wang, and Yuexian Zou. Video referring expression comprehension via transformer with content-conditione query. In Proceedings of the 1st International Workshop on Deep Multimodal Learning for Information Retrieval, pages 39-48, 2023.
[39] Yujia Zhang, Qianzhong Li, Yi Pan, Xiaoguang Zhao, and Min Tan. Multi-stage image-language cross- generative fusion network for video-based referring expression comprehension. IEEE Transactions Image Processing (TIP), 2024.
[40] Arka Sadhu, Kan Chen, and Ram Nevatia. Video object grounding using semantic roles language description. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10417-10427, 2020.
[41] Wei Wang, Junyu Gao, and Changsheng Xu. Weakly-superrised video object grounding via stable context learning. In Proceedings of the 29th ACM International Conference Multimedia (ACM MM), pages 760-768, 2021.
[42] Wei Wang, Junyu Gao, and Changsheng Xu. Weakly-supervised video object grounding via learning uni-modal associations. IEEP. Transactions on Multimedia (TMM)) 33:6329-6340, 2022.
[43] Zhu Zhang, Zhou Zhao, Zhijie Lin, Baoxing Huai, and Jing Yuan. Object-awan aware multi-branch relation net- works for spatio-temporal vidco grounding. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence (IJCAI), pages 1069-1075, 2021.
[44] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision (ECCV), pages 213-229, 2020.
[45] Weikang Wang, Jing Liu, Yuting Su, and Weizhi Nic. Efficient apatio-tempon video grounding with semantic-guided feature decomposition. In Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pages 4867-4876, 2023.
[46] Zihang Lin, Chaolei Tan, Jian-Fang Hu, 7hi Jin, Tiancai Ye, and Wei-Shi Zheng. Collaborative static and dynamic lasion-language streams for spatio-temporal video grounding. Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 23100-23109, 2023.
[47] Laila Bashmal, Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Mansour Zuair, and Farid Melgani. Capera: Captioning events in aerial videos. Remote Sensing, 158))21339 2023.
[48] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 1780-1790. 2021.
[49] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phras correspondences for richer image-to-sentence models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2641- 2649, 2015.
[50] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint urXiv:1504.00323 2015.
[51] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV) 123:32 73, 2017.
13

---

[52] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:1907.116999 2019.
[53] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), (4111141444111770077333333 2021.
[54] Tao Ye, Wenyang Qin, Zongyang Zhao, Xiaozhi Gao, Xiangpeng Deng, and Ouyang. Real-time object detection network in uav viv-iision based on can and transformer. IEEE Transactions on Instrumentation and Measurement, 72:1- 13, 2023.
[55] Shuo Wang. Chunlong Xia, Feng Lv, and Yifeng Shi. RT-DETRv3: Real-time end-to-end object detection with hicrarchical dense positive supervision. arXiv preprint arXiv:2409.08447 2024.
[56] Chuyi Li. Lulu Li, Hongliang Jiang. Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li. Meng Cheng, Weiqiang Nie, et al. Yolov6: A single-stage object detection framework for industrial applications. arXiv preprint wxxi::209.029777 2022.
[57] Chien- Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Yolov7: Trainable bag-of- ffeebeisss sets state-of of-the-at art for real time object detectors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7464 7475, 2023.
[58] Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. Many-shot in-context learning multimodal foundation models. arXiv preprint arXiv:2405.09750 097998, 2024.
[59] Zhangyang Qi. Ye Fang. Mengchen Zhang, Zeyi Sun, Tong Wu, Ziwei Liu, Dahua Lin, Jiaqi Wang, and Hengshuang Zhao. Gemini vs gpt-4v: preliminary comparison and combination laiiunla models through qualitative cases. arXiv preprint arXiv: 12. 15011, 2023.
[60] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13733-13742, 2021.
[61] Hao Zhang, Cong Xu, and Shuaijic Zhang. Inner-iou: more effective intersection over union loss with auxiliary bounding box. arXiv preprint arXtv:2311.02877 2023.
[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
14

---

Technical Appendices and Supplementary Material
In the supplementary material, we will introduce the following content to complement the details of our study.
Scc.
More Introduction
Sec.
More Benchmark Details
Sec.
More Details of the Methodology
Sec.
More Experiments
Sec.
Licenses
Sec.
Limitations and Future Work
Scc.
Societal Impact
A
More Introduction
We provide some data samples of existing benchmarks natural scenes, and our UAV-SVG benchmark samples are shown in Figure The SVG task in natural scenes has predominantly focused the ego-centric fixed front perspective and simple scene, which only provides very limited view and environment. The referred object covers a significant portion the frame image. Moreover, the main objects referred are mainly human beings.
Query:
A gray elephant walks from left to right.
Query:
boy kicking ball to older man
Natural scenes
Queryl:
little boy with Christmas hat catching a yellow toy.
Query2: What is caught by the squatting boy on the floor?
Query:
The man in blue clothes speaks, and the blue follows him and walks forward.
Query:
The riding the electric bike the flower bed form upper left of the roundabout the lower left.
UAV-SVG
Query:
The green bus that is heading to the upper right, from the lower left corner of Bell Tower roundabout.
Figure 6: Spatial video grounding data samples in natural scenes and our UAV-SVG data samples.
15

---

B More Benchmark Details
B.1 Dataset Annotation
The construction pipeline of the newly proposed dataset shown Figure Step Manual Data Cleaning and
Tube Correction. Due to the hard defects of WebCAV-3M, such as signal instability, black screen, sudden change of inter-frame resolution, and target out of view, we carried out data cleaning and spatio-temporal tube clipping. The temporal tubes and bounding boxes of the video sequences manually ensure accuracy. Referring to JID-sentecce [6]. we delete videos with less than 9 frames. Step Referring Expression Generation. We design the object-specific prompt with human priors to generate referring expressions by the Gemini 1.5 Pro model. Specifically, we incorporate WebUAV-3M's manually annotated object descriptions into the prompt. The Gemini 1.5 augments more referring expressions with reference to the keyframe images and human prior information. Due to Gemini's 's limitation in continuous input processing, we integrate keyframes into a single composite image for input. Gemini 1.5 Pro is capable of handling contexts of up to million tokens, which is currently the longest context window of any large model. In addition, many studies [58] [59] and reviews have shown that the Gemini 1.5 Pro performs better than GPT-4o. Step 3: Manual Checking and Error Fixing. Our team strives to maintain non-arbiiuous and high-quality annotation through manual quality control. Each expression is manually checked to determine whether the described attributes are correct and whether the referred object can be uniquely distinguished. Correct any emror that exist in the raw expressions. If an instance is difficult to describe uniquely and precisely or is hard to distinguish from other objects, discard this sample. Our manual verification of the dataset takes about 3 months.
verified
B.2 Analyses of UAV-SVG
This section analyzes the salient differences between UAV-SVG and existing video grounding benchmarks. The three most widely used video-based spatial visual grounding datasets for natural scenes include VID-sentence 6 VidSTG 77. and HC-STVG [8) Our analysis encompasses the following viewpoints.
Video resolution and scales bounding boxes. In Figure 7 (a), the distribution pattern of the video resolution in different benchmarks is illustrated by the different color circle distributions and the area of circles. It is evident that UAV-SVG's video resolution (red) is more widely distributed and contains more high-resolution videos than other datasets. The natural scene vidco is mainly concentrated in the area of 1, 1,200 1,000. UAV-SVG contains more video than this resolution, even up 2,000 pixels wide or high. Figures (b). (c), and (d), the relative area and absolute area of the bounding box are shown respectively. It is clear that UAV-SVG exhibits greater scale differences compared to other datasets. The relative area and absolute area of the bounding box in UAV-SVG are small, and the absolute area is mainly concentrated within 200 pixels. This data reveals the challenges of small object grounding for the UAV-SVG benchmark.
50000
100
200000
Hoathed
XM05
no 20 Will 1D0 Wi (ps)
VMSTO ViI--aren L1A/90
VAARO
AD 1000
de Vidoo rosdution distibationn
(11 ReReeaaive a0ea hounding bowes
Forkccannt dicteluatinn beundinng
Suxc 120 Bounding bas sinc distribution
Figure 7: A comparative analysis of our UAV-SVG against the HC-STVG, VidSTG, and VID- Sentence benchmarks. (a) represents the distribution of video resolution (height and width), and the area of each circle is proportional to the number corresponding resolution. (b) and (c) illustrate the distribution patterns of the relative area of each frame bounding box with respect to the frame size and the pixel count (in terms of the product of height and width). (d) shows the density distribution map of the bounding box size of our UAV-SVG dataset in detail.
Lengths of expressions. As shown in Table I the deseriptions of UAV-SVG contain vocabulary of 3,242 words. The minimum and maximum length of the sentences are and 46, respectively. The descriptions have an average of 16.39 words. We demonstrate the word count distribution of the expression lengths in Figures (a) and (b). We note that the expression lengths of VID-sentence and VidSTG are shorter and are mainly distributed the left of 15. HC-STVG and our UAV-SVG are much more widely distributed. Figure (c) shows the word clouds for descriptions of UAV-SVG. We can see that UAV-SVG covers a wide range descriptions. including objects, attributes, relationships, motions, etc. Longer expressions can accommodate more details about the object's 's attributes, appearance, location, relationships, and changes, which increases the semantic space that the model needs to consider and brings challenges spatial video grounding.
16

---

nan
bott for
Wind coert jencenii lenght
Wind moee (reuannian loggft
Figure (a) and (b) show the sentence lengths' distribution. (c) presents the word cloud of UAV-SVG vocabulary with each word proportional to its frequency. (d) shows the proportion of each object superclass in UAV-SVG.
Quantities of object classes. As shown in Figure (d), all videos of UAV-SVG are divided into 12 superclasses, including person, animal, vehicle, building, vessel, public transport, aircraft, agricultural machinery, industry machine, plant, artifact, and natural object. More specifically, UAV-SVG includes 216 object classes and 73 motion classes in total. The histograms I0 respectively. We can observe that the entire aerial videos and the number of videos in each set of superclasses present long-tail distribution. For example, the vehicle and person superclasses contain 941 and 849 vidcos, respectively, while the natural object and plant superclasses only have 21 and 16 vidcos. For example, the sedan and SUV objects the vehicle superclass contain 210 and 164 vidcos, respectively, while the police van only video. These reflect the true distribution objects the aerial video source. As shown Table [ our UAV-SVG clearly has more diverse and richer object categories than other natural scene datasets. These long-tail distributions and rich object classes pose significant challenges in building accurate and robust grounding models for aerial video.
object classes and the motion classes are shown Figures and
50
Figure 9: The number of videos per group of object classes. The colors of the bar chart represent 12 superclasses, from left to right: aircraft, vehiele, building, public transport, vessel, people, animal, artifact, agricultural machinery, and natural object. The detailed classification of the people superclass shown in Figure TO Best viewed by zooming in.
200
150
100
50
Figure 10: The number of videos per group of motion classes. Best viewed by zooming in.
17

---

Target position distribution. The distribution of the normalized target center position in different datasets is from movies and carefully selected with human-centric clips. object centers of HC-STVG are clearly distributed around y=0.5 and below. Because movies are mainly eye-level line shots, that is, the camera is roughly parallel to the line of sight of the subject, and the character placed in the middle of the picture. The videos of VID-sentence and VidSTG are both hand-held or ego-centric fixed front perspectives, which provide only a very limited view. VidSTG is radial in three directions from the center point the left, right, and down. The position distribution of VID-sentence radiates from the center the periphery, and UAV-SVG similar. In addition, UAV-SVG also contains number of scattered ohjects distributed in the edge parts. In Figure [1] the targets of UAV-SVG in the training and test sets have similar position distributions, concentrated (d... highlighted) in the central region of the images.
shown in Figure [I HC-STVG'ss videos
UAV-SVG
HC-STVG
VID-Sentence
VidSTG
Tmin
Test
64
64
Figure 11: Target position distributions. Best viewed by zooming in.
Other statistics UAV-SVG contains over 2.01 million frames across 3,564 videos and offers 216 highly diverse object categories. The total duration and average duration of videos are 18.7h and 18.86s, shown in Table There are 17,820 video-sentence tube triples our newly constructed UAV-SVG dataset, including 14,060 training triples, 845 validation triples, and 2.915 testing triples, shown in Table We believe that the unique characteristics of UAV-SVG can open the door for the spatial aerial video grounding paradigm with practically useful and broader real-life applications.
[
Table 6: Dataset Statisties of the UAV-SVG dataset.
#Query
#Vidco
#Super.
#Object.
#Motion.
Train
14,060
2,812
12
202
67
Val
845
169
10
50
14
Test
2,915
583
12
113
36
All
17,820
3,564
12
216
73
B.3 Dataset Characteristics
Spatio-temporal video grounding in natural scenes is quite different from that in acrial scenes. As shown in Table [ the scale of referring expressions VID-sentence and HC-STVG relatively small. Moreover, the objects of "DD-sentence are limited to two superclasses, animal and vehicle. HC-STVG is a human-centric video dataset with only humans as its objects. VidSTG had the largest data size, but the proportion of objects human and animal superclasses reaches 92.12%. In summary, the scenarios of the above datasets are simple and the number of object categories limited. In contrast, UAV aerial videos have broader field of view and encompass wider and more complex range object categories, posing a higher challenge to the spatio- temporal video grounding task. The top left corner of Figure 2 illlsstrates some challenging samples. Specifically, compared to existing datasets, our proposed UAV-SVG dataset has the following characteristics and challenges:
Fast Motion (FM): The motion of the object is fast, resulting in large bounding box differences between each frame, such as wind turbine blade.
2. Illumination Variations (/V): The illumination of the target region changes due to the sun irradiation and the movement of the camera. There is also low-illumination problem dusk and night.
3. Partial Occlusion (PO): The object is partially occluded in the video sequence, such as the truck entering the interior of the viaduct.
18

---

4. Camera Motion (CM): The UAV carrying the camera may suddenly move and cause the picture to change rapidly.
5. Scale Variations (SV): The object bounding box ratio varies greatly between different frames, such as ship the frame.
Viewpoint Changes (VC): Due to the co-motion of the camera the object, the viewpoint changes, which seriously affects the appearance the object. example, a cable car moves on a cable.
ROTation (ROT): Objects continuously rotate video sequence, such as car driving on curved highway.
8. Aspect Ratio Variations (ARV): The aspect ratio of the object bounding box varies greatly, such as a high-speed train running at high speed.
9. Low Resolution (LR): Due to the large sky view field of UAV, most of the objects in aerial video are small targets with low resolution, especially the human and car.
10. DEFormation (DEF): Objects are deformable during tracking, such as the player in intense motion on the baskethall court.
11. Background Clutter (BC): In the of aerial photography, the background and the object often have similar appearance, which is easy to be confused and difficult to distinguish.
12. Motion Blur (MB): The object region blurred by the fast motion of the target the camera, such a fast flying white bird.
C More Details of the Methodology
As shown in Figure proposed SAVG-DETR consists of acrial vidco-text feature extractor, multi- modality multi-scalc spatio-temporal encoder, and hicrarchical progressive decoder. In this section, we present a more concrete exposition for our multi-modality multi-scale ppatio-tempory encoder (Sec. C.T), hicrarchical progressive decoder (Sec. C.2) and improved spatial grounding loss function (Sec.
C.1 Multi-Modality Multi-Scale Spatio-Temport Encoder
Cross-scale Multi-modal Fusion
Crost-scale Visual Fusion
Multi-scale Multi-modal Encoding
Single-scale
Single-scale
Single-scale
Single-scale
Single-scale Multi-modal
Single-scale
Single-scale Multi-modal
Multi-modal
Intra-scale
Cross-scale
C
Decoupled
Enhanced
Langnage modality
modality
Multi mudality
CNNs
Figure 12: The different types of multi-modal multi-scale encoder variants and the evolution process. The number of blue arrows indicates the number of visual scales.
The introduction of multi-scale features UAV object detection can accelerate convergence and improve performance [ Inspired by this, we introduce multi-scale visual fcatures into the traditional multi- modality spatio-temporal encoder. However, computing the self-attention between multi-scale visual features and language features for each frame is computationally expensive and unaffordable. To overcome this problem, rethink the structure of the spatio-temporal encoder. In order intuitively understand the design idea, show the different types of encoder variants and evolution process in Figure [2] The details are follows:
The initial variant directly connects multi-moda and multi-scale features without any encoding or fusion, which will not effectively used for decoding.
Variant B inserts single-scale multi-modal encoder to first conduct intra-scale multimodal encoding. followed by feature concatenation for output.
Variant C directly feeds concatenated multi-modal multi-scale features into a transformer encoder, performing rross-scale cross-modal interaction.
Variant D decouples multi-scale molti-modal interaction into two cascaded processes of intra-scale multi- modal encoding and CNN-based moss-scale multi-moda. fusion.
19

---

xN
temporal interaction layer
learnable
video token
snatial ctiin layer
frame T
spatial interaction layer
frame
frame
learnable
visual
language
frame token
token
token
Figure 13: Intra-scale multi-modality interaction branch is stack of N layers. Each layer consists of spatial interaction layer and temporal interaction layer. The spatial interaction layer fuses information across spatial dimensions and vision-lange lasion-language modalities locally in each frame. The temporal interaction layer fuses information across temporal dimensions in the global video-language context.
Variant E proposes a more refined and enhanced framework that cascades intra-scale multi- interaction and cross-scale visual fusion based D. Our framework only performs intra-scale multi- modal interaction cost of variant D. This because high-level features contain richer semantic concepts about the object and can capture the referred object entity in conjunction with the language modality, However, multi-modal interaction of low-level features lacking semantic information may introduce confusion redundancy with high-level multi-modal interactions.
and visual-only multi-scale fusion, which further reduces the computational
Based on the above analysis, we propose the Intra-Scale Multi-Modann Modality Interaction Branch and the Cross-Scale Visual-only Fusion Branch.
C.1.1 Intra-scale Multi-modality Interaction Branch
This branch aims to model multi-modality interactions between the language features Fr and high-level aerial video features F.s (Fuss E NeC%NN.51' Ji=1" Specifically, the branch consists of N layer encoder based on self-attention. Each layer starts with a spatial interaction layer followed by a temporal interaction layer, as shown Figure [3] The spatial interaction layer can conduct local spatial attention on each frame. We introduce learnable embedding F. _ (namely frame token) in f-th frame capture the spatial context of the referred object through intra-modality inter-modality interactions. Then, we formulate the joint input tokens of of the spatial interaction layer on the t-th frame as:
/ =
[
F
F
Nus 2b.
F., F!).
(13)
Fos,
Language tokens
After obtaining the input (CC described above, apply the self-attention dimensions to embed æf. retain positional and modality information, we add learnable position encodings the input of each layer. Thanks to the attention mechanism, frame tokens F = { IT fuse information across spatial dimensions of visual and textual modalities. The spatial interaction layer conducts local spatial modeling for each frame but lacks global temporal modeling which is important for the prediction consistency among aerial video frames. To address this issue, we propose the temporal interaction layer which applies the self-attention across temporal dimensions between learnable frames tokens. Similarly, we introduce learnable embedding F" Raxe (namely video token) capture the global aerial video-text context. We formulate the joint imput tokens * the temporal interaction layer as:
spatial
Aem
F F*b
(14)
frame tokens
where fem E (T(+T) To retain the temporal information, we add a sinusoidal temporal position encoding the positional part of the input a fem After the intra-scale multi-modality interaction branch, split the aerial video features F. é xx and language features FÑ R xCxXMM, from the output R (xxx (N1+) layer N and split the output tem layer N frame tokens F TxC and video
20

---

tokens F" E ERXC. Then we imput the high-level video features into the cross-scale visual-only fusion branch, the language features into the hierarchical progressive decoder, and the frame tokens and video tokens into the query and position generator.
C.1.2 Cross-scale Visual-only Fusion Branch
The structure of this branch is illustrated in Figure 14 The flattened multi-scale features { FF.3. Fe4, F restored to the same shape the feature maps (Sa, Sa Ss} input. The fusion block including convolutions and RepBlocks 660 can two adjacent scale features into a new feature. First, to enhance the detection performance of small and occluded objects, we introduce the MSFF-FE module on the large-scale low-level feature F.3. This module employs cross-stage partial strategy to fuse spatial and frequency domain information from multiple scales, thereby maximally preserving the details of small objects. Second, to prevent the loss of critical spatial details during vanilla downsampling. strategy that preserves dual-domain information. Finally, to ensure that the semantic information of high-level features can be fully transferred to low-level features, we introduce the SAC module. By aligning and fusing scale Ss with Ss, enhance the semantic representation capability of low-level visual features. The detailed calculation process of the MSFF-FE module, FD strategy, and SAC module
introduce frequency-focure focused down-sampling
be found in the literature 23 After the cross-scale visual-only fusion procedure, we flatten fused multi-scale features and harvest T2'xC XCx Nui (i 3, 5). Then we input the multi-level visual features to our hierarchical progressive decoder for grounding object:
multi-level visual features F. E IR
Lewet love, V Viihh -toel F 03, od: F
(15)
Fusion
FF
Convi=1 st
SiLU
FD
p
Fe
Fusion
1x1
1x1
Conv
Conv
FD
SAC
RepBlock x3
Fa
MSFF-FE
Fusion
The structure of cross-scale visual-only fusion branch. MSFF-FE denotes the multi-scale feature fusion with frequency enhancement module. FD denotes frequency-focused wown-sampling. SAC represents semantic alignment and calibration module.
Figure 14:
C.2 Hierarchical Progressive Decoder
Our proposed hierarchical progressive decoder consists three submodules: multi-level language modulation module, the query and position generator, and the multi-level progressive spatial decoder. The modulation module integrates multi-scale visual information into language features. The generator yiclds context-rich object queries and initial reference anchor boxcs. Previous object detectors [56 (7)) amplify the importance of low-level features to enhance small object detection capabilities. Therefore, the multi-level progressive spatial decoder decodes from high level low level, and gradually increases the number decoding layers for lower-level features. Unlike existing methods, our method does not directly regress the bounding box by cascading prediction head after the decoder. To improve the prediction consistency, reference anchors as positional embedding constrict the grounding region at each decoder layer. The queries are used to predict offsets update reference anchors each layer. This framework progressively refines reference anchors of the referred object, eventually grounding the spatial tube.
C.2.1 Multi-level Language Modulation Module
As shown in Figure the multi-levad language modulation module is cross cross-attention mechanism with the SoftMax activation function. The resulting multi-level language features at different levels are:
Liow
Lmuid teaul, Lhigh- = FÝ, F..
(16)
where Fi = {Fi)= (i (i=3) 3, 5)5. The subsequent spatial decoder can utilize multi-level language features guidance to make queries decode the more relevant object regions in different levels of visual features.
21

---

C.2.2 Query and Position Generator
The structure shown in Figure Firstly, we project the video tokens F as the initial queries:
Q W F by:
(17)
where Q E TR. and Wa and by are lcarnable parameters. The Qi is temporally replicated 7 times for cach frame resulting in object queries Q" = {Q)) t==-" The positional part of the initial queries Q is computed using the initial reference anchor boxes B For the subsequent decoder layers, these queries and anchor boxes are updated iteratively.
C.2.3 Multi-level Progressive Spatial Decoder
As shown in Figure 4 each layer consists of six submodules: positional embedding, temporal self-attention, language aross-ateettion, visual cross-attention, feed forward network (FFN), and offsets generator.
First, the corresponding positional embedding process of the m-thh layer follows:
P! MLP (iinEmbed(6,"
(18)
The multi-layer perception (MLP) consists of fully connected layers. In addition to positional embedding P sinusoidal temporal positional encoding added to the positional part of the object queries.
Then, the temporal self-attention applies multi-head self-attention (MHSA) the object queries along the temporal dimension. The long temporal dependence modeled by temporal self-attention across frames. Next, based on multi-head cross-attention (MHCA), MPSD utilizes multi-level language and visual features as guidance to progressively decode the object position from high-level to low-level. The lower-level feature contains more spatial details, which is beneficial to spatial grounding. We update the object queries Q" in each spatial decoder layer.
C.3 Training Objectives
The ground-truth spatial tube contains the bounding box sequence B [b.) The final predicted box sequence is obtained by B = (6 The existing literature 19 20 utilizes the weighted sum of standard L1 loss Lt, and the Generalized Intersection over Union (GIoU) loss LaroU the spatial grounding loss. However, LatoU is less effective for small objects in bounding box regression, especially when the IoU value low. Inspired by existing work [6]], adopt larger auxiliary bounding boxes calculate losses, as illustrated Figure [5] We introduce a scaling-up ratio r to control the width and height of auxiliary ground -ruth boxes
B and auxiliary predicted boxes B Formally, the auxiliary IoU calculated
Aux-loU Overlap
B
0
(19)
Union
B UUB
Then, we define the auxiliary GloU loss as:
LAuzCtal LGI + IoU Aux-IoU.
(20)
Finally, the spatial aerial video grounding loss defined as:
L = de, Lz, (B, B) + Auucioo (B, B).
(21)
Predicted Box
Ground-truth Box
Auxiliary Predicted
Auxiliary Ground-truth Box.
(xg (6")
Area Overlap
Auxiliary Area of Overlap
Figure 15: The illustration of larger auxiliary bounding boxes.
22

---

D More Experiments
D.1 Implementation Details
To make a fair comparison, we follow the previous work [7 20 and utilize the ResNet-101 [62 as the visual encoder and RoBERTa [52] as the language encoder. For the transformer- based attention, the number of attention heads is set to 8 and the hidden dimension of feed-forwart networks in the attention layer is set to 2,048. We empirically use hyper-parameters. N = 6, M 6, AL, 5, and AAuzGooo 4. We set the initial learning rates to x 10" for the visual backbone, x 10 for the language backbone, and 10 for the rest of the network. The learning rate follows lincar schedule with warm-up for the language encoder and the learning rate is dropped by 0.1 after 6 epochs for the rest of the network. We use the AdamW optimizer and weight decay rate 10 (0 for training 20 epochs. Video data augmentation includes spatial random resizing and spatial random cropping preserving box annotations. We sample frames per second for videos and we uniformly sample 100 frames for videos with more than 100 sampled frames. The proposed SAVG-DETR trained using PyTorch on NVIDIA L20 48G GPUs with I video per GPU and the whole optimization takes around 4 days. Due to the limits of GPU memory, the resolution of input on the UAV-SVG dataset is set to 224.
D.2 Baselines and Quantitative Analysis
Baselines:
They methods for Video Referring Expression Comprehension, which only use single-frane images and expressions to complete object localization. Co-grounding and DCNet combine adjacent frames to complete object localization.
Co-grounding B3 and DCNet 36
It the first transformer- architecture inspired by DETR, specifically including an efficient video-text encoder and space-time decoder. The encoder models spatial multi-modal interactions over sparsely sampled frames.
TubeDETR 17
To alleviate feature alignment inconsistency and prediction inconsistency, it proposes a novel multi-modal template that explicitly constricts the grounding region and associates predictions between video frames.
STCAT 18
It decomposes 3D spatio-temporal features into 2D motion and ID object embedding, which can effectively reduce the computational complexity. Based on this strategy, attention is used to capture modsl interactions on multiple space-time scales.
SGFDN 45
CG-STVG [19] It learns the discriminant instance context of the object in each decoding stage, serves as guide to enhance the target awareness in the next decoding stage, and also helps to generate better new instance context to improve the localization.
(20: It utilizes the pre-trained representations from foundational spatial grounding models bridge the semantic gap between natural language and diverse visual content, and can effectively respond to open-vocabulary and closed-set scenarios.
VideoGrounding-Dii
Quantitative Analysis:
Our SAVG-DETR outperforms the state-of the-arts consistently in all evaluation metrics. Co- grounding and DCNet are unable to handle T-frame global video features and lack spatio-temporal context modeling. For example, in Figure R the bus is heading towards the upper right. Co-grounding and DCNet have lower performance as they fail to provide a global understanding of object space motion.
particular, we surpass the strong baseline VideoGrounding-DDe -INOO on the m m_vloU and m fAce metrics with absolute gains of up to 3.32% and 3.02%, respectively. We consistently outperform CG-STVG significantly despite that CG-STVG mines relevant and beneficial instance context during decoding. which shows the great potential of our method. Because there very difficult to mine discriminative instance context accurately by using single high-level scale. If the Iearned context contains irrelevant or even harmful information, the object queries will localize the wrong region in the decoding stage.
lot of small objects and similar objects in the aerial scene, it is
SGFDN can enjoy advanced performance with less computational complexity in natural video datascts, but has the worst performance in the SAVG task. This is precisely because its 3D decomposition strategy will lose a lot of object-related semantic information and motion information in aerial video, resulting in performance damage. This proves the challenge of the UAV-SVG dataset.
The suboptimal performance achieved by STCAT is attributed to former framework. The movement of UAVs, coupled with the motion of ground objects, causes the grounding model to face the drawbacks of feature aligmment and grounding inconsistency. STCAT proposes global context modeling generate multi-mon modal templates, enjoying more consistent cross-mode feature alignment and
consistency-ard aware trans-
23

---

grounding capabilities across frames. Our model still outperforms it on the m vIoU and m fAce metrics with absolute gains of up 3.01% and 1.65%, respectively.
Similar to VideoGrounding- DiIiooounndn--DIOO, TubeDETR adoptes DETR-like architecture enhanced by temporal ag- gregation modules. However, TubeDETR performs spatial multi-modal interactions over sparsely sampled frames. The difference between adjacent frames of acrial vidco is large and sparse sampling results in the loss of fine-grained spatio-temporal information.
In summary, the challenging SAVG task requires more delicate architectures or targeted improvements, and relatively generic improvements cannot significantly boost performance.
D.3
Additional Ablation Studies
In this section, we conduct ablation studies to demonstrate the effectiveness of each component in our proposed SAVG-DETR framework.
D.3.1 Effect of the intra-scale multi-modality interaction branch
The intra-scale multi multi-modality interaction branch must perform local spatial modeling for each frame, followed by global temporal modeling across the entire aerial video. This is crucial for conducting multi-modallty spatio-temporal interaction and modeling. To demonstrate the validity of this branch, we remove the temporal interaction layer (TII.) as a variant of our model. For two specific variants, video tokens are the average pooling or max pooling frame tokens, respectively. The quantitative ablation results are reported in Table 5 (a), which show distinct performance drop without TIL. "SIL" represents the spatial interaction layer. In contrast, learnable tokens tend to be more equitable and flexible. In the TIL. they adaptively aggregate the information in frame tokens to model the global object semantic information. While other variants are generated directly from frame tokens, which involves biases to the specific context of the corresponding local spatial information.
D.3.2 Effect of the object queries initialization strategies
(b),, we present ablation studies on the effectiveness of different object query initialization strategies. The initialization strategies in existing grounding methods mainly include learnable zero vector [8] average pooling max pooling language tokens D. Among advanced visual grounding or video grounding methods, initializing object queries with zero vectors the most common. As shown in the first row of (b) this approach achieves similar performance to our method. When the object queries are initialized with the language tokens, performance degradation occurs. Specifically, average pooling sets the same attention weights equally for each word, max pooling selects one word token for each sentence to set its weight as and that of others as 0. As empirically shown in the table, the best choice is to video tokens for object query generation in our proposed query and position generator.
D.3.3 Ablation study on the multi-level progressive spatial decoder
The design choice of our proposed multi-level progressive spatial decoder is ablated on five aspects.
The time encoding and temporal self-attention responsible for modeling the long-range temporal interactions the object queries. As shown in 5 our full decoder model is compared to variants without time encoding. without temporal self attention, and without both. The variant without both is equivalent to a pure space-only decoder in the visual grounding task, which predicts each frame independently. The comparison shows that having temporal self-attention results in hetter performance. When using both time encoding and temporal self-attention, substantial performance gains are achieved over the space-only decoder.
As shown in Table 5 (d), our decoder is compared to variants without multi-level visual features, without multi-level language features, and without both. The variant without both corresponds to a decnder with a single high-level scale feature and cannot achieve competitive performance on the SAVG task. Language cross-attention and visual tross-attention enable queries to probe features within frames. The queries are enriched to produce the final contextualized representation used to generate the spatial tubes. Therefore, multi-level language visual features can bring additional improvements (row vs. row or row 3). However, multi-level visual features can provide richer visual context for queries and gain more than multi-level language features (row 2 vs. row 3). Finally, the best performance can be achieved when using multi-level vision-language language features modulated by multi-scale visual features can be used as guide to help queries aggregate more object-related spatial information in multi-level visual features. Therefore, our full-fledged model a great improvement over the single-scale decoder only (row vs. row 4).
fcatures. The
To demonstrate the effectiveness our proposed progressive decoding paradigm based on offset generation, our decoder is compared to variants without the offsets generator, without positional embedding based on reference anchors, and without both. The variant without both degenerates into a framework with learnable positional embeddings and cascade of decoder and prediction head. This degenerate baseline directly adopts the final
24

---

Table 7: Ablation on the scaling- up ratio of auxiliary loss.
Methods
m_vloU
vloU@0.3
vIoU@0.5
m. fAce
f.cc@0.3
[Aco@0.5
(y=0.90)
19.60
27.74
16.36
20.61
21.36
16.33
GloU (r=1.00)
26.55
37.84
22.20
28.15
34.20
25.42
AuxGloU (r=1.10)
26.83
36.84
22.92
28.80
34.72
26.00
AuxGloU (r=1.15)
27.15
38.18
22.85
28.82
35,85
26.55
AuxGloU (r=1.20)
21.83
29.95
17.12
21.17
23.84
18.87
object queries predict the spatial tubes, which is the classic paradigm among the advanced visual grounding or video grounding methods. As shown in (c), our complete framework is a significant improvement over the baseline (row 1 vs. row 4). Furthermore, we can observe performance degradation in the other two variants (row 2 and row 3). Without positional embedding based on reference anchors, the offset generator cannot accurately predict the object's spatial offsets without the restriction of the grounding region (row 2). In contrast, object queries cannot accurately predict the ohject's spatial position one step under the grounding region limitation (row 3). Our framework uses reference anchors to guide object queries at each decoding stage to predict offsets of the referred object and updates the reference anchor boxes for the next decoding stage. The framework iteratively refines reference anchors of the referred object, which helps improve the prediction consistency.
Table (f), we ovaluate our decoder when employing different numbers of decoder stages. more decoder layers are used, the accuracy increases steadily until the saturation point is = This reflects the importance of multi-stage progressive reasoning for acrial video grounding. The progressive spatial decoder queries multi-level language features and collects multi-level visual information in multiple rounds, enabling the referred aerial object be identified and localized more accurately. Since the accuracy does not improve M =7, employ decoder layers for our decoder by default.
Furthermore, we provide different level decoder combinations based on decoder layers and decoding from high to low. The results are reported in (g). We observe that under our decoding framework, the addition of lower-level details can gradually improve grounding performance. When the number of high, middle, and low-level decoders is the second best performance can be achieved. When the number of low-level decoders greater than the high-level semantic information is reduced even no, and the performance is significantly reduced (row and row 6). This shows that our decoder achieves better trade-off between low-level detail information and high-level semantic information.
D.3.4 Effect of the scaling-up ratio of loss
To explore the effect of the scaling-up ratio, we start with and gradually increase the ratio. In Table 7 experiments demonstrate that setting scaling-up ratio r to 1.15 an appropriate choice. When is T, the auxiliary bounding box is the same as the actual bounding box, and the LAuaCIoU degenerates to the Carav. When the ratio increases to 1.2, performance decreases significantly. This is due to the scale difference between the larger auxiliary bounding boxes and the actual bounding boxes. If the ratio is too large, the auxiliary bounding boxes cannot reflect the actual bounding box distribution and quality of regression results, and the performance will be greatly lost. To prove that calculating loU with the smaller scale auxiliary bounding boxes can not be beneficial for the presence of a large number of small object samples, we also set to 0.9. The results show that the smaller scale auxiliary bounding boxes will make the performance loss more serious.
D.4
Visualization Analysis
D.4.1 Qualitative Results
In Figure [6 we present more qualitative examples obtained from the UAV-SVG benchmark, comparing our results with other methods. In Expressions 1, 2. and 3, multiple small, similar moving objects of the same class appear in video frames, presenting significant challenge. In Expression 3, most of the other methods localize other people playing basketball the court. STCAT (yellow) localizes near the boy carly on, but later experiences bounding box drift and is confused with the spectator outside the field. This is due to the semantic bias "on the right half of the court" that occurs later with the movement of the drone. However, our SAVG-DETR (red) provides stable tracking based early results with high prediction consistency across frames. In the challenging example with low illumination in Expression 4, although some methods (STCAT (yellow), SGFDN (cyan). CG-STVG (pink)) can detect the referred object later, whereas our model maintains decent performance throughout the video. In Expression 5, it is necessary to understand both the object semantic and the complex spatial motion, and to reason about the regional location of the container truck. In Figures [8 and 19 more qualitative examples.
we present
25

---

Expression 1: The onlyy white spocdiona sowards jeft of the sou, sailing with rubber dinghy.
Experssioo only white driving near building right comer,
Expression 3: The boy wearing black pants and aneakess who is playing Baskelball on right half the count and Bist one countinng from the top.
Expression sincledecker
ships
Expression
The container truck driving, letmoot highway side video
Groun Truth
SAVG-DETR
TubsDETR
STOAT
SCITN
DG-TVV
VideCeeundngg-iiO
Figure 16: Qualitative results different methods on the UAV-SVG benchmark. Best viewed by zooming in.
Expression
The fountain is spraying middle circular pool on river bank behind bridge.
Expression 2: The white and light rail coming from distance and driving on the tracks.
Expression
The white minivan that driving towards the bridge in the far right lane, the lower left
Expression
The man lying down the grass, who waiting others help him, located the middle of the grassy field.
Ground Truth
SAVG-DETR
TubeDETR
STCAT
SGEDN
CG-STVG
VideoGrounding-DIN
Figure 17: Some failure qualitative examples of the proposed method. Best viewed by zooming in.
26

---

D.4.2 Failure Analysis
we show some failure examples. In Expression 1, as the drone gets closer to the fountain, some methods (SGFDN (cyan), CG-STVG (pink), TubeDETR (bluc)) gradually localize the object. Although our SAVG-DETR (red) fails locate it accurately, by reasoning with the spatial position of the circular pool-river hank-bridge, the red box can cover the green object region. In Expression all methods struggle locate the "light rail" or only partially detect the object. Our SAVG-DETR (red) can cover semantic region when the light rail is far away. In Expression 3, the video showcases an extremely tiny white minivan, and there are many similar small white vehicles. In this challenging example, all methods fail to locate it, whereas our model predicts large enveloping box (red) for stable tracking of the small "minivan" (green). Other methods often predict semantically irrelevant regions. In Expression this scenario involves significant viewpoint changes. Our bounding box (red) can not accurately localize the region (green), but is adjacent to the region of the "man lying down on the grass". Even in the case of a final grounding failure, our SAVG-DETR framework can maintain semantic relevance and stable consistency across frames. This is attributed to our decoding paradigm based offset generation.
In Figure 17
E Licenses
We choose WcbUAV-3M ( as our data sourcc. We have properly credited the ereators or original owners of assets used in the paper, and follow the license GNU General Public License v3.0.
F Limitations and Future Work
Although method is specifically designed for the SAVG task and surpasses the existing SoTA methods, achieving the best balance between performance and complexity the same time, there are still some limitations. First, our method does not fully explore the real-time performance in practical applications. In practical applications, real-time grounding and tracking of the object are required. Second, acrial scenes often have occlusion situations, and our method has not been optimized for this problem yet.
In the future. we consider further expanding the multi- modtiim and multi-scale spatio-temporal modeling method to further enhance the model's ability capture small-scale objects and occlusion objects. Moreover, also necessary explore potential for smaller sizes and faster models.
G Societal Impact
In light of the fact that the era of low-altitude economy and the field of spatial temporal intelligence has just begun, we establish comprehensive benchmark for future advancements spatial aerial video grounding. To the best of our knowledge, this is the first to support the video grounding in the low-altitude UAV. With the widespread application of unmanned aerial vehicles (UAVs) globally. many tasks are currently performed in the sky, such as UAV-based goods delivery, urban traffic/security patrol, industrial inspection, disaster rescue in bad weather, and scenery tour. Since SAVG can naturally combine UAVs' visual and text signals complete object localization and tracking more effectively, achicving this grounding ability is crucial for advancing towards low-altitude intelligence. We believe this work will open up avenues for this new kind of SAVG.
However, this also raises concerns about how the SAVG model with strong tracking capabilities could be inappropriately used in the community, such as for illegal UAV surveillance and tracking.
27

---

Expression
right sighttccing cruise ship salling
Expression 7: On the read in the Lnver right oumeer, the wiste aodan deiving to the kel.
Expression The man the suit standing the upper right corner.
Expression sing the upper right cocner,
right going
Expression 10. The yellow flut dnving the winding mountsin rrad.
Expression man sking right.
Expression
Expression
salling dight docner.
Expression 14: On the vélie water sonoter the left satling towards the
Expression
right wearing
walking
Experssion 1b. The girl in white with long hair in the midéle of the bridge to walking to the left
Expressio rightside vinduct highee lecel, white street steeper driving Rosordd trgs
VideoGrounding
Figure 18: More qualitative results. Continuation of Figure 16
28

---

Expression
right oocner, driving
Expression 19: The man in the and dress is rurning the right consex.
Expression 20:
The link boy dotiaes sunsing in the zeii comner.
Experssioo
rding
žicycle the bridge ecross uppes coener.
Expression 22. The Imle boy white and black the highest stepa.
Expression
countng
the square, perionming is
Expression
building river bekdgg
VideeGreundinn DIND
Figure 19: More qualitative results. Continuation of Figure 18
29

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The content the abstract and introduction accurately reflects the contribution and the scope of the paper, and the main contributions are summarized in the concluding section of the introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations of our approach in Section F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to- peech-to-texx text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cros--referenced.
30

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We fully describe the proposed model and implementation details Section and we submit our main code the form of a zipped file in additional supplementary materials. To further ensure reproducibility, will soon open source the complete code of our method.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) contribution a new model (e.g.. large language model), there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soume source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce main experimental results, as described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: Once the paper is accepted, we will open source our complete code and dataset. The code and dataset have heen submitted as supplementary material ZIP format.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
http:///nippcccppuliiie
guides/Cood CodeeubdiiSSiic http:/oouuuaiisiiioooooolool ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
31

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://niaa.cc/ppin for more details.
Buzdee//DodeSuumiissioooPPo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: We have specified all the training and test details Section D.I of the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- lion about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would too computationally expensive.
Guidelines:
The answer means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The sufficient experimental implementation details are presented Section D.I of the supplemental material.
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neur//: http///rrrriii.ccpai///eee/eeeee
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics. Our work conforms to the
NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that roquire a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential impacts are discussed in Section of the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will require users to adhere to specific usage guidelines to access the model and datasets, ensuring that they are used responsibly and mitigate the risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
33

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the creators or original owners of assets used in the paper, and usc the license CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are introduced in the paper well documented and is the documentation provided alongside assets?
Answer: [Yes]
Justification: new assets introduced this paper be well documented upon their release.
Guidelines:
The answer means the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
34

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMSs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: In our paper, the LLM used only editing (e.g.. grammar, spelling, word choice).
Guidelines:
answer NA means that core method development this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy should or should not be described.
for what
35


Reviews:
[{"review_id": "qJcjxB3vPx", "reviewer": "Reviewer_kDhD", "summary": "This paper makes contributions to UAV-based spatial video grounding (SVG) by introducing UAV-SVG, a large-scale benchmark addressing the lack of specialized datasets for aerial video analysis, and proposing SAVG-DETR, a novel framework with task-specific designs that outperforms existing methods.", "questions": "Please see the Weaknesses part above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper makes a valuable contribution by establishing UAV-SVG, a novel and challenging benchmark for spatial video grounding in aerial scenes, addressing a critical gap in the field.\n2) The proposed SAVG-DETR architecture demonstrates thoughtful design choices tailored to the unique demands of aerial video analysis, leveraging transformer-based approaches effectively.\n3) The method achieves state-of-the-art performance, supported by comprehensive experiments and rigorous evaluation.\n4) The manuscript is well-written, with clear explanations and thorough analysis, including insightful discussions of failure cases that enhance future research directions.\n\nWeaknesses:\n1) Figure 3(b) requires revision, as the origin and purpose of the input blue and pink boxes are unclear. These elements should be explicitly explained or visually adjusted to align with the output from phase (a) to avoid confusion.\n2) The problem definition, currently relegated to supplementary materials, should be integrated into the main Methodology section to improve readability and ensure immediate context for readers.\n3) I believe this aerial SVG task is rather difficult when it comes to grounding upon dense objects environment (a common scenario in aerial imagery). The authors should provide additional analysis in the camera-ready version to clarify whether performance limitations stem from task complexity or methodological constraints, offering potential pathways for improvement.", "was_revised": true, "final_justification": "Thank you for addressing my previous concerns. I maintain my original score.", "rebuttal": ">**Q1**: Figure 3(b) requires revision, as the origin and purpose of the input blue and pink boxes are unclear. These elements should be explicitly explained or visually adjusted to align with the output from phase (a) to avoid confusion.\n\n**A1**: Thank you for your insightful feedback and for highlighting this point of confusion. We agree that the clarity of Figure 3(b) can be significantly improved. The blue and pink boxes are indeed critical inputs to our proposed encoder, and we appreciate the opportunity to clarify their roles and origins.\n1. The Blue Boxes (Visual Features): The blue boxes represent the high-level visual features $S_5$ extracted by the Visual Encoder in Figure 3(a). All three scales of visual features ($S_3$, $S_4$, $S_5$) represent by the blue boxes of varying shades.\n2. The pink box represents a learnable frame token ($F^f$). As we explain in Section 4.1 and illustrate in detail in Figure 13 of the supplementary material, this token is introduced within the Intra-scale Multi-modality Interaction Branch. Its purpose is to act as an aggregator, capturing the most relevant spatial context of the referred object for each frame after the vision and language features have interacted.\n\nTo address this, we will revise Figure 3 in the camera-ready version to make the data flow more explicit and intuitive.\n\n\n>**Q2**: The problem definition, currently relegated to supplementary materials, should be integrated into the main Methodology section to improve readability and ensure immediate context for readers.\n\n**A2**: We appreciate this excellent suggestion. The reviewer is correct that placing the problem definition at the beginning of the Methodology section is standard practice and enhances the paper's flow and readability. Our initial decision to place it in the supplementary material (as noted in Section 4, line 163 ) was solely due to the strict space constraints of the initial submission.\n\nIn the camera-ready version, we will move the \"Problem Definition\" from Section C.1 of the supplementary material  to the beginning of Section 4 (Methodology). This will provide readers with immediate formal context for the SAVG task before we describe the technical details of our SAVG-DETR framework, improving the overall structure and coherence of the paper.\n\n>**Q3**: I believe this aerial SVG task is rather difficult when it comes to grounding upon dense objects environment (a common scenario in aerial imagery). The authors should provide additional analysis in the camera-ready version to clarify whether performance limitations stem from task complexity or methodological constraints, offering potential pathways for improvement.\n\n**A3**: Thank you for this insightful comment. We agree that grounding in dense object environments is a core challenge of the SAVG task and a primary motivation for our work. As noted in our introduction, aerial videos often feature a \"wide field of view and a large scene scale, containing dense and numerous objects\". This inherent task complexity, combined with challenges like small object scale and background clutter, pushes the boundaries of current grounding methods.\n\nThe performance limitations observed are a result of both the profound difficulty of the task and the constraints of current methodologies.\n1. Task Complexity: The UAV-SVG benchmark was intentionally constructed to include such challenging scenarios (e.g., Expression 2 and 5 in Figure 5 , showing dense vehicles). These scenarios require not only recognizing an object but also understanding its complex spatial relationships with many similar-looking distractors, often described by subtle linguistic cues.\n2. Methodological Advances and Constraints: Our proposed SAVG-DETR is specifically designed to address this. The multi-modality multi-scale spatio-temporal encoder and the hierarchical progressive spatial decoder  work in concert to integrate high-level semantic context with low-level spatial details, which is crucial for distinguishing a target from nearby distractors. The superior performance of our method over strong baselines in Table 2  demonstrates that our approach is a significant step forward. However, we acknowledge that our method still has constraints. When objects are extremely similar and densely packed, precisely interpreting complex language (e.g., \"the second actor counting from left to right\" ) remains an open research problem.\n\nIn the camera-ready version, we will add a dedicated subsection for a more detailed analysis of this issue. This new section will:\n1. Analyze performance in dense scenes, drawing on qualitative examples from our failure analysis (e.g., Figure 17 ) to discuss specific cases where our model succeeds or fails in distinguishing targets from dense distractors.\n2. Clarify the primary sources of error, attributing them to either highly ambiguous visual information or the model's limitations in processing complex relational language.\n3. Offer more detailed pathways for improvement, expanding on our Limitations section . We will suggest future research directions, such as incorporating explicit object-to-object relation modeling modules or leveraging more advanced large language models for deeper semantic reasoning to better handle these complex, dense scenarios."}, {"review_id": "V5ymph9l0A", "reviewer": "Reviewer_P9vU", "summary": "The paper proposes a new task, Spatial Aerial Video Grounding. UAV-SVG, a large-scale benchmark. They further introduce SAVG-DETR, a transformer-based architecture designed to address the challenges of aerial scenes, including small object size, camera motion, and complex spatial context.", "questions": "See the weakness", "limitations": "No. While the paper discusses some technical limitations, it overlooks potential negative societal impacts.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is logically structured, and most technical sections are clearly explained despite the method's complexity.\n\n- The paper presents a well-engineered framework (SAVG-DETR). The evaluations include comparisons with baselines, thorough ablations, and qualitative visualizations.\n\n- The introduction of the UAV-SVG large-scale benchmark, which is underrepresented in existing spatial video grounding datasets.\n\n- The proposed SAVG-DETR introduces several original architectural elements that directly address the challenges of aerial video grounding.\n\nWeaknesses\n\n- While the overall organization is sound, certain sections, particularly the encoder/decoder descriptions, are overly dense and could be more readable. Long technical paragraphs, repeated use of phrases like \"computational explosion.\"\n\n- The benchmark relies partially on LLM-generated expressions (Gemini). While this enables scale, the paper lacks a deeper discussion or analysis of potential risks.\n- The evaluation is heavily focused on UAV-SVG. It remains unclear how well SAVG-DETR generalizes to other grounding datasets or aerial sources (e.g., satellite or oblique-view videos).\n\n- Although the authors decouple the encoder to reduce complexity, the final model still requires substantial computational resources (e.g., 203.08 GFLOPs and 28.7 GB of memory, as shown in Table 4).", "was_revised": true, "final_justification": "The authors addressed my concerns", "rebuttal": ">**Q1**: While the overall organization is sound, certain sections, particularly the encoder/decoder descriptions, are overly dense and could be more readable. Long technical paragraphs, repeated use of phrases like \"computational explosion.\"\n\n**A1**: Thank you for this constructive feedback on the paper's readability. We agree that clarity is crucial, especially in the technical sections. The density of the methodology section was partly a consequence of fitting our detailed architecture into the page limit.\n\nTo address this, we will thoroughly revise Sections 4.1 and 4.2 in the camera-ready version to improve their structure and flow. Our planned revisions include:\n1. Breaking Down Paragraphs: We will divide the long, dense paragraphs into shorter, more digestible ones, with each focusing on a single component or concept.\n2. Using Structural Elements: We will incorporate bulleted or numbered lists to delineate the key steps within our proposed encoder and decoder modules. This will make the architecture's logic easier to follow.\n3. Refining Language: We will carefully revise the text to improve the prose and vary our phrasing, avoiding the overuse of terms like \"computational explosion\".\n\nWe are confident that these revisions will make the core methodology of our paper significantly more accessible to readers.\n\n>**Q2**: The benchmark relies partially on LLM-generated expressions (Gemini). While this enables scale, the paper lacks a deeper discussion or analysis of potential risks.\n\n**A2**: Thank you for raising this important point about the responsible use of  LLMs in dataset creation. We recognize the potential risks, such as factual inaccuracies or biases, and we designed a rigorous, human-centric pipeline specifically to mitigate them.\nOur defense against these risks is a three-step process, as detailed in Section B.1 of the supplementary material :\n1. Guided Generation with Human Priors: We did not use the LLM in a zero-shot fashion. Instead, we provided Gemini with \"object-specific prompt with human priors\", which included manually annotated object descriptions from the original WebUAV-3M dataset. This step grounds the LLM's output in factual, human-verified information from the start.\n2. LLM-based Augmentation: The model was then used to augment these descriptions, generating more diverse and descriptive referring expressions.\n3. Rigorous Manual Verification: This is the most critical step. Every single expression generated by the LLM was subjected to meticulous manual checking and correction by our annotation team. As we state, \"Each expression is manually checked to determine whether the described attributes are correct and whether the referred object can be uniquely distinguished. Correct any errors that exist... If an instance is difficult to describe uniquely and precisely... discard this sample\". This extensive verification process took approximately three months to complete.\n\nTo make this clearer in the paper, we will add a dedicated discussion in the camera-ready version that explicitly addresses the potential risks of LLM usage and details how our human-in-the-loop pipeline systematically neutralizes them.\n\n>**Q3**: The evaluation is heavily focused on UAV-SVG. It remains unclear how well SAVG-DETR generalizes to other grounding datasets or aerial sources (e.g., satellite or oblique-view videos).\n\n**A3**: We appreciate this question regarding the generalizability of our model. Our main objective in this work was to introduce and formally define the novel Spatial Aerial Video Grounding (SAVG) task , a challenging and underexplored area within the low-altitude domain. To this end, we constructed the first large-scale benchmark, UAV-SVG, which presents a unique set of challenges not found in existing datasets. These include:\n* A vast number of extremely small objects, often only a few pixels in size.\n* Rapid and complex camera motion inherent to UAV platforms.\n* A bird's-eye view perspective encompassing wide, dense, and complex scenes.\n\nWe agree that exploring the adaptability of our SAVG-DETR is a valuable avenue for future research.\n* Other Grounding Datasets: While we believe our model could be adapted, its core strengths are in handling the extreme conditions of UAV data, which are less prevalent in ground-level datasets.\n* Other Aerial Sources: While satellite video is very important, there is a lack of SVG datasets for satellite video. Applying our method there would be a compelling, but different, research problem.\n\nOur proposed model, SAVG-DETR, was specifically engineered to tackle these distinct challenges. Key components, such as the decoupled multi-scale encoder , the multi-level progressive decoder for enhancing small object grounding , and the offset-based decoding paradigm for improving prediction consistency across frames with erratic motion, were designed with the explicit properties of the SAVG task in mind.\n\nOur evaluation is intentionally focused on the UAV-SVG benchmark, as it is the most appropriate and direct way to validate the effectiveness of our proposed solutions against the specific problems we set out to solve. As our experiments in Table 2 demonstrate, state-of-the-art methods designed for standard ground-level video grounding show a significant performance drop on UAV-SVG, which confirms that SAVG is a distinct and more challenging task requiring a specialized approach.\n\nTherefore, our main goal was to demonstrate that SAVG-DETR sets a new SOTA on the challenging task it was designed for. While we believe the architectural principles are sound, evaluating its direct generalization to other domains is a significant research direction in itself and falls outside the scope of this foundational work. We will state this more clearly in our \"Limitations and Future Work\" section in the camera-ready paper, positioning the exploration of our model's applicability to other datasets as an important avenue for future investigation.\n\nThank you again for your constructive feedback. We hope this clarifies that our focused evaluation was a deliberate choice to rigorously assess our solution against the novel problem we introduced.\n\n\n>**Q4**: Although the authors decouple the encoder to reduce complexity, the final model still requires substantial computational resources (e.g., 203.08 GFLOPs and 28.7 GB of memory, as shown in Table 4).\n\n**A4**: Thank you for this astute observation. We acknowledge that our model is computationally intensive in absolute terms. Our claim of reducing complexity needs to be understood in the context of our specific architectural goals.\n\nThe core challenge we address is incorporating multi-scale features into a multi-modal transformer architecture, which is essential for grounding small objects in high-resolution aerial videos. A naive implementation of this (Variant C in Table 4) is computationally infeasible and resulted in an out-of-memory error on our hardware. Our key innovation—decoupling the encoder into an intra-scale multi-modal branch and a cross-scale visual-only branch—is what makes this powerful approach computationally tractable. As shown in Table 4, our decoupling strategy not only works but is also more efficient than a more straightforward cascaded design (Variant D), reducing FLOPs by 17% while improving performance.\n\nThe high resource usage is a direct trade-off for the significant leap in performance on the challenging UAV-SVG benchmark. The extensive multi-scale processing is precisely what allows SAVG-DETR to outperform other methods by a large margin (e.g., +7.02% m_vIoU over the efficient SGFDN model ).\n\nWe agree that efficiency is critical for practical applications. As stated in our \"Limitations and Future Work\" section, exploring \"smaller sizes and faster models\"  is a key priority. We will emphasize this trade-off and the need for future work on model efficiency more explicitly in the final version of the paper."}, {"review_id": "jKhR44vRSt", "reviewer": "Reviewer_VN3G", "summary": "The paper introduces a novel task called Spatial Aerial Video Grounding (SAVG), which aims to localize objects in UAV videos based on natural language queries. The authors highlight the limitations of existing spatial video grounding methods, which primarily focus on fixed or handheld camera perspectives with simple scenes, and propose a new benchmark, UAV-SVG, featuring over 2 million frames and 216 diverse object categories. To address the challenges of UAV-based grounding (e.g., small objects, complex motion, and viewpoint changes), the authors propose SAVG-DETR, a transformer-based architecture with three key innovations:\n1.Decoupled encoder: Separates multi-modality and multi-scale spatio-temporal modeling to reduce computational complexity.\n2.Hierarchical progressive decoder: Enhances small object grounding by integrating multi-scale features into language modulation and progressively refining predictions.\n3.Offset-based decoding: Improves prediction consistency across frames by iteratively updating reference anchors.\nExperiments show SAVG-DETR outperforms state-of-the-art methods on the UAV-SVG benchmark, with comprehensive ablation studies validating its design choices.", "questions": "1. Can SAVG-DETR run in real-time on typical UAV hardware (e.g., Jetson AGX)? Inference time/FPS metrics would clarify its practicality for field applications.\n2. How does the model perform on rare object categories (e.g., \"police van\")? Stratified results per superclass would highlight robustness to long-tail distributions.\n3.Could you add a visual example (e.g., in supplementary material) showing how reference anchors refine across decoding stages? This would make the mechanism more intuitive.\n4.Will UAV-SVG be publicly released? If so, under what license? Clarity here is crucial for reproducibility.\n5.Are there systematic failure patterns (e.g., objects below a certain size or speed)? A deeper analysis could guide future improvements.", "limitations": "The authors discuss limitations, such as real-time performance and occlusion handling, and address societal concerns (e.g., surveillance misuse). To further improve the paper, it is recommended that:\n(1) Propose specific mitigation strategies for misuse (e.g., access controls for the dataset/model).\n(2) The experiments focus on partial occlusions (e.g., trucks under bridges), but severe occlusions (e.g., objects hidden behind buildings) are not systematically tested. The offset-based decoding may struggle when the target disappears for multiple frames.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1.Strengths\nThe introduction of UAV-SVG addresses a critical gap in the field, providing a benchmark that reflects real-world UAV challenges like small objects, illumination variations, and fast motion. The dataset's scale (2M+ frames) and diversity (216 categories) make it a valuable resource for future research. The paper is well-written, with clear figures and supplementary material that enhance reproducibility.\n\n2.Weaknesses\nWhile the paper is strong overall, a few areas could be improved. \n(1) The computational efficiency of SAVG-DETR is not thoroughly analyzed. Given that UAVs often operate with limited hardware, real-time performance metrics (e.g., FPS on edge devices like Jetson AGX) would strengthen the practical applicability of the method.\n(2) The benchmark's long-tail distribution (e.g., rare object categories like \"police van\") may introduce bias, yet the paper does not stratify results by object frequency. Providing per-category performance would better illustrate robustness.\n(3) The offset-based decoding mechanism, while effective, could benefit from a more intuitive explanation—perhaps a visual example showing how reference anchors evolve across decoding stages.", "was_revised": true, "final_justification": "After carefully reading the author's response, I found that for Q1, the author explained the discrepancy between FLOP/memory metrics and actual drone deployments, and that resource-constrained devices like the Jetson AGX will be considered in the future. Regarding Q2, the author's rebuttal was unconvincing and failed to dispel my confusion. Regarding Q3, the author currently only plans to build a visualization, which is not yet available. Regarding Q5, the author has not yet provided supporting materials. In summary, quantitative minority class analysis and visualization were promised but not yet delivered. If included in the current submission, these features would have been more likely to be accepted. Therefore, I maintain my original rating.\n\nThe author further responded to questions 2, 3, and 5 and provided additional experiments. The responses resolved my confusion about the paper, and after revisions, the paper now meets the requirements for publication.", "rebuttal": ">**Q1**: Can SAVG-DETR run in real-time on typical UAV hardware (e.g., Jetson AGX)? Inference time/FPS metrics would clarify its practicality for field applications.\n\n**A1**: This is a crucial question regarding the practical deployment of our method. We acknowledge that our current analysis, which focuses on FLOPs and memory usage on high-end GPUs, does not include inference speed on edge devices. Our initial focus was on maximizing grounding accuracy and developing a robust architecture to handle the unique difficulties of aerial videos, such as tiny objects and extreme motion, rather than optimizing for inference speed on edge hardware.\n\nGiven its current complexity (203.08 GFLOPs ), SAVG-DETR in its full form is not optimized for real-time performance on resource-constrained hardware like a Jetson AGX. Achieving this would require dedicated model optimization. We are currently deploying the algorithm and conducting practical verification on low-altitude UAVs. We believe this is a critical next step for real-world application. In the camera-ready version, we will expand our \"Limitations and Future Work\" section to explicitly discuss a roadmap toward a real-time version, which are essential for porting complex transformer models to edge platforms. \n\n\n>**Q2**: How does the model perform on rare object categories (e.g., \"police van\")? Stratified results per superclass would highlight robustness to long-tail distributions.\n\n**A2**: This is an excellent suggestion. The long-tail distribution of object classes is a core challenge of the UAV-SVG benchmark. While our overall results in Table 2 demonstrate strong performance, we agree that a stratified analysis would offer deeper insights into the model's robustness.\n\nFor the camera-ready version, we will add a new table to the supplementary material providing a performance breakdown across the 12 object superclasses defined in our dataset analysis. This will quantitatively show how SAVG-DETR performs on both frequent categories (like 'vehicle' and 'person') and rarer ones. Our qualitative results (e.g., Figures 16-19) already hint at this robustness by showing successful grounding of less-common objects like a \"white water scooter\" and a \"white sightseeing cruise ship\", but we look forward to supporting this with rigorous quantitative data.\n\n\n>**Q3**: Could you add a visual example (e.g., in supplementary material) showing how reference anchors refine across decoding stages? This would make the mechanism more intuitive.\n\n**A3**: We completely agree. A visual aid would greatly demystify the progressive decoding mechanism. We will create a new figure for the supplementary material that illustrates this process. The figure will show a sample video frame and visualize how the predicted bounding box evolves through the decoder layers—starting from an initial, coarse reference anchor and iteratively refining its position and size at each stage based on the predicted offsets, ultimately converging on the ground-truth object. This will provide a clear, intuitive understanding of our offset-based decoding paradigm.\n\n>**Q4**: Will UAV-SVG be publicly released? If so, under what license? Clarity here is crucial for reproducibility.\n\n**A4**: Yes, absolutely. We are committed to open and reproducible research. The UAV-SVG dataset and the source code for our SAVG-DETR model will be made publicly available upon the acceptance of our paper. We have already included them as anonymized supplementary materials for review. We plan to release the UAV-SVG dataset under the Creative Commons CC-BY-NC-SA 4.0 license to encourage academic use while preventing commercial exploitation. The code will be released under a permissive MIT License. We will add an explicit statement to this effect in the paper's conclusion.\n\n\n>**Q5**: Are there systematic failure patterns (e.g., objects below a certain size or speed)? A deeper analysis could guide future improvements.\n\n**A5**: Thank you for pushing for a more rigorous error analysis. Our initial \"Failure Analysis\" section (Sec. D.5.2 ) identifies several challenging scenarios, such as tiny objects among dense distractors and significant viewpoint changes. However, we agree that a more systematic breakdown of failure cases would be highly beneficial for guiding future research.\n\nIn the camera-ready version, we will expand this section to categorize and analyze failure patterns based on specific object and scene attributes. This analysis will investigate model performance under conditions of:\n1. Extreme Small Object Size: Objects occupying fewer than 200 pixels.\n2. High Motion: Cases with significant motion blur or large inter-frame displacement.\n3. Severe Occlusion: Instances where the target is fully occluded for multiple consecutive frames.\n4. Linguistic Ambiguity: Scenarios where complex spatial or relational language leads to incorrect grounding.\n\n**Responses to Limitations**\n\n**(1) Propose specific mitigation strategies for misuse (e.g., access controls for the dataset/model).**\n\n**A(1)**: This is a very important point. We will expand our \"Societal Impact\" section  to include specific, actionable mitigation strategies. As mentioned in our response to the NeurIPS checklist, our release plan will incorporate:\n1. Gated Access and User Agreement: We will release the dataset and models through a registration process where users must agree to a license that explicitly prohibits use for surveillance, illegal tracking, or any other unethical applications.\n2. Purposeful Data Curation: We note that the dataset was curated from public sources and is focused on general scenes like traffic, public spaces, and scenery, intentionally avoiding private or sensitive environments.\n\n**(2) The experiments focus on partial occlusions... severe occlusions... are not systematically tested.**\n\n**A(2)**: This is a valid observation and a key limitation of the current work. Our offset-based decoding mechanism  is indeed most effective when the target is at least partially visible, allowing the reference anchors to be updated. During prolonged, full occlusions, the model would likely struggle as the visual signal disappears.\nWe will make this limitation more explicit in the revised manuscript. We will enhance the \"Limitations and Future Work\" section  to identify severe occlusion as a primary challenge for future research. We will also propose potential solutions to explore, such as integrating motion prediction models (e.g., Kalman filters) to propagate object state during occlusion or developing a re-detection module to re-acquire the target after it reappears. This provides a clear path for building upon our work to create even more robust systems."}, {"review_id": "z3g48tZofT", "reviewer": "Reviewer_GFfV", "summary": "In this paper, the author proposes a video grounding UAV-view dataset. It is based on an existing tracking dataset, WebUAV-3M.\nThey also propose a detection method for the small objects. The main idea is easy to follow. \nMy main concern is the dataset contribution.\nThe work seems to borrow bboxes and sentences from WebUAV-3M, and do incremental filtering and augmentation.", "questions": "Please see the cons.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The dataset makes sense. It is based on an existing tracking dataset, WebUAV-3M.\n2. The performance is competitive.\n\nCons:\n\n1. Some dataset details are missing. \n\n- Will the dataset introduce some biases due to the Gemini or other off-the-shelf models? \n- So there is only one target in every frame of the dataset? since the dataset is based on the single-object tracking dataset.\n\n2.  Dataset contribution. \n\n- Are the bbox annotations all from WebUAV-3M? \n- The author only does the text augmentation and filtering?  I am confusing with \"The Gemini 1.5 augments more referring expressions with reference to the keyframe images and human prior information. \" \n- From Figure 2 bottom, I do not see the neccessary to annotate more texts. All sentences are similar.\n\n3. The second contribution about multi-scale and multi-modality.\nI do not see the much difference from the Q-former in Blip or DeTR structure. \n\n4. The third contribution about multi-scale is similar with the second contribution. \n\n5. Some existing datasets, GeoText-1652, also has similar bbox-sentence annotations. \n\n6. According to the table 4, the comparison is not so fair. \nThe proposed method, despite the same backbone with TubeDETR or SGFDN, are with more added fusion blocks. The 203 FLOPS are much larger than 144 or 53. The memory usage are also doubled.", "was_revised": true, "final_justification": "Thank you. The author addressed my questions about the dataset. I think the annotation efforts are not significant compared to the early dataset. The technical contribution is not significant as well. So I slightly raise my score.", "rebuttal": ">**Q1**: Some dataset details are missing. (1) Will the dataset introduce some biases due to the Gemini or other off-the-shelf models? (2) So there is only one target in every frame of the dataset? since the dataset is based on the single-object tracking dataset.\n\n**A1-(1)**: Thank you for these important clarifying questions. We designed a rigorous, human-centric pipeline specifically to mitigate biases.\n1. Guided Generation with Human Priors: We provided Gemini with \"object-specific prompt with human priors\", which included manually annotated object descriptions from the original WebUAV-3M dataset. This step grounds the LLM's output in factual, human-verified information from the start.\n2. LLM-based Augmentation: The model was then used to augment these descriptions, generating more diverse and descriptive referring expressions.\n3. Rigorous Manual Verification: This is the most critical step. Every single expression generated by the LLM was subjected to meticulous manual checking and correction by our annotation team. This extensive verification process took approximately three months to complete.\n\nThis ensures the final expressions are of high quality and significantly reduces the risk of model-induced biases.\n\n**A1-(2)**: The reviewer is correct in observing that each video-query pair in UAV-SVG benchmark is designed to localize one unique object. However, we would like to respectfully clarify that this is an intentional and fundamental design choice aligned with the established definition of the task, rather than a limitation imposed by the data source.\n1. The fundamental goal of visual grounding, and by extension SVG, is to localize the specific object or region uniquely described by a natural language query. This one-to-one mapping is a standard and defining characteristic of the task.\n2. We firmly believe that establishing a robust methodology to solve the single, unique object SAVG task is a critical and highly challenging first step. Mastering this is essential before tackling more complex scenarios.\n3. Our future plans involve a direct progression from this single-object paradigm to multi-object localization. Ultimately, we aim to build towards a generalized SAVG framework capable of localizing any number of objects (0 to N) described in a complex query.\n\n>**Q2**: Dataset contribution. (1) Are the bbox annotations all from WebUAV-3M? The author only does the text augmentation and filtering? (2) I am confusing with \"The Gemini 1.5 augments more referring expressions with reference to the keyframe images and human prior information.\" From Figure 2 bottom, I do not see the necessity to annotate more texts. All sentences are similar.\n\n**A2-(1)**: The initial bounding box annotations were sourced from WebUAV-3M. However, our contribution involved a \"manual data cleaning and spatio-temporal tube correction\" process . We manually verified and corrected the spatio-temporal tubes to ensure high accuracy and temporal consistency, which was a labor-intensive but necessary step to guarantee the quality of the benchmark.\n\n**A2-(2)**: Our contribution goes beyond simple augmentation and filtering. The process was a comprehensive, three-stage pipeline as shown in Figure 2: (1) Video and annotation correction, (2) Guided expression generation, and (3) Rigorous manual verification. Gemini generated a richer set of expressions describing the same object from different perspectives (e.g., appearance, motion, location) based on keyframe images and human prior information. The \"with reference to the keyframe images\" means that to sample an average of 4 frames of images as the input for Gemini. The \"human prior information\" refers to object-specific prompt with human priors, that is, the artificial description of the first frame object in WebUAV-3M.\n\n**Diversity of Expressions**: We appreciate you pointing out the example in Figure 2. While the sentences all refer to the \"white harvester,\" they are designed to test a model's understanding of different linguistic constructs. For instance, the expressions describe the object based on:\n- Action: \"working in the field\" \n- Motion: \"moving from left to right\" \n- Fine-grained Attributes: \"with red accents that is harvesting the yellow rice\" \n- Spatial Context: \"traveling in a row of the field\" \nThis linguistic diversity is crucial for building robust grounding models that can handle the varied ways humans describe objects.\n\n>**Q3 & Q4**: The second contribution about multi-scale and multi-modality. I do not see the much difference from the Q-former in Blip or DeTR structure. The third contribution about multi-scale is similar with the second contribution.\n\n**A3 & A4**: Thank you for prompting us to better articulate the novelty of our contributions. There appears to be a misunderstanding, which we are happy to clarify. Our second and third contributions are distinct and focus on the encoder and decoder, respectively.\n\n**Contribution (ii): Novelty of the Encoder vs. Q-Former/DETR**:\n\nOur encoder is fundamentally different from standard DETR and Q-Former. Our primary innovation is an architecture designed for multi-scale spatio-temporal inputs. To overcome the computational explosion, our encoder efficiently decouples the multi-modality and multi-scale spatio-temporal modeling into two branches:\n1. Intra-scale Multi-modality Interaction: Instead of fusing the language query with all visual scales, our encoder performs this expensive cross-modal attention only on the highest-level, most semantically rich visual features. This captures the core conceptual alignment efficiently.\n2. Cross-scale Visual-only Fusion: The crucial detail from lower-level visual features (which is vital for localizing small aerial objects) is then integrated through a separate, lightweight, CNN-based fusion branch. This avoids the quadratic complexity of transformer attention for this step.\n\nQ-Former/DETR is designed for static image-text representation learning. A standard Q-Former or DETR-like structure would be computationally prohibitive if applied directly to the high-dimensional input of our SAVG task (multiple frames, multiple visual scales, plus a language query).\n\n**Contribution (iii): Novelty of the Decoder**:\n\nTo enhance small object grounding, we propose the modulation module to integrate multi-scale information into language features and the multi-level progressive spatial decoder to decode from high level to low level. This contribution concerns our decoder, which is distinct from the encoder. \nThe decoder processes features from high-level to low-level, progressively refining the prediction. This hierarchical approach is tailored for grounding, especially for small objects where fine details from low-level features are critical.\n\nIn the camera-ready version, we will revise the text to more clearly delineate these distinct contributions to our encoder and decoder architectures.\n\n>**Q5**: Some existing datasets, GeoText-1652, also has similar bbox-sentence annotations.\n\n**A5**: Thank you for your valuable feedback. While GeoText-1652 indeed provides bounding box and sentence annotations for aerial scenes, it has several critical limitations that make it unsuitable for the specific task of SVG.\n1. GeoText-1652 is primarily oriented towards localizing large-scale, static geographical objects, with a significant focus on buildings. Consequently, the annotated bounding boxes in GeoText-1652 are typically very large. A primary challenge of our UAV-SVG dataset is the prevalence of dynamic small objects. \n2. This task requires referring expressions that are precise and unambiguous to uniquely identify a specific object and its spatio-temporal behavior, especially in cluttered scenes with similar distractors. Upon our careful examination of GeoText-1652, we found the quality of its sentence annotations to be insufficient for this purpose. The language used is often ambiguous and contains inaccuracies.\n\n\n>**Q6**: According to the table 4, the comparison is not so fair. The proposed method, despite the same backbone with TubeDETR or SGFDN, are with more added fusion blocks. The 203 FLOPS are much larger than 144 or 53. The memory usage are also doubled.\n\n**A6**: We sincerely thank the reviewer for this insightful comment and appreciate the opportunity to clarify the fairness and rationale behind our experimental comparisons. We respectfully argue that the comparison is indeed appropriate and highlights the core contributions of our work.\n1. To effectively address the task of aerial object localization, we found it necessary to introduce multi-scale visual features. This makes a fair comparison with single-scale methods like TubeDETR or SGFDN challenging. SGFDN is a method specifically designed for efficiency, and while it has high computational efficiency, its performance is very low. TubeDETR is more efficient than our method, but its performance is also lower.\n2. Directly incorporating multi-scale visual features into TubeDETR leads to a computational explosion. The analysis of Variant C in Table 4 proves this point; this direct multi-scale, multi-modal transformer approach resulted in prohibitive memory requirements and a substantial increase in FLOPs. Our analysis of variants A through E demonstrates that our proposed method achieves the best balance between performance and efficiency among the feasible multi-scale approaches.\n3. This situation also explains why a significant body of research is dedicated to designing multi-scale DETR methods. Standard DETR-based models do not perform well on UAV aerial targets with their unique challenges. Introducing multi-scale features inevitably increases complexity, which is an unavoidable trade-off for achieving higher accuracy in this domain. Therefore, we believe our experiments are reasonable and appropriately contextualized within the problem space, highlighting the effectiveness of our proposed architecture in handling these complexities."}]

Author rebuttal:

Meta-review: Paper Summary:  
This paper introduces a new large-scale benchmark for Spatial Video Grounding (SVG) specifically tailored to UAV footage. Concurrently, the authors propose SAVG-DETR, a Transformer-based framework designed to handle the unique challenges of aerial scenes, such as small objects and complex camera motion.

Main Strengths and Weaknesses:  
As acknowledged by all reviewers, the primary strength of the paper is the creation of the challenging UAV-SVG benchmark, which addresses a critical gap in the field and provides a valuable resource for future research. A key weakness, noted by multiple reviewers, is the high computational cost (in terms of GFLOPs and memory) of the proposed SAVG-DETR model, which raises questions about its practical applicability on resource-constrained UAV hardware.

Rebuttal Analysis:  
The rebuttal effectively addresses the concerns raised by the reviewers. Reviewer VN3G was initially unconvinced but was satisfied after a second round of responses that included additional experiments, ultimately concluding the paper meets publication standards. Reviewer GFfV found their questions answered and slightly raised their score, though they remained somewhat skeptical about the significance of the contribution of the dataset compared to existing work.

Final Justification and Recommendation:  
All four reviewers converged on a positive recommendation, viewing the paper as a solid technical contribution. The core contributions—a much-needed dataset for aerial video analysis and a corresponding high-performing model—are significant. The authors successfully addressed most of the technical and clarity issues raised during the review process. While valid concerns remain regarding the computational efficiency for real-world deployment, the consensus is that the strengths and robust contributions to an under-explored area outweigh these weaknesses.
Decision: Accept (poster)