Title: Where Does It Exist from the Low-Altitude: Spatial Aerial Video Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Aerial Video

Spatial Video Grounding

Multi modality Spatio-temporal Interaction

Hierarchical Progressive  Decoder
Abstract: The task of localizing an object's spatial tube based on language instructions and video, known as spatial video grounding (SVG), has attracted widespread interest. Existing SVG tasks have focused on ego-centric fixed front perspective and simple scenes, which only involved a very limited view and environment. However, UAV-based SVG remains underexplored, which neglects the inherent disparities in drone movement and the complexity of aerial object localization. To facilitate research in this field, we introduce the novel spatial aerial video grounding (SAVG) task. Specifically, we meticulously construct a large-scale benchmark, UAV-SVG, which contains over 2 million frames and offers 216 highly diverse target categories. To address the disparities and challenges posed by complex aerial environments, we propose a new end-to-end transformer architecture, coined SAVG-DETR. The innovations are three-fold. 1) To overcome the computational explosion of self-attention when introducing multi-scale features, our encoder efficiently decouples the multi-modality and multi-scale spatio-temporal modeling into intra-scale multi-modality interaction and cross-scale visual-only fusion. 2) To enhance small object grounding ability, we propose the language modulation module to integrate multi-scale information into language features and the multi-level progressive spatial decoder to decode from high to low level. The decoding stage for the lower-level vision-language features is gradually increased. 3) To improve the prediction consistency across frames, we design the decoding paradigm based on offset generation. At each decoding stage, we utilize reference anchors to constrict the grounding region, use context-rich object queries to predict offsets, and update reference anchors for the next stage. From coarse to fine, our SAVG-DETR gradually bridges the modality gap and iteratively refines reference anchors of the referred object, eventually grounding the spatial tube. Extensive experiments demonstrate that our SAVG-DETR significantly outperforms existing state-of-the-art methods. The dataset and code will be available at here.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author proposes a video grounding UAV-view dataset. It is based on an existing tracking dataset, WebUAV-3M.\nThey also propose a detection method for the small objects. The main idea is easy to follow. \nMy main concern is the dataset contribution.\nThe work seems to borrow bboxes and sentences from WebUAV-3M, and do incremental filtering and augmentation.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The dataset makes sense. It is based on an existing tracking dataset, WebUAV-3M.\n2. The performance is competitive.\n\nCons:\n\n1. Some dataset details are missing. \n\n- Will the dataset introduce some biases due to the Gemini or other off-the-shelf models? \n- So there is only one target in every frame of the dataset? since the dataset is based on the single-object tracking dataset.\n\n2.  Dataset contribution. \n\n- Are the bbox annotations all from WebUAV-3M? \n- The author only does the text augmentation and filtering?  I am confusing with \"The Gemini 1.5 augments more referring expressions with reference to the keyframe images and human prior information. \" \n- From Figure 2 bottom, I do not see the neccessary to annotate more texts. All sentences are similar.\n\n3. The second contribution about multi-scale and multi-modality.\nI do not see the much difference from the Q-former in Blip or DeTR structure. \n\n4. The third contribution about multi-scale is similar with the second contribution. \n\n5. Some existing datasets, GeoText-1652, also has similar bbox-sentence annotations. \n\n6. According to the table 4, the comparison is not so fair. \nThe proposed method, despite the same backbone with TubeDETR or SGFDN, are with more added fusion blocks. The 203 FLOPS are much larger than 144 or 53. The memory usage are also doubled.\n\nQUESTIONS:\nPlease see the cons.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel task called Spatial Aerial Video Grounding (SAVG), which aims to localize objects in UAV videos based on natural language queries. The authors highlight the limitations of existing spatial video grounding methods, which primarily focus on fixed or handheld camera perspectives with simple scenes, and propose a new benchmark, UAV-SVG, featuring over 2 million frames and 216 diverse object categories. To address the challenges of UAV-based grounding (e.g., small objects, complex motion, and viewpoint changes), the authors propose SAVG-DETR, a transformer-based architecture with three key innovations:\n1.Decoupled encoder: Separates multi-modality and multi-scale spatio-temporal modeling to reduce computational complexity.\n2.Hierarchical progressive decoder: Enhances small object grounding by integrating multi-scale features into language modulation and progressively refining predictions.\n3.Offset-based decoding: Improves prediction consistency across frames by iteratively updating reference anchors.\nExperiments show SAVG-DETR outperforms state-of-the-art methods on the UAV-SVG benchmark, with comprehensive ablation studies validating its design choices.\n\nSTRENGTHS AND WEAKNESSES:\n1.Strengths\nThe introduction of UAV-SVG addresses a critical gap in the field, providing a benchmark that reflects real-world UAV challenges like small objects, illumination variations, and fast motion. The dataset's scale (2M+ frames) and diversity (216 categories) make it a valuable resource for future research. The paper is well-written, with clear figures and supplementary material that enhance reproducibility.\n\n2.Weaknesses\nWhile the paper is strong overall, a few areas could be improved. \n(1) The computational efficiency of SAVG-DETR is not thoroughly analyzed. Given that UAVs often operate with limited hardware, real-time performance metrics (e.g., FPS on edge devices like Jetson AGX) would strengthen the practical applicability of the method.\n(2) The benchmark's long-tail distribution (e.g., rare object categories like \"police van\") may introduce bias, yet the paper does not stratify results by object frequency. Providing per-category performance would better illustrate robustness.\n(3) The offset-based decoding mechanism, while effective, could benefit from a more intuitive explanation—perhaps a visual example showing how reference anchors evolve across decoding stages.\n\nQUESTIONS:\n1. Can SAVG-DETR run in real-time on typical UAV hardware (e.g., Jetson AGX)? Inference time/FPS metrics would clarify its practicality for field applications.\n2. How does the model perform on rare object categories (e.g., \"police van\")? Stratified results per superclass would highlight robustness to long-tail distributions.\n3.Could you add a visual example (e.g., in supplementary material) showing how reference anchors refine across decoding stages? This would make the mechanism more intuitive.\n4.Will UAV-SVG be publicly released? If so, under what license? Clarity here is crucial for reproducibility.\n5.Are there systematic failure patterns (e.g., objects below a certain size or speed)? A deeper analysis could guide future improvements.\n\nLIMITATIONS:\nThe authors discuss limitations, such as real-time performance and occlusion handling, and address societal concerns (e.g., surveillance misuse). To further improve the paper, it is recommended that:\n(1) Propose specific mitigation strategies for misuse (e.g., access controls for the dataset/model).\n(2) The experiments focus on partial occlusions (e.g., trucks under bridges), but severe occlusions (e.g., objects hidden behind buildings) are not systematically tested. The offset-based decoding may struggle when the target disappears for multiple frames.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new task, Spatial Aerial Video Grounding. UAV-SVG, a large-scale benchmark. They further introduce SAVG-DETR, a transformer-based architecture designed to address the challenges of aerial scenes, including small object size, camera motion, and complex spatial context.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is logically structured, and most technical sections are clearly explained despite the method's complexity.\n\n- The paper presents a well-engineered framework (SAVG-DETR). The evaluations include comparisons with baselines, thorough ablations, and qualitative visualizations.\n\n- The introduction of the UAV-SVG large-scale benchmark, which is underrepresented in existing spatial video grounding datasets.\n\n- The proposed SAVG-DETR introduces several original architectural elements that directly address the challenges of aerial video grounding.\n\nWeaknesses\n\n- While the overall organization is sound, certain sections, particularly the encoder/decoder descriptions, are overly dense and could be more readable. Long technical paragraphs, repeated use of phrases like \"computational explosion.\"\n\n- The benchmark relies partially on LLM-generated expressions (Gemini). While this enables scale, the paper lacks a deeper discussion or analysis of potential risks.\n- The evaluation is heavily focused on UAV-SVG. It remains unclear how well SAVG-DETR generalizes to other grounding datasets or aerial sources (e.g., satellite or oblique-view videos).\n\n- Although the authors decouple the encoder to reduce complexity, the final model still requires substantial computational resources (e.g., 203.08 GFLOPs and 28.7 GB of memory, as shown in Table 4).\n\nQUESTIONS:\nSee the weakness\n\nLIMITATIONS:\nNo. While the paper discusses some technical limitations, it overlooks potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes contributions to UAV-based spatial video grounding (SVG) by introducing UAV-SVG, a large-scale benchmark addressing the lack of specialized datasets for aerial video analysis, and proposing SAVG-DETR, a novel framework with task-specific designs that outperforms existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper makes a valuable contribution by establishing UAV-SVG, a novel and challenging benchmark for spatial video grounding in aerial scenes, addressing a critical gap in the field.\n2) The proposed SAVG-DETR architecture demonstrates thoughtful design choices tailored to the unique demands of aerial video analysis, leveraging transformer-based approaches effectively.\n3) The method achieves state-of-the-art performance, supported by comprehensive experiments and rigorous evaluation.\n4) The manuscript is well-written, with clear explanations and thorough analysis, including insightful discussions of failure cases that enhance future research directions.\n\nWeaknesses:\n1) Figure 3(b) requires revision, as the origin and purpose of the input blue and pink boxes are unclear. These elements should be explicitly explained or visually adjusted to align with the output from phase (a) to avoid confusion.\n2) The problem definition, currently relegated to supplementary materials, should be integrated into the main Methodology section to improve readability and ensure immediate context for readers.\n3) I believe this aerial SVG task is rather difficult when it comes to grounding upon dense objects environment (a common scenario in aerial imagery). The authors should provide additional analysis in the camera-ready version to clarify whether performance limitations stem from task complexity or methodological constraints, offering potential pathways for improvement.\n\nQUESTIONS:\nPlease see the Weaknesses part above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}