Title: CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Transformer

Multi-Object Relationship Modeling

Aerial Videos
Abstract: Video scene graph generation (VidSGG) has emerged as a transformative approach to capturing and interpreting the intricate relationships among objects and their temporal dynamics in video sequences. In this paper, we introduce the new AeroEye dataset that focuses on multi-object relationship modeling in aerial videos. Our AeroEye dataset features various drone scenes and includes a visually comprehensive and precise collection of predicates that capture the intricate relationships and spatial arrangements among objects. To this end, we propose the novel Cyclic Graph Transformer (CYCLO) approach that allows the model to capture both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The proposed approach also allows one to handle sequences with inherent cyclical patterns and process object relationships in the correct sequential order. Therefore, it can effectively capture periodic and overlapping relationships while minimizing information loss. The extensive experiments on the AeroEye dataset demonstrate the effectiveness of the proposed CYCLO model, demonstrating its potential to perform scene understanding on drone videos. Finally, the CYCLO method consistently achieves State-of-the-Art (SOTA) results on two in-the-wild scene graph generation benchmarks, i.e., PVSG and ASPIRe.

CYCLO : Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos

Abstract

Video scene graph generation (VidSGG) has emerged as a transformative approach to capturing and interpreting the intricate relationships among objects and their temporal dynamics in video sequences. In this paper, we introduce the new AeroEye dataset that focuses on multi-object relationship modeling in aerial videos. Our AeroEye dataset features various drone scenes and includes a visually comprehensive and precise collection of predicates that capture the intricate relationships and spatial arrangements among objects. To this end, we propose the novel Cyclic Graph Transformer (CYCLO) approach that allows the model to capture both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The proposed approach also allows one to handle sequences with inherent cyclical patterns and process object relationships in the correct sequential order. Therefore, it can effectively capture periodic and overlapping relationships while minimizing information loss. The extensive experiments on the AeroEye dataset demonstrate the effectiveness of the proposed CYCLO model, demonstrating its potential to perform scene understanding on drone videos. Finally, the CYCLO method consistently achieves State-of-the-Art (SOTA) results on two in-the-wild scene graph generation benchmarks, i.e ., PVSG and ASPIRe.

1 Introduction

Visual scene understanding has shown significant progress in extracting semantic information from images and videos using deep learning algorithms [1, 2, 3]. Building upon the significant progress in visual scene understanding using deep learning algorithms [4, 5], video scene graph generation (VidSGG) extends the concept of Scene Graph Generation (SGG) from static images to dynamic video, representing object relationships within a graph structure that evolves over time. VidSGG [6, 7, 8] focus on the temporal dimension by constructing a dynamic graph structure, encapsulating the spatial and temporal relationships among object interactions across frames. This helps in understanding human-object interactions [9, 10], temporal events [11, 12, 13], and reasoning [14, 15]. However, drone-captured videos present unique challenges due to larger image sizes and higher object density in Unmanned Aerial Vehicle (UAV) datasets [16, 17, 18]. Despite recent advances in tiny object detection [19, 20, 21], current algorithms still need to effectively model object interactions and their temporal evolution in aerial videos, which have various applications in surveillance, disaster response.

In this paper, we introduce AeroEye, the first dataset for Video Scene Graph Generation in dronecaptured videos featuring Aer ialO bliqueEye views. AeroEye distinguishes itself by showcasing a rich tapestry of aerial videos and an extensive set of predicates describing the intricate relations and positions of multi-objects. To address multi-object relationship modeling in aerial videos from the AeroEye dataset, we propose the Cyclic Graph Transformer (CYCLO). This new approach can establish circular connectivity among frames and enables the model to capture direct and long-range temporal relationships. By continuously updating history across a ring topology, CYCLO allows the model to handle sequences with inherent cyclic patterns, facilitating the processing of object relationships in the correct temporal order. Furthermore, CYCLO provides several advantages to VidSGG, including the ability to model periodic and overlapping relationships, predict object interactions by reasoning from previous cycles, facilitate information transfer across frames, and efficiently utilize long sequences, addressing the limitations of prior methods [22, 8]. They usually struggle with long-term dependencies due to the diminishing influence of inputs over time.

Contributions of this Work. There are three main contributions to this work. First, we introduce a new AeroEye dataset for VidSGG in drone videos, augmented with numerous predicates and diverse scenes to capture the complex relationships in aerial videos. Second, we propose the CYCLO approach, utilizing circular connectivity among frames to enable periodic and overlapping relationships. It allows the model to capture long-range dependencies and process object interactions in the appropriate temporal arrangement. Finally, the proposed CYCLO approach outperforms prior methods on two large-scale in-the-wild VidSGG datasets, including PVSG [7] and ASPIRe [8]. Interestingly, using the same method ( e.g ., our CYCLO), the ratio of correct predictions to incorrect predictions (R/mR) on AeroEye is higher than PVSG (Tables 4 and 6), despite having more predicates (Table 1) and tiny objects. This suggests that our dataset is less visually ambiguous than PVSG.

2 Related Work

In this section, we review the existing datasets and benchmarks for Visual Scene Graph Generation, followed by a summary of the key challenges and issues related to Video Scene Graph Generation.

2.1 Visual Scene Graph Generation Datasets and Benchmarks

Datasets. VisDrone [17], DOTA [16], and SODA-A [36] image datasets, along with UAVid [46], UAVDT [37], and MAVREC [18] video datasets, offer high-resolution UAV datasets that enable precise object detection in dynamic scenes. While these UAV datasets focus on object detection, the Visual Genome [23] pioneered image-based SGG, and the Action Genome [6] dataset extended this concept to capture dynamic interactions within videos. Recently, ASPIRe [8] and SportsHHI [29] emphasize diverse human-object relationships and sports-specific player interactions. Additionally, PSG-4D [47] expands the VidSGG to encompass the 4D domain, bridging the gap between raw visual data and high-level understanding. In Table 1, we present a comparative overview of UA V-based and SGG datasets for images and videos, emphasizing their unique characteristics and advantages.

Benchmarks. The existing benchmark focuses on Image Scene Graph Generation (ImgSGG) and Video Scene Graph Generation (VidSGG). ImgSGG identifies and categorizes relationships between objects within an image into predefined relational categories, including Transformer-based

perspectives. # denotes the number of corresponding items. The best values in drone

blocks are highlighted

.

methods [27, 48, 49, 50] and generative-based models [51, 52, 53]. VidSGG leverages the dynamic nature of object interactions over time to better identify relationships, as the temporal dimension of videos provides a richer context for understanding semantic connections within the scene. Current methods using hierarchical structures [54, 8] or Transformer architectures [22, 55, 56, 57] excel at capturing long-range dependencies and complex interactions, advancing video understanding in video captioning[11, 12, 13], visual question answering [14, 15], and video grounding [58, 59, 60].

2.2 Video Scene Graph Generation

VidSGG can be categorized into two main types based on the granularity of its graph representation. Video-level SGG represents object trajectories as graph nodes, capturing constant relations between objects for a video. Various methods have been proposed to address this problem, incorporating Conditional Random Fields [61], abstracting videos [62], and iterative relation inference techniques [63] on fully connected spatio-temporal. However, focusing primarily on recognizing video-level relations directly based on object-tracking [64, 65, 66] results and neglecting frame-level scene graphs results in a cumbersome pipeline highly dependent on tracking accuracy. In contrast, Frame-level SGG defines the graph at the frame level, allowing relations to change over time. The releases of the benchmark datasets [67, 7, 8] have prompted the development of VidSGG models. TRACE [54], for instance, employs a hierarchical relation tree to capture spatio-temporal context information, while CSTTran [22] uses a spatio-temporal transformer to solve the problem. Recently, hierarchical interlacement graph (HIG) [8] abstracts relationship evolution using a sequence of hierarchical graphs.

2.3 Discussions

In this subsection, we conceptually compare our proposed approach with relationship modeling concepts discussed in Section 2.2 as illustrated in Fig. 2. In addition, we highlight the advantages of our approach and discuss the properties that distinguish it from these existing methods.

Concepts. The progressive approach fuses pairwise features between object pairs at each frame, encoding the object relationship at that specific step, followed by a fully connected layer to classify the predicate types. However, it processes frames independently without considering the temporal context. The batch-progressive approach employs a transformer block with positional embeddings on the fused query features. The hierarchical approach represents a video as a sequence of graphs, integrating temporal and spatial information at different levels. The node and edge features are updated at each hierarchical level based on the previous level to capture evolving object relationships.

Limitations. While the batch-progressive approach considers temporal information, both these progressive and batch-progressive approaches have limitations in modeling the full complexity of temporal dynamics and dependencies in the video. In contrast, the hierarchical graph approach can capture complex interactions and relationships between objects by considering the temporal evolution of graphs at different granularity levels. However, the hierarchical graph requires analyzing the entire video before constructing the graph ( i.e . offline method). These limitations underscore the need for more advanced approaches to efficiently model temporal dynamics, adaptively update memory to handle evolving video data, and accurately capture the intricate relationships between objects.

Advantages of Our Design. Inspired by previous work [68, 69], which processes temporal features through iterative feedback loops and circular updating, we propose the CYCLO approach that circularly incorporates an updated history of relationships. In contrast to these methods, which focus on frame-level updates influenced by global features, our approach constructs and refines scene graphs for each frame, capturing static spatial relationships between objects and their dynamic evolution over time. By leveraging circular connectivity, CYCLO establishes a continuous loop of temporal information, ensuring no temporal edge is treated as a boundary. It enables the Transformer to operate online and then capture and update relationships between objects more effectively, correcting erroneous connections. The theoretical properties are included in Section B of Appendices.

3 The Proposed AeroEye Dataset

In this section, we detail the AeroEye dataset annotation process and provide the dataset statistics.

3.1 Dataset Collection

Data Preparation. We leverage videos from the ERA [40] and MAVREC [18] datasets to construct our AeroEye dataset. ERA consists of diverse scenes ranging from rural to urban environments in extreme conditions ( e.g . earthquake, flood, fire, mudslide), daily activities ( e.g . harvesting, plowing, party, traffic collision), and sports activities ( e.g . soccer, basketball, baseball, running, swimming). MAVREC features sparse and dense object distributions and contains typical outdoor activities characterized by many vehicle classes, incorporating viewpoint changes and varying illumination.

a)Progressive

Pair-wise

Fusion

MLP

→pred

MLP

→pred

MLP

pred

MLP

pred

c)Hierarchical

b)Batch-Progressive

MLP

→pred

Transformer

Encoder

+

Positional

MLP

→pred

Encoding.

'd)Ourapproach

pred

MLP

-to theleft-

jpfront of

person

-opponents-

-totheright-

/teammates

hoop

opponents

teammates

opponents

in front of

opposite

opposite

opposite

behind

ball

person

passing

opponents

person

Relationship Classes and Instance Formulation. In Fig. 3, we focus on two aspects of object relationships: positions ( e.g . in front of, behind, next to) and relations, which consist of movement actions ( e.g . chasing, towing, overtaking) and collision actions ( e.g . hitting, crashing, colliding). These relationships are semantically complex and require detailed spatio-temporal context reasoning for recognition. Following previous PVSG benchmark datasets [6, 7, 8], we define relationship instances at the frame level, considering the long-term spatial-temporal context. Each instance is formulated as a triplet <s,o, p> , where s and o denote the bounding boxes of the subject and object, and p represents the predicate ( i.e . position and relation), included in Tables A.8, A.9 which are summarized in Fig. 4. In addition, Fig. A.11 presents selected samples from our AeroEye dataset.

3.2 Data Specification

Data Annotation. We annotate keyframes at 5FPS to capture frequent and rapid changes in positions and relations in aerial videos, reducing redundancy while keeping up with interaction changes. Our two-stage annotation pipeline first performs object localization and tracking and then relationship instance annotation . To generate diverse predicates, we leverage the GPT4RoI [70] model, which combines visual and linguistic data to generate detailed

cyclinglanecuiture

llecting

facing

P

二

isolated

regulated

holdingalong

Industria1

acrossscenic

1

wo

U

SutpTtng

marathon

dys

agricultural

followi

1oading

tending

school

leisoure

process

tasks

inside

naritine

aligned

seasonal

farm

riverside.

infrastructure festival

navigating

AeM

city streets

scenicroute

public

C1.NC

traffic

leading

path

movement

surveil

ouer

landslide

moving

in

or

b.0

park

conductingarearoute

crashing

mudslide

affected

congested

g

beach

Droad

nergency

ButppaM

s

travelir

2

outdoor

g

festive

urban

managing

hosting

os

surroundeddockingmusic

t

1

1ive

race

atmosphere

racing

engaged

event

wa ter

handling

en

operatingoactivities

conffict

utilizing

joying

erging

game

participating

concert

fatiar

teamcohesion

celebrating

undergoing

waterfront

historical

descriptions of object relationships within specified regions of interest. Although we annotate relationship instances frame by frame, as illustrated in Fig. 3, we easily create relationship tubes using the provided object tracking ID by connecting the same pair of objects with the same relationship predicate across consecutive frames. The annotation file includes object information ( i.e ., bounding boxes, category names, and tracking IDs) and relationships within each frame. Details on the quality control process and annotation examples can be found in Section A.2 of the Appendices.

25000

8

Frames

Interactivities

Number of Interactivities

Nmber of Frames

20000

D

(l5s bo)

15000

5

10000

5000

2

0

Constructing

Conflict

Cycling

Mudslide

Landslide

Flood

Swimming

Harvesting

Basketball

Boating

Running

Police

Chase

Party

Fire

Car Racing

NonEvent

Ploughing

Baseball

Soccer

Park

Campus

Harbour

Religious Activity

Traffic Collision

Traffic Congestion

Parade Protest

Post Earthquake

Traffic Monitoring

Data Statistics. The AeroEye dataset is a collection of 2,260 videos with 261,503 frames, annotated with over 2.2 million bounding boxes across 56 object categories typically observed from aerial , oblique , and ground perspectives captured by drone. Specifically, our AeroEye dataset consists of 384 predicates divided into two relationship categories: 135 positions and 249 relations. The key strength of AeroEye is its richness in relationships. On average, each video in the dataset has 127 frames, providing moderate temporal depth for capturing detailed interactions. The average number of frames per scene is 8,970, indicating substantial variability and complexity. AeroEye is rich in relationships, with more than 43 million relationship instances. In Table 1, we provide a detailed comparison with related datasets, while Fig. 5 presents statistics on the AeroEye dataset. In addition, predicate definitions and additional statistics are discussed in Sections A.1 and A.4 of the Appendices.

4 The Proposed Approach

In this section, we present our CYCLO approach, including the Spatial Attention Graph and the Cyclic Temporal Graph Transformer . The Spatial Attention Graph captures spatial dependencies and interactions between objects within the frame. In contrast, the Cyclic Temporal Graph Transformer models temporal relationships across frames, capturing short-term and long-term dynamics.

4.1 Problem Formulation

Given an input video with T frames, we construct dynamic scene graphs {G t } T t =1 that encode the relationships among objects within these frames. Each graph G t ( V t , E t ) captures static relationships, where node V consists of objects and edge E denotes the relationship between objects. Each object v i ∈ V has an object category v c i ∈ C v and box coordinates v b i ∈ [0 , 1] 4 . Each relationship e j ∈ E represents the j -th triplet ( s j , o j , p j ), where subject s j and object o j and predicate p j ∈ C p .

4.2 Spatial Attention Graph

Self-attention mechanisms in one-stage object detectors [71, 19] model relationships between objects, allowing insights into the dynamics between entities without relying on additional contextual information. For example, in an aerial parking lot video with cars, vans, and people, if the self-attention layer strongly connects the queries representing the person and the car, it suggests an interaction, such as the person entering the vehicle. Inspired by [50], in our CYCLO approach, to construct the static graph in each frame t , we utilize the DETR decoder to establish bidirectional connections among object queries. In particular, we compute relational representations at each layer l by concatenating the query and key vectors, Q l t and K l t , for every object pair. This process ensures the layer l -1 output seamlessly transitions as input to layer l . We omit the superscript L related to the final layer to simplify the following discussion. At each frame, ̂ R l a,t captures the dynamic interplay of relations at layer l , utilizing their query and key vectors. Furthermore, ̂ R z,t leverages object queries in the final layer for object detection. These relationships are formally defined in Eqn. (1).

Here, ϕ W s and ϕ W o are the linear transformations that process subject and object features, enabling the model to consider both object characteristics and their interrelationships comprehensively. In addition, gating mechanisms g l a,t and g z,t dynamically modulate the contributions from different layers. These gated representations from all layers are then integrated to construct a relation matrix:

where ϕ W G denotes the linear weight applied during the gating process. Finally, the relation matrix is fed into the three-layer perception (MLP) with ReLU activation and a sigmoid function σ , which predicts multiple relationships ( p j ) between pairs of objects ( s j , o j ) . Mathematically, ̂ G t = σ ( MLP ( ̂ R t )) is the graph representation at frame t -th, where ̂ G t ∈ R N × N ×|C p | .

Discussion. Transformer-based approaches to VidSGG effectively capture interactions and temporal changes through self-attention mechanisms, creating detailed scene graphs that reflect video dynamic

relationships. However, these models often struggle to represent the directional and historical aspects of the relationship accurately. While effective at identifying token correlations, the scaled dot-product fails to consider their temporal or spatial ordering. This oversight is particularly critical in videos, where understanding the historical context of relationships is essential. For example, the sequence of relationships leading up to a car crash, including speeding, lane changing, and passing, must be considered. Each interaction change provides crucial historical information that contextualizes the final relationships, vital for enhancing prediction and interpretation. In addition, traditional selft-attention [72] does not adequately capture this essential sequential, directional , and historical information [73]. It highlights the need for advancements in transformer architectures to more effectively integrate the direction and historical sequence of interactions within videos.

4.3 Cyclic Temporal Graph Transformer

We present the Cyclic Spatial-Temporal Graph Transformer to refine the spatial attention graph in each scene, capturing temporal dependencies via subject-object relationships across adjacent frames.

Cyclic Attention. As mentioned in Section 4.2, self-attention does not adequately capture directional and historical information . Therefore, we propose the cyclic attention (CA), defined as in Eqn. (3).

In Eqn. (3), η is a shift term enabling cyclical indexing via mod T . The cyclical indexing, illustrated in Fig. 6, allows for continuous sequence processing by connecting the end to the beginning, which is crucial for predicting movements in dynamic interactions where past events influence future actions ( e.g . a car navigating a roundabout). Cyclical indexing differs from standard self-attention as it is a permutation equivariant without positional encodings. In contrast, cyclical attention is non-permutation equivariant, which depends on the original sequence order. This property is crucial for multiobject relationship modeling, where maintaining the chronological order of interactions is essen-

Queries

(

𝐐

𝑻

)

(

𝐊

𝑻

)

𝐐

𝟑

𝐊

𝑻

^	𝐓

Queries

(

𝐐

𝟏

)

(

𝐊

𝟏

)

𝐐

𝑻

𝐊

𝟏

^	𝐓

Queries

(

𝐐

𝟐

)

(

𝐊

𝟐

)

𝐐

𝟏

𝐊

𝟐

^	𝐓

Queries

(

𝐐

𝟑

)

(

𝐊

𝟑

)

𝐐

𝟐

𝐊

𝟑

^	𝐓

tial. For instance, in a surveillance scenario, the sequence of a car stopping for a pedestrian must preserve the order of events, ensuring the vehicle stops before the pedestrian appears.

Temporal Graph Transformer. Our Temporal Graph Transformer refines spatial attention graphs, { ̂ G t } T t =1 , into a sequence of dynamic graphs {G t } T t =1 , leveraging the temporal dynamics and spatial interactions of objects across video frames. Our approach employs a series of cyclic attention blocks configured within multi-head attention to refine object representations by integrating features from adjacent frames. The core of our approach is the integration of cyclic attention into a multi-head structure, which processes the sequence of input features ̂ Z = { ̂ Z t } T t =1 , represented as in Eqn. (4).

where ϕ W c , ϕ W i q , and ϕ W i k denote the linear transformations. Each head h i ( t ) computes the cyclic attention, integrating information across the video to enhance the temporal relationship at each frame. The outputs from various heads at each frame are integrated into Z ′ t , derived from concatenating all attention head outputs. These heads process features cyclically across different representation subspaces to capture the temporal evolution of relationships in the video. Then, Z is obtained by applying layer normalization (LN) and a skip connection to the aggregated features Z ′ t , where Z = LN ( Z ′ + ̂ Z ) . This step ensures that Z ′ t is stabilized and effectively integrated with the original features ̂ Z , thus dynamically updating the scene graph and ensuring temporal coherence.

In addition, Z t is utilized to construct a new relation matrix R t by applying the transformations in Eqn. (1) and (2). This updated matrix R t refines the relationship dynamics captured in static frames

by correcting spurious or incomplete relationships and incorporating previously omitted ones using the temporal context from the frame sequence. As a result, G t comprehensively represents persistent and transient interactions, including their direction and historical sequence within the video.

Loss Function. Visual object relationships involve predicates that may appear quite similar, such as 'parking next to' and 'stopping next to'. Thereby, we utilize a multi-label margin loss, as in [22]:

In Eqn. (5), r represents a subject-object pair, and P + and P -correspond to positive and negative predicates, respectively. The term ϕ ( r, p ) measures the compatibility of the pair r with the predicate p . Additionally, object distributions are modeled using neural networks with ReLU activation and batch normalization. Cross-entropy loss is applied during the learning process. The total loss is a combination of the multi-label margin loss L p and the cross-entropy loss L ce , defined as:

where λ is a weight balancing the contribution of the cross-entropy loss L ce .

5 Experimental Results

In this section, we discuss the benchmark dataset evaluations and comparisons with SOTA methods.

5.1 Implementation Details

Dataset. We use 10-fold cross-validation on the AeroEye dataset, including 1,797 videos for training and 463 videos for testing. We also evaluate our performance on PVSG [7] and ASPIRe [8] datasets.

Settings. We employ DINO [19] to extract the spatial attention graphs (in Section 4.2). DINO is trained with ResNet-50 backbone and 1500 queries on MAVREC, achieving 92 . 35 mAP on the validation set. The pre-trained detector is applied to baselines, and parameters are fixed during subsequent task training. Our model is trained on 8 × A6000 GPUs using 12 epochs with AdamW optimizer (initial learning rate of 1 e -5 and a batch size of 1 ), gradient clipping (max norm of 5 ).

Evaluation Metrics. We evaluate models on two standard tasks in image-based scene graph generation followed by previous work [74, 7] that are predicate classification ( PredCls ), scene graph classification ( SGCls ), and scene graph detection ( SGDet ). While SGCls predicts relationships given ground truth objects, SGDet involves detecting objects and predicting relationships. These tasks are evaluated using Recall (R@K) and mean Recall (mR@ K ), where K ∈ { 20 , 50 , 100 } .

5.2 Ablation Study

Semantic Dynamics in Cyclic Attention. By altering η (in Eqn. (3)), we consider the permutation or non-equivariance equivariance. If the predictions systematically adapt to the shifts induced by

different η values, it demonstrates a degree of permutation equivariance . Conversely, if the predictions change in ways that do not correspond systematically to these shifts, it may indicate non-permutation equivariance . Table 2 shows a decrease in performance, implying disrupted temporal patterns and non-permutation invariance , indicated by unpredictable output changes relative to input shifts.

Cyclic Dependency. To further validate the cyclic dependency of our model, we discarded frames from every successive frame. Removing frames disrupts the temporal continuity of the sequence, which is crucial for maintaining the the cyclic nature of video. If the cyclic model presupposes that each frame has direct relationships with its adjacent frames circularly, then removing frames could sever these relationships, potentially diminishing the ability to leverage cyclical patterns effectively. Indeed, Table 3 informs a decrease in Recall and mean Recall when frames were reduced.

5.3 Comparisons with Baseline Methods

Table 4 shows CYCLO outperforms other methods across metrics and tasks, surpassing Transformer by 2.95%, 3.03%, and 2.44% in the PredCls, SGCls, and SGDet at R@20, and maintaining its lead even at higher Recall thresholds. Moreover, Table 5 shows significant improvements at mR@20, mR@50, and mR@100, with CYCLO leading HIG by 0.86% in the PredCls task at mR@20 and exceeding HIG by 1.12% and 1.32% in the SGCls and SGDet tasks, respectively, at mR@20. In addition, the consistent performance and low standard deviation of the CYCLO model across Table 4 and 5 demonstrate its robustness and overall superiority. Fig. 7 displays that CYCLO can capture the evolving relationships between objects by updating their positions and interactions.

5.4 Comparisons with State-of-the-Art Methods

We compare our performance on two recent benchmark datasets on VidSGG ( i.e . PVSG and ASPIRe).

Performance on PVSG . In Table 6, we report that the CYCLO approach achieves superior performance compared to other models on the PVSG dataset, particularly in terms of Recall and mean Recall metrics. In particular, CYCLO surpasses HIG and Transformer by 1.23% and 1.81% at R@20, respectively. In addition, it consistently outperforms both models at higher recall rates, including R@50 and R@100, showcasing its effectiveness across various thresholds. Our approach also slightly

driver

driver

polices

police

police

police"

car

approaching

investigating

"investigating-

investigating-

investigating-

investigating-

polices

polices

polices

polices

investigating

polices

polices

investigating

polices

polices

polices

polices

polices"

improves the mean Recall, as the experimental results demonstrate. These results highlight the robustness of our CYCLO approach in recognizing and handling periodic actions, such as cooking, washing, cleaning, exercising, and other routine tasks frequently represented on the PVSG dataset.

Performance on ASPIRe . The ASPIRe dataset includes five distinct interactivity types. However, we focus on position and relation to ensure a fair comparison. As shown in Table 7, CYCLO consistently outperforms existing models across multiple recall and mean recall thresholds. Notably, at R@20, our CYCLO approach outperforms the HIG method, the second-best model, by 0.69% in position and a more substantial 5.03% in relation. Additionally, CYCLO shows significant gains in mean Recall across all evaluated thresholds, demonstrating its effectiveness in tackling the long-tail distribution.

6 Conclusions

We have introduced CYCLO, a novel approach that effectively captures periodic and overlapping relationships, handles extended sequences, and minimizes information loss, making it suitable for complex temporal modeling. In addition, we presented AeroEye, a comprehensive and diverse dataset composed of drone-captured scenes, specifically designed to represent intricate object relationships and spatial positions in aerial videos. Through extensive experiments on the AeroEye dataset and two large-scale in-the-wild datasets ( i.e . ASPIRe and PVSG), we demonstrated the robustness and effectiveness of CYCLO in capturing dynamic interactions and evolving relationships over time.

Limitations. Although our CYCLO approach has achieved impressive performance, it may reveal limitations when dealing with incomplete or discontinuous videos. The periodic and cyclic attention mechanisms, crucial for capturing temporal and spatial object relationships, heavily rely on video continuity and completeness. Interruptions in the sequence, such as missing or discontinuous frames, disrupt the formation of accurate cyclical references, leading to inconsistent and incorrect predictions.

Broader Impacts. The proposed approach improves the capture of object interactions and temporal evolution in aerial and in-the-wild videos, which is critical for surveillance, disaster response, traffic management, and precision agriculture applications. By modeling object interactions over time, CYCLO supports more informed decision-making, leading to safer and more sustainable practices. This advancement opens the door for future work developing surveillance systems that can model complex relationships from drone videos. However, it is important to recognize the potential risks associated with this approach, particularly the possibility of using it for unauthorized surveillance.

Acknowledgment. This work is partly supported by J.B. Hunt Transport Services (JBHunt), NSF Data Science and Data Analytics that are Robust and Trusted (DART), NSF SBIR Phase 2, and Arkansas Biosciences Institute (ABI) grants. We also acknowledge Thanh-Dat Truong for invaluable discussions and the Arkansas High-Performance Computing Center (AHPCC) for providing GPUs.

References

Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2t: Pyramid pooling transformer for scene understanding. IEEE transactions on pattern analysis and machine intelligence , 2022. 2

Naga VS Raviteja Chappa, Pha Nguyen, Page Daniel Dobbs, and Khoa Luu. React: recognize every action everywhere all at once. Machine Vision and Applications , 35(4):102, 2024. 2

Kha Gia Quach, Ngan Le, Chi Nhan Duong, Ibsa Jalata, Kaushik Roy, and Khoa Luu. Non-volume preserving-based fusion to group-level emotion recognition on crowd videos. Pattern Recognition , 128:108646, 2022. 2

Naga Venkata Sai Raviteja Chappa, Pha Nguyen, Thi Hoang Ngan Le, Page Daniel Dobbs, and Khoa Luu. Hatt-flow: Hierarchical attention-flow mechanism for group-activity scene graph generation in videos. Sensors , 24(11):3372, 2024. 2

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 815-824, 2023. 2

Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10236-10247, 2020. 2, 3, 5, 21

Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, et al. Panoptic video scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18675-18685, 2023. 2, 3, 5, 8, 21

Trong-Thuan Nguyen, Pha Nguyen, and Khoa Luu. Hig: Hierarchical interlacement graph approach to scene graph generation in video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024. 2, 3, 4, 5, 8, 21

Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu-Wing Tai, Cewu Lu, and Chi-Keung Tang. Interactiveness field in human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20113-20122, 2022. 2

Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17152-17162, 2023. 2

Hanhua Ye, Guorong Li, Yuankai Qi, Shuhui Wang, Qingming Huang, and Ming-Hsuan Yang. Hierarchical modular network for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17939-17948, 2022. 2, 3

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10714-10726, 2023. 2, 3

Wang Lin, Tao Jin, Ye Wang, Wenwen Pan, Linjun Li, Xize Cheng, and Zhou Zhao. Exploring group video captioning with efficient relational approximation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15281-15290, 2023. 2, 3

Mahmoud Khademi and Oliver Schulte. Deep generative probabilistic graph neural networks for scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 11237-11245, 2020. 2, 3

Bruno Cesar de Oliveira Souza, Marius Aasan, Helio Pedrini, and Adin Ramirez Rivera. Selfgraphvqa: A self-supervised graph neural network for scene-based question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4640-4645, 2023. 2, 3

Jian Ding, Nan Xue, Yang Long, Gui-Song Xia Xia, and Qikai Lu. Learning roi transformer for detecting oriented objects in aerial images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 2, 3

Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):7380-7399, 2021. 2, 3

Aritra Dutta, Srijan Das, Jacob Nielsen, Rajatsubhra Chakraborty, and Mubarak Shah. Multiview aerial visual recognition (mavrec): Can multi-view improve aerial visual perception? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. 2, 3, 4

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations , 2023. 2, 6, 8

Chang Xu, Jian Ding, Jinwang Wang, Wen Yang, Huai Yu, Lei Yu, and Gui-Song Xia. Dynamic coarse-tofine learning for oriented tiny object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7318-7328, 2023. 2

Xiang Yuan, Gong Cheng, Kebing Yan, Qinghua Zeng, and Junwei Han. Small object detection via coarse-to-fine proposal generation and imitation learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6317-6327, 2023. 2

Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatialtemporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16372-16382, 2021. 2, 3, 4, 8

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017. 2, 3, 18

Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5410-5419, 2017. 3

Yuanzhi Liang, Yalong Bai, Wei Zhang, Xueming Qian, Li Zhu, and Tao Mei. Vrr-vg: Refocusing visually-relevant relationships. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10403-10412, 2019. 3

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700-6709, 2019. 3

Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European Conference on Computer Vision , pages 178-196. Springer, 2022. 3, 4, 21

Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In Proceedings of the 25th ACM international conference on Multimedia , pages 1300-1308, 2017. 3

Tao Wu, Runyu He, Gangshan Wu, and Limin Wang. Sportshhi: A dataset for human-human interaction detection in sports videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024. 2, 3

Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval , pages 279-287, 2019. 3

Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10668-10677, 2020. 3

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision , pages 1-23, 2022. 3

Chang Xu, Jinwang Wang, Wen Yang, Huai Yu, Lei Yu, and Gui-Song Xia. Detecting tiny objects in aerial images: A normalized wasserstein distance and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing , 190:79-93, 2022. 3

Gong Cheng, Jiabao Wang, Ke Li, Xingxing Xie, Chunbo Lang, Yanqing Yao, and Junwei Han. Anchorfree oriented proposal generator for object detection. IEEE Transactions on Geoscience and Remote Sensing , 60:1-11, 2022. 3

Luigi Riz, Andrea Caraffa, Matteo Bortolon, Mohamed Lamine Mekhalfi, Davide Boscaini, André Moura, José Antunes, André Dias, Hugo Silva, Andreas Leonidou, et al. The monet dataset: Multimodal drone thermal dataset recorded in rural scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2545-2553, 2023. 3

Gong Cheng, Xiang Yuan, Xiwen Yao, Kebing Yan, Qinghua Zeng, Xingxing Xie, and Junwei Han. Towards large-scale small object detection: Survey and benchmarks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 2, 3

Dawei Du, Yuankai Qi, Hongyang Yu, Yifan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian. The unmanned aerial vehicle benchmark: Object detection and tracking. In Proceedings of the European conference on computer vision (ECCV) , pages 370-386, 2018. 2, 3

Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 549-565. Springer, 2016. 3

Tung Minh Tran, Tu N Vu, Tam V Nguyen, and Khang Nguyen. Uit-adrone: A novel drone dataset for traffic anomaly detection. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2023. 3

Lichao Mou, Yuansheng Hua, Pu Jin, and Xiao Xiang Zhu. Era: A data set and deep learning benchmark for event recognition in aerial videos [software and data sets]. IEEE Geoscience and Remote Sensing Magazine , 8(4):125-133, 2020. 3, 4

Murari Mandal, Lav Kush Kumar, and Santosh Kumar Vipparthi. Mor-uav: A benchmark dataset and baselines for moving object recognition in uav videos. In Proceedings of the 28th ACM international conference on multimedia , pages 2626-2635, 2020. 3

Ilker Bozcan and Erdal Kayacan. Au-air: A multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 8504-8510. IEEE, 2020. 3

Isha Kalra, Maneet Singh, Shruti Nagpal, Richa Singh, Mayank Vatsa, and PB Sujit. Dronesurf: Benchmark dataset for drone-based face recognition. In 2019 14th IEEE International Conference on Automatic Face &Gesture Recognition (FG 2019) , pages 1-7. IEEE, 2019. 3

Margherita Bonetto, Pavel Korshunov, Giovanni Ramponi, and Touradj Ebrahimi. Privacy in mini-drone based video surveillance. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG) , volume 4, pages 1-6. IEEE, 2015. 3

Slim Hamdi, Samir Bouindour, Hichem Snoussi, Tian Wang, and Mohamed Abid. End-to-end deep one-class learning for anomaly detection in uav video stream. Journal of Imaging , 7(5):90, 2021. 3

Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing , 165:108-119, 2020. 2, 3

Jingkang Yang, Jun Cen, Wenxuan Peng, Shuai Liu, Fangzhou Hong, Xiangtai Li, Kaiyang Zhou, Qifeng Chen, and Ziwei Liu. 4d panoptic scene graph generation. Advances in Neural Information Processing Systems , 36, 2024. 2

Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1948619496, 2022. 3

Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for an end-to-end scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024. 3

Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024. 3, 6

Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6292-6301, 2023. 3

Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language model for weakly supervised scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024. 3

Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. arXiv preprint arXiv:2312.00093 , 2023. 3

Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu. Target adaptive context aggregation for video scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13688-13697, 2021. 3

Siddhesh Khandelwal and Leonid Sigal. Iterative scene graph generation. Advances in Neural Information Processing Systems , 35:24295-24308, 2022. 3, 4

Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy-Chowdhury. Unbiased scene graph generation in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22803-22813, 2023. 3, 4

Trong-Thuan Nguyen and Minh-Triet Tran. A transformer-based approach for dynamic referee assistance. In 2023 International Conference on Multimedia Analysis and Pattern Recognition (MAPR) , pages 1-6. IEEE, 2023. 3

Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23090-23099, 2023. 3

Chaolei Tan, Zihang Lin, Jian-Fang Hu, Wei-Shi Zheng, and Jianhuang Lai. Hierarchical semantic correspondence networks for video paragraph grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18973-18982, 2023. 3

Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng. Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23100-23109, 2023. 3

Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi. Video relationship reasoning using gated spatio-temporal energy graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10424-10433, 2019. 3

Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao. Video relation detection with spatio-temporal graph. In Proceedings of the 27th ACM international conference on multimedia , pages 84-93, 2019. 3

Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM international conference on Multimedia , pages 3654-3663, 2021. 3, 4

Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Luu. Type-to-track: Retrieve any object via promptbased tracking. Advances in Neural Information Processing Systems , 36, 2024. 3

Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Son Lam Phung, Ngan Le, and Khoa Luu. Multi-camera multi-object tracking on the move via single-stage global association approach. Pattern Recognition , 152:110457, 2024. 3

Kha Gia Quach, Pha Nguyen, Huu Le, Thanh-Dat Truong, Chi Nhan Duong, Minh-Triet Tran, and Khoa Luu. Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13784-13793, 2021. 3

Jingwei Ji, Ranjay Krishna, Fei-Fei Li, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR , 2020. 3

Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-and-anticipation transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13824-13835, 2023. 4

Shuqiang Cao, Weixin Luo, Bairui Wang, Wei Zhang, and Lin Ma. A circular window-based cascade transformer for online action detection. arXiv preprint arXiv:2208.14209 , 2022. 4

Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 , 2023. 5, 17

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020. 6

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 7

Thanh-Dat Truong, Quoc-Huy Bui, Chi Nhan Duong, Han-Seok Seo, Son Lam Phung, Xin Li, and Khoa Luu. Direcformer: A directed attention in transformer approach to robust action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20030-20040, 2022. 7

Cewu Lu, Ranjay Krishna, Michael Bernstein, and Fei-Fei Li. Visual relationship detection with language priors. In ECCV , 2016. 8

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 18

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-ajudge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. 18

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 18

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017. 22

Appendices

A The AeroEye Dataset

A.1 Relationship Definition

Our AeroEye dataset focuses on capturing the spatial positions and relationships between objects, including person-to-object and object-to-object interactions. Table A.8, A.9 provides a comprehensive list of the defined positions and relationships. Furthermore, Table A.10 offers guidance on the final relationship vocabulary specific to each scene category within the dataset.

A.2 Annotation Pipeline

To capture frequent and rapid changes in aerial videos while reducing redundancy, we annotate keyframes at 5FPS. At each frame, our annotation pipeline consists of two stages:

Stage 1: Object Localization and Tracking. We manually annotate bounding boxes with predefined categories, providing precise object localization and consistent tracking throughout the video.

Stage 2: Relationship Instance Annotation. To generate diverse predicates, we leverage the GPT4RoI [70] model, which combines visual and linguistic data to generate detailed descriptions of object relationships within specified regions of interest. The process involves the following steps:

Text Generation: We leverage GPT4RoI integrateed instruction tuning with a large language model (LLM) to enhance interactions with regions of interest (RoI) within images. This model transforms bounding box references into language instructions, enabling detailed descriptions and reasoning about specific image regions, thus improving image understanding granularity and accuracy. It utilizes a variety of transformed multimodal datasets, including COCO [75] and Visual Genome [23], to refine the alignment between visual and linguistic data, ensuring precise responses to spatial queries.

In particular, We input bounding boxes around objects with prompts such as, "What is the relationship between <object_1> in <region_1> and <object_2> in <region_2> ?", where <object_ i > is a category name that labeled in Stage 1, and GPT4RoI replaces <region _ i > tags in these instructions with results from RoIAlign, derived directly from the features of image. The model uses RoIAlign to extract region-specific features and combine them with language embeddings. The resulting multimodal embeddings are then interpreted by the Vicuna model [76], an instance of LLaMA [77].

Predicate Summarization and Selection: We employ a custom-designed filter to categorize the generated text into relationship types. This filter utilizes a combination of keyword matching, dependency parsing, and semantic analysis to identify and classify the predicates accurately. The filter is designed to handle sentence structure and terminology variations, ensuring that the identified predicates are correctly mapped to their corresponding relationship types. Furthermore, the filter incorporates a confidence scoring mechanism to prioritize high-quality predicates and filter out irrelevant or ambiguous ones. The final selection of predicates undergoes human oversight, where experienced annotators review and validate the filtered results. This manual verification step ensures the highest accuracy and relevance of the identified predicates, mitigating potential errors introduced by automated processing.

Quality Control To maintain the highest standard of annotation quality, we implement the following comprehensive measures:

Stage 1: Object Localization and Tracking:

-All bounding box annotations are performed manually by skilled annotators without relying on automated detection or tracking models. This ensures precise object localization tailored to the specific characteristics of aerial videos.

-We employ a rigorous double-checking process, where each frame in every video is carefully reviewed by a second annotator. This step helps identify and rectify any inaccuracies in bounding box placement or dimensions.

-In cases where object identities are inconsistent across frames due to occlusion, visual similarity, or other challenges, annotators meticulously correct the object numbers to maintain consistent tracking throughout the video.

Stage 2: Relationship Instance Annotation:

-Annotators undergo extensive training using carefully curated examples from previous VidSGG datasets. This training familiarizes them with the intricacies of extracting predicates generated by the LLM and ensures a deep understanding of the annotation guidelines and best practices.

-To minimize individual biases and ensure the robustness of annotations, we implement a repeated annotation process. Each video is distributed to multiple annotators, who independently extract and record the relationship instances. This redundancy allows for cross-validation and helps identify potential discrepancies or ambiguities.

-In cases where annotators disagree on the extracted predicates or their categorization, a highly experienced meta-annotator is assigned to review the conflicting annotations. The meta-annotator carefully examines the video content, considers the perspectives of the individual annotators, and makes the final decision on the annotation record. This hierarchical review process ensures consistency and accuracy across the dataset.

By employing these rigorous quality control measures at each stage of the annotation pipeline, we ensure the highest level of accuracy , consistency , and completeness in relationship instances.

A.3 Data Format

Our annotations are stored in JSON (JavaScript Object Notation) format organized as below:

Basic Image Information. This section contains the fundamental attributes of each image:

file_name ( str ): The name of the image file.

height ( int ): The height of the image in pixels.

width ( int ): The width of the image in pixels.

image_id ( int ): A unique identifier for the image.

frame_index ( int ): The index of the frame within the video sequence.

video_id ( int ): An identifier for the video to which this image/frame belongs.

Metadata. This section includes the metadata key, which is a list of segments within the image. Each segment contains:

id ( int ): Unique identifier for the segment.

category_id (int): Identifier for the category of the object in the segment.

iscrowd ( 0 or 1 ): 0 for a single object and 1 for a cluster of objects.

area ( int ): The area covered by the segment in the image.

The annotations key contains a list of corresponding bounding boxes for each entry in metadata , each tagged with a specific category_id :

bbox ( list ): [x_center, y_center, width, height] of the bounding box.

bbox_mode ( 0 or 1 ): Bounding box mode.

category_id ( int ): Identifier for the object category in the bounding box.

track_id ( int ): Identifier to track the bounding box across different frames.

Relationship Attributes. This section encompasses lists of positions and relations for each segment, including two different metadata_id s to represent the interactivity between two segments:

positions ( list ): List of position relations between segments, each containing:

-metadata_id ( int ): Identifier for the first segment.

-metadata_id ( int ): Identifier for the second segment.

-position_id ( str ): Identifier for position relation between the segments.

relations ( list ): List of other relations between segments, each containing:

-metadata_id ( int ): Identifier for the first segment.

-metadata_id ( int ): Identifier for the second segment.

-relation_id ( str ): Identifier for relationship between the segments.

These descriptors represent lists specifying various subject, object, and interactivity aspects for each bounding box within the annotations and metadata . For example, [3, 0, 5] indicates that the third and first metadata segments share the relationship with an ID of 5.

Number of Objects(logscale)

5

4

2

0

car

person

streetlight

traffic light

tractor

boat

flag

van

basketballhoop

truck

ball

snq

tree

house

paddle

bicycle

traffic sign

crane

screen

banner

fire

bench

bridge

balloon

motorcycle

fence

airplane

animal

umbrella

trumpet

motorbike

tram

building

lake

scooter

ladder

pool

ship

other

bird

statue

horse

wind turbine

scoreboard

storage tank

score board

baby stroller

church

dam

train

castle

tent

waterfall

tank

Number of Instances (log scale) per Position Predicate

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

saxoq <ljeuad punoue

around ritual objects

indifferent

(bue"

ISUI

aAoqe

更

aso

centerec

E

around refreshment tabl

outside

uMop

in lane

0%

apun

o peat

A.4 Additional Statistics

We present object, position, and relationship statistics per category in Fig. A.8, A.9, and A.10.

A.5 Data Samples

Fig. A.11 presents selected samples from our AeroEye dataset, distinguished by its detailed bounding box annotations and meticulous relationship descriptions across various scenarios. As outlined in Section 3.2, each frame within AeroEye is annotated with precision and contextual relevance, ensuring clarity and avoiding the common ambiguities, such as generic or overlapping labels, prevalent in other datasets. A key feature of AeroEye is its categorization of relationships into positions and relations, as illustrated by the curved arrows representing positions and straight arrows denoting relations in Fig. A.11. This multifaceted approach to annotation renders AeroEye uniquely comprehensive when compared to other datasets [6, 27, 7, 8]. The meticulous annotation process and the meticulously curated, information-rich nature of the AeroEye dataset firmly establish it as an invaluable resource, poised to catalyze significant advancements in relationship modeling within drone videos.

B Concepts

B.1 Progression

Relationship Representation. In each frame, a detector provides to a set of object features { ̂ v 1 t , . . . , ̂ v N ( t ) t } , bounding boxes { ̂ b 1 t , . . . , ̂ b N ( t ) t } , and category distributions { ̂ d 1 t , . . . , ̂ d N ( t ) t } for

Number of Instances(log scale)per Relation Predicate

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

cooperating

6uIAou

traversing

insideconstructionzone

ied

jeado leuasnpui Subeueu

loed

2

friends dancing toge

u ueua

s

residents evacuating tog

enjoying concert atmo

9

too

E

helpi

beoqun

ubuubie

lane

serving food

8

qeu

puessauauemeSuisie

05

eoa

ieus

je uequn budwenau

0%

movinginfrontof

person

K

towarding

"firefighters.

car

front

moving in the left lane?

person

car

moving to the right

hoop

-movingbehind

firefighters-

person

in front of.

next to

next to

players

car

behind

—-moving to the left

the detected objects. These elements are used to define the relationships ̂ x k t :

where [ · , · ] denotes concatenation, φ denotes a flattening operation, ⊙ represents element-wise addition, and the linear transformation matrices ϕ W s , ϕ W o , and ϕ W u . ̂ u t ij implies the feature map of the union box, computed by the RoIAlign [78], while f is a function that converts the bounding boxes of the subject ̂ b t i and object ̂ b t j into a feature map with the same dimensions as ̂ u t ij . The semantic embedding vectors ̂ s t i and ̂ s t j are obtained from the object categories ̂ c t i and ̂ c t j , respectively.

In the progressive approach, features v i t for each object are obtained using a detector such as Faster RCNN in each frame. These features are utilized as specified in Eqn. (B.7) to compute the relationship features ̂ x k t . Finally, these features are processed through multilayer perceptions (MLPs) followed by softmax activation functions to classify the types of relationships between objects.

B.2 Batch Progression

In contrast to the Progressive approach, the Batch Progressive approach passes the relationship features through a Transformer architecture before classification.

A set of relationship features ̂ X t = { ̂ x 1 t , ̂ x 2 t , . . . , ̂ x K ( t ) t } is fed into the Transformer Encoder , which focuses on understanding the spatial context by inputting these relationship features into a sequence of identical self-attention layers, defined as:

Each n -th layer receives the output from the ( n -1) -th layer as its input, iteratively refining to enhance the representation of spatial relations embedded in the features, where n denotes the layer number. The outputs are then processed by the Decoder , which captures temporal dependencies between frames, applying a sliding window over the sequence of spatially contextualized representations:

where η is the window size, and T is number of frames. The positional encoding E f = [ e 1 , . . . , e η ] is embedded into the input to maintain sequence order:

Equation (B.10) enables the decoder to process each batch using self-attention layers, combining relation representations ̂ Z i with positional encodings E f .

B.3 Hierarchical Graph

The hierarchical interlacement graph (HIG) method abstracts video content by representing temporal relationships using a multi-level graph structure. Higher-level graph cells encompass broader segments of video frames, allowing the HIG to efficiently capture and model the temporal dependencies and connections across different time scales within the video. In this method, object features ̂ v i t from individual frames are spatially and temporally fused to form graph nodes. These nodes are interconnected across successive frames, resulting in a series of interconnected frame-based graphs { G t ( V t , E t ) } T t =1 , where each G t is defined by its vertices V t and edges E t . As the graph traverses, the total number of graphs decreases, ultimately resulting in a singular graph representing the entire video. This hierarchical graph operates on predefined hierarchical levels L . At each level l , the temporal scope is adjusted to T l = T -l +1 , and a new graph is generated that is specific to that level and timeframe. Therefore, node features within each graph are dynamically updated through the computation and aggregation of messages from adjacent nodes, computed by weight matrices for each level l and node pair ( ̂ v i , ̂ v j ) . This iterative refinement process is applied across all levels, resulting in a consolidated graph structure and updated feature set. Finally, relationship features between each node pair ( ̂ v i , ̂ v j ) are fused and analyzed to classify the relationships.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Contributions in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Limitations in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: While this paper does not include purely theoretical results, it does incorporate derivations of relevant formulas, along with their respective assumptions and complete proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see the Subsection 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our dataset is being distributed at https://uark-cviu.github.io/projects/CYCLO/. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see Subsection 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Subsection 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see Broader Impacts in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original papers that produced the code or dataset are cited properly, including authors, title, publication venue, year, and a link if available.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper introduces a new dataset, which is discussed in Section 3 and Appendix A.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the VidSGG task. Specifically, it built a new UAV-based VidSGG dataset named AeroEye, and also further propose a CYCLO approach for using cyclic attention over the VidSGG task.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n\n2. From my perspective, a high quality dataset and large scale dataset is of critical imporantance over the VidSGG area. I thus appreciate the effort by the authors.\n\nWEAKNESSES:\n(See the questions section below)\n\nQUESTIONS:\nOverall, while I tend to accept this work, I still have the following concerns and suggestions w.r.t. the current version of the submission:\n\n(1) It seems to me that, while the dataset focuses on the UAV scenario, the direct and long-range temporal relationships focused by the proposed method is not closely related to the UAV scenario. I appreciate if the connection (if any) can be more clearly explained.\n\n(2) Besides comparing with those existing UAV datasets, I suggest the authors to also include a statistical discussion (in a table format maybe) between AeroEye and those existing VidSGG datasets.\n\n(3) In line 43, the authors claim that \"They [5, 19] usually struggle with long-term dependencies due to the diminishing influence of inputs over time.\" This claim should be made with better support.\n\n(4) Moreover, I believe that the connectivity between the proposed CYCLO method and scene graph needs to be better discussed. This is important since the usage of \"cyclic attention\" solely in the video context seems to be not a new stuff already (e.g., Transformer Tracking with Cyclic Shifting Window Attention CVPR 2022). Thus, for authors to clearly indicate the novelty of their CYCLO method, from my perspective, it is necessary to (a) discuss how CYCLO is designed for scene graph more clearly and (b) discuss the difference between CYCLO and existing cyclic attention methods in the video context.\n\nLIMITATIONS:\nThe authors have discussed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-15"}

comments: SUMMARY:
This paper tackles an interesting task for understanding video scenes that focuses on modeling object relationships in aerial videos. Specifically, it introduces a new dataset, the AeroEye dataset, and proposes a novel approach, CYCLO, to better model the video object relationships. Experimental results demonstrate the effectiveness of CYCLO. CYCLO also achieves state-of-the-art performance on two scene graph generation benchmarks.

STRENGTHS:
- The paper is well-written, easy to follow, and presents many key points clearly.
- Introducing the AeroEye dataset is a valuable contribution, as it fills an important gap in video scene graph generation datasets by providing a drone perspective relation dataset.
- The design of CYCLO is both interesting and inspiring. The paper also demonstrates its superior performance compared to prior solutions.

WEAKNESSES:
- It would be great to report the inference cost of the proposed approach. 
- It is unclear how the proposed solution works in live (online) mode, e.g., video streaming.

QUESTIONS:
- In the ablation study (line 265), when a frame is discarded, is it something like that the frame is set to some random noise?

LIMITATIONS:
I don't have concerns about the potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new problem: modeling multi-object relationships from a drone's perspective. To address this, the authors propose the AeroEye dataset and introduce the Cyclic Graph Transformer (CYCLO) method. This method captures both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The authors not only validate the CYCLO approach on the AeroEye dataset but also test it on the PVSG and ASPIRe datasets, demonstrating the effectiveness of their method.\n\nSTRENGTHS:\n1. The authors have introduced the problem of multi-object relationship modeling from a drone's perspective for the first time and constructed the AeroEye dataset, which effectively fills a gap in the field of multi-object relationship modeling and has significant application value.\n2. The CYCLO method proposed by the authors is not only useful for the dataset introduced in this paper, AeroEye, but is also a versatile method that can be applied to general Video SGG tasks. It has been tested on datasets like PVSG and achieved good performance.\n3. The structure of the paper is clear, the writing is standard and fluent, making it easy to understand.\n\nWEAKNESSES:\n1. The method proposed in this paper could benefit from a clearer network architecture figure, which would allow everyone to better understand the method presented.\n2. Is there an issue with non-differentiability in the Cyclic Attention described in Eq. 3? It would be helpful if the authors could provide further explanation of the Cyclic Attention mechanism.\n3. It is suggested to include some analysis of bad cases, which would help future researchers understand from which directions further optimizations can be made.\n\nQUESTIONS:\n1. The current model proposed by the authors shows low mR@K scores on the AeroEye dataset. However, from the figures in the supplementary material, the long-tail distribution of relations in this dataset does not appear to be very severe. What could be causing the low mR@K scores? If possible, I would like to see some typical bad cases.\n\nLIMITATIONS:\nIt is recommended that the authors consider the limitations of this paper not only from a technical perspective but also from a societal standpoint. Given that relationship modeling from a drone's perspective may lead to widespread applications in surveillance and potentially significant impacts, this aspect requires special consideration.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a video scene graph generation dataset called AeroEye on aerial videos and a framework called cyclic graph transformer to tackle the problem of video scene graph generation. The authors annotated the ERA and MAVREC dataset with keyframes at 5FPS. They manually annotated the frames for bounding box localization for each frames along with the tracking. The relationship annotations were done using a GPT4RoI model. The proposed cyclic graph transformer uses a cyclic attention mechanism which the author claims, are able to capture the direct and long-term temporal dependencies.\n\nSTRENGTHS:\n1. The proposed video scene graph dataset for aerial videos with a diverse set of predicates can offer more granular and nuanced understanding of dynamic interactions and relationships within aerial footage\n2. The dataset will be publicly available\n3. The proposed approach for circular attention for dynamic online scene graph generation seems promising\n\nWEAKNESSES:\n1. the paper is very difficult to follow. There are separate discussion sections which somewhat disrupts the flow.\n2. line 82 seems incomplete\n3.Line 118: 'no temporal edge is treated a boundary' can it not be a disadvantage as well since it does not takes into account for an event boundary?\n4. A very brief discussion of annotation procedure should be in the main paper such as manual annotation of bounding boxes and relationship annotations done by GPT4RoI model.\n5. For loss function section, it would be better to include  the overall loss equation. Does the object distribution refers to the object detection loss using the DETR?\n6. For table 2, you can refer to the shift value term from equation 3. It will be easy to follow. Table 2 refers to the ablation studies 'Semantic Dynamics in Cyclic Attention'. I think this section and the section followed by it should have a detailed explanation.\n\nQUESTIONS:\n1. Did you do evaluation on the 5 keyframes for each videos?\n2. point 5 in weakness section\n\nLIMITATIONS:\nyes, the limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}