CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos
Authors: Trong-Thuan Nguyen
Pha Nguyen
Xin Li
Jackson Cothren
Alper Yilmaz
Khoa Luu
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph Transformer
Multi-Object Relationship Modeling
Aerial Videos
Primary area: machine_vision

Abstract: Video scene graph generation (VidSGG) has emerged as a transformative approach to capturing and interpreting the intricate relationships among objects and their temporal dynamics in video sequences. In this paper, we introduce the new AeroEye dataset that focuses on multi-object relationship modeling in aerial videos. Our AeroEye dataset features various drone scenes and includes a visually comprehensive and precise collection of predicates that capture the intricate relationships and spatial arrangements among objects. To this end, we propose the novel Cyclic Graph Transformer (CYCLO) approach that allows the model to capture both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The proposed approach also allows one to handle sequences with inherent cyclical patterns and process object relationships in the correct sequential order. Therefore, it can effectively capture periodic and overlapping relationships while minimizing information loss. The extensive experiments on the AeroEye dataset demonstrate the effectiveness of the proposed CYCLO model, demonstrating its potential to perform scene understanding on drone videos. Finally, the CYCLO method consistently achieves State-of-the-Art (SOTA) results on two in-the-wild scene graph generation benchmarks, i.e., PVSG and ASPIRe.

CYCLO Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos
Nguyen', Pha Nguyen', Xin Li2, Jackson Cothren
Alper Yilmaz', Khoa Luu'
'University of Arkansas "State University of New York at Albany
'Ohio State University
[[thuann, panguyen, jeothre, thoaluu]@uarkkeen edu "x1i480albany.. edu "yilmaz. 150osu. edu
mark-cviu, bark-cviu.github.ioo.
/
Figure 1: Multi- Object Relationship Modeling in Aerial videos analyzes a drone- captured video to detect and refine object relationships time. The CYCLO model first identifies relationships between objects in individual frames and then incorporates temporal information about object positions and interactions to refine the understanding of those relationships across the video sequence. (Best viewed in colors)
Abstract
Video scene graph generation (VidSGG) has emerged as transformative approach to capturing and interpreting the intricate relationships among objects and their temporal dynamics in video sequences. In this paper, introduce the new Aero- Eye dataset that focuses on multi-object relationship modeling in aerial videos. Our AeroEye dataset features various drone scenes and includes visually compre- hensive and precise collection of predicates that capture the intricate relationships and spatial arrangements among objects. To this end, propose the novel Cyclic Graph Transformer (CYCLO) approach that allows the model to capture both direct and long-range temporal dependencies by continuously updating the history of interactions in circular manner. The proposed approach also allows one to handle sequences with inherent cyclical patterns and process object relationships in the correct sequential order. Therefore, it can effectively capture periodic and overlap- ping relationships while minimizing information loss. The extensive experiments on the AeroEye dataset demonstrate the effectiveness of the proposed CYCLO model, demonstrating its potential to perform scene understanding on drone videos. Finally, the CYCLO method consistently achieves State-of-the-Arrt (SOTA) results on two in-the-wild scene graph generation benchmarks, i.e., PVSG and ASPIRe.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Visual scene understanding has shown significant progress in extracting semantic information from images and videos using deep learning algorithms [1, 3]. Building upon the significant progress in visual scene understanding using deep learning algorithms [4, 5], video scene graph generation (VidSGG) extends the concept of Scene Graph Generation (SGG) from static images to dynamic video, representing object relationships within graph structure that evolves over time. VidSGG [6. 7, 8] focus on the temporal dimension by constructing dynamic graph structure, encapsulating the spatial and temporal relationships among object interactions across frames. This helps in understanding human-object interactions [9, 10], temporal events [1 [11, 12, 13], and reasoning [14, 15]. However, drone-captured videos present unique challenges due to larger image sizes and higher object density in Unmanned Aerial Vehicle (UAV) datasets [16, 17, 18]. Despite recent advance in tiny object detection [19, 20, 21], current algorithms still need to effectively model object interactions and their temporal evolution in aerial videos, which have various applications in surveillance, disaster response.
In this paper, we introduce AeroEye, the first dataset for Video Scene Graph Generation in drone- captured videos featuring Aerial-Oblique-ER Eye views. AeroEye distinguishes itself by showcasing a rich tapestry of aerial videos and an extensive set of predicates describing the intricate relations and positions of multi-objects. To address multi-object relationship modeling in aerial videos from the Ayr dataset, we propose the Cyclic Graph Transformer (CYCLO). This new approach can establish circular connectivity among frames and enables the model to capture direct and long-range -range temporal relationships. By continuously updating history across a ring topology, CYCLO allows the model to handle sequences with inherent cyclic patterns, facilitating the processing of object relationships in the correct temporal order. Furthermore, CYCLO provides several advantages to VidSGG, including the ability to model periodic and overlapping relationships, predict object interactions by reasoning from previous cycles, facilitate information transfer across frames, and efficiently utilize long sequences, addressing the limitations of prior methods [22, 8]. They usually struggle with long-term dependencies due to the diminishing influence of inputs over time.
Contributions of this Work. There are three main contributions to this work. First, we introduce a new AeroEye dataset for VidSGG in drone videos, augmented with numerous predicates and diverse scenes to capture the complex relationships in aerial videos. Second, we propose the CYCLO approach, utilizing circular connectivity among frames to enable periodic and overlapping relationships. It allows the model to capture long-range dependencies and process object interactions in the appropriate temporal arrangement. Finally, the proposed CYCLO approach outperforms prior methods on two large-scale in-the-wild VidSGG datasets, including PVSG [7] and ASPIRe [8]. Interestingly, using the same method (e-g., our CYCLO), the ratio of correct predictions to incorrect predictions (R/mR) on AeroBye is higher than PVSG (Tables 4 and 6), despite having more predicates (Table 1) and tiny objects. This suggests that our dataset is less visually ambiguous than PVSG.
2 Related Work
In this section, we review the existing datasets and benchmarks for Visual Scene Graph Generation, followed by summary of the key challenges and issues related to Video Scene Graph Generation.
2.1 Visual Scene Graph Generation Datasets and Benchmarks
Datasets. VisDrone [17], DOTA [16], and SODA-A [36] image datasets, along with UAVid [46], UAVDT [37], and MAVREC [18] video datasets, offer high-resolution UAV datasets that enable precise object detection in dynamic scenes. While these UAV datasets focus on object detection, the
Visual Genome [23] pioneered image-based SGG, and the Action Genome [6] dataset extended this concept to capture dynamic interactions within videos. Recently, ASPIRe [8] and SportsHHI [29] emphasize diverse human-object relationships and sports-specific player interactions. Additionally, PSG-4D [47] expands the VidSGG to encompass the 4D domain, bridging the gap between raw visual data and hegh- understanding. In Table 1, we present a comparative overview of UAV-based and SGG datasets for images and videos, emphasizing their unique characteristics and advantages.
Benchmarks. The existing benchmark focuses on Image Scene Graph Generation (IngSGG) and Video Scene Graph Generation (VidSGG). ImgSGG identifies and categorizes relationships between objects within an image into predefined relational categories, including Transformer-based

---

Table I: Comparison of available datasets for scene graph generation. The top two blocks present image and video scene graph datasets, while the two focus image and drone video datasets. Viewpoints include five types: from ego (Ist-person) view to 3rd-person view, and drone- cooneccapture perspectives, which our main tocus in this work (VIX in colors), including aerial (top-down), oblique (slanted), and ground (eye-level)
perspectives. # denotes the number of corresponding items. The best values in drone blocks are highlighted
Datasets
Wideos #Frames
Resolation
Amnotations
#ObjCIs #ReiChs #Scenes
Viewpoints
BBox Relation #Iustances
3rd ego serial oblique ground
Cenome [23]
38M
VG-150 [24]
88K
2.8M
150
50
x
VeR-VG [25]
59K
282.4K
1.6K
117
/
x
GQA [26]
85K
1.7K
310
/
PSG [27]
VidVRD[22)
296.2K
1920
1080
/
15.1K
35
132
/
X
Action Genome [6]
10K
234.3K
1250 x 720
/
476.3K
25
25
X
ASPIRe [S]
1.5K
16M
1280
720
<
167.8K
833
4.5K
/
SportsHH1 (29)
VidOR [30]
10K
55.4K
360
SUK
80
VIASTGG [30]
10K
55.4K
(40
360
S0K
80
50
/
X
EPIC-KITCHENS [32]
700
11.5K
1920
1080
V
454,3K
[7]
1920
DOTA [16]
11.3K
1490 x 957
/
18M
18
X
X
DIOR-R [34]
23.5K
X00
192.5K
20
MONET [35]
S3K
162K
X
X
SODA-A [36
2.5K
47761
2777
/
S72.1K
VisDrone [17]
288
261.9K
3840
2160
2.6M
10
UAYIT [37]
40.7K
540
0.84M
Stanford Drone [38]
10K
929.5K
5
UTT-ADrone [39]
51
206 2K
1920 1080
/
210.5K
8
ERA [40]
2,9K
343.7K
640
25
MOR-UAV
AU-AIR [-2]]
32.8K
1920
1080
132K
8
DroneSURF [433]
200
411.5K
1280 x 720
V
786K
1
X
X
MiniDrone [44]
23.3K
224 224
Rumning [45]
UAVid [46]
4096 2160
8
X
MAVREC [18]
537K
3840
2160
1.IM
10
X
X
AeroEye (Ours)
2.3K
261.5K
3840
2160
18M
57
384
29
X
methods [27, 48, 49, 50] and generative- based models [51, 52, 53]. VidSGG leverages the dynamic nature of object interactions over time to better identify relationships, as the temporal dimension of videos provides richer context for understanding semantic connections within the scene. Current methods using hierarchical structures [54, 8] or Transformer architectures [22, 55, 56, 57] excel at capturing long-range dependencies and complex interactions, advancing video understanding in video captioningl 11, 12, 13], visual question answering [14, 15], and video grounding [58, 59, 60].
2.2 Video Scene Graph Generation
VidSGG can be categorized into two main types based on the granularity of its graph representation. Video-level SGG represents object trajectories as graph nodes, capturing constant relations between objects for video. Various methods have been proposed to address this problem, incorporating Con- ditional Random Fields [61], abstracting videos [62]. and iterative relation inference techniques [63] on fully connected spatio-temporal. However, focusing primarily on recognizing video-level relations directly based on object-tracking [64, 65, 66] results and neglecting frame-level, scene graphs results in cumbersome pipeline highly dependent on tracking accuracy. In contrast, Frame-level SGG defines the graph at the frame level, allowing relations to change over time. The releases of the benchmark datasets [67, 7, 8] have prompted the development of VidSGG models. TRACE [54], for instance, employs a hierarchical relation tree to capture spatio-temporal context information, while CSTTran [22] uses a spatio-temporal transformer to solve the problem. Recently, hierarchical interlacement graph (HIG) [8] abstracts relationship evolution using sequence of hierarchical graphs.
2.3 Discussions
In this subsection, we conceptually compare our proposed approach with relationship modeling concepts discussed in Section 2.2 as illustrated in Fig. 2. In addition, we highlight the advantages of approach and discuss the properties that distinguish it from these existing methods.
3

---

Concepts. The progressive approach fuses pairwise features between object pairs at each frame, encoding the object relationship at that specific step. followed by fully connected layer classify the predicate types. However, it processes frames independently without considering the temporal context. The batch-progressive approach employs a transformer block with positional embeddings on the fused query features. The hierarchical approach represents video sequence of graphs, integrating temporal and spatial information different levels. The node and edge features are updated each hierarchical level based on the previous level to capture evolving object relationships.
Limitations. While the batch-proo prooresvogeeesivve approach considers temporal information, both these progressive and batch-progressivy approaches have limitations in modeling the full complexity of temporal dynamics and dependencies in the video. In contrast, the hierarchical graph approach can capture complex interactions and relationships between objects by considering the temporal evolution of graphs different granularity levels. However, the hierarchical graph requires analyzing the entire video before constructing the graph (i.e. offline method). These limitations underscore the need for more advanced approaches to efficiently model temporal dynamics, adaptively update memory to handle evolving video data, and accurately capture the intricate relationships between objects.
Advantages of Our Design. Inspired by previous work [68, 69], which processes temporal features through iterative feedback loops and circular updating, we propose the CYCLO approach that cireularly incorporates an updated history of relationships. In contrast to these methods, which focus on frame-level updates influenced by global features, our approach constructs and refines scene graphs for each frame, capturing static spatial relationships between objects and their dynamic evolution over time. By leveraging circular connectivity, CYCLO establishes a continuous loop of temporal information, ensuring no temporal edge is treated as boundary. enables operate online and then capture and update relationships between objects more effectively, correcting erroneous connections. The thcoretical properties are included in Section of Appendices.
Transformer to
The Proposed AeroEye Dataset
In this section, we detail the AeroEye dataset annotation process and provide the dataset statistics.
3.1 Dataset Collection
Data Preparation. We leverage videos from the ERA [40] and MAVREC [18] datasets to construct our AeroEye dataset. ERA consists of diverse scenes ranging from rural to urban environments in extreme conditions (e.g. earthquake, flood, fire, mudslide), daily activities (e.g. harvesting, plowing, party, traffic collision), and sports activities (e.g. soccer, basketball, baseball, running, swimming). MAVREC features sparse and dense object distributions and contains typical outdoor activities characterized by many vehicle classes, incorporating viewpoint changes and varying illumination.
Progressive
Polnwise Fusion
MLP
prod
pred
Hierarchical
Batch- Prograssee
MIP
Transformar Encoder
Postzona
approach
MLP
Figure 2: Comparisons of CYCLO and existing relationship modeling: (a) Progression [27, 63]: frame- wise fusion and classification; (b) Butch-progression [22, 55, 56]: temporal transformer: (c) Hierarchy [8]: spatiotemporal graph; (d) Our CYCLO approach: circular connectivity for capturing temporal dependencies.

---

opposte
opponants
fron
righ
opponents
tearrnatee
opponients
opposite
opposho
opposit
opposiN
behind
Basketball scene from the ERA dataset
b) scene graph depicting our annotations relationships between objects.
Figure 3: Example annotation in dataset. In Fig. 3b, straight arrows denote relationships between objects. while curved arrows indicate positions of the objects. Nodes of same color represent the same object, and the labels on the edges specify the predicate cach relationship. (Best viewed colors)
Relationship Classes and Instance Formulation. In Fig. 3, we focus on two aspects of object relationships: positions (e.g. in front of, behind, next to) and relations. which consist of movement actions (e.g. chasing, towing, overtaking) and collision actions (e.g. hitting, crashing, colliding). These relationships are semantically complex and require detailed spatio-temporal context reasoning for recognition. Following previous PVSG benchmark datasets [6, 7, 8], we define relationship instances the frame level, considering the long-term spatial-temporal context. Each instance is formulated as triplet <s, o, p>>, where and denote the bounding boxes of the subject and object, and p represents the predicate (i.e. position and relation), included in Tables A.8, A.9 which are summarized in Fig. 4. In addition, Fig. A.11 presents selected samples from our AeroEye dataset.
3.2 Data Specification
Data Annotation. We annotate keyframes at SFPS to capture frequent and rapid changes in positions and relations in aerial videos, re-
DEFOSS Sconic
agricultural
AITEM
infrgseructure
S
traffic
str eet leading
Lusks
along
Building
g
Feeestee scenie route naviga
gingiingg
ducing redundancy while keeping up with in- teraction changes. Our two-stage annotation pipeline first performs object localization and tracking and then relationship instance annota- lever-
JOO
FlOER
Enteree
route of
ferted
Buttderr
e
O
g
ceppon
teinch
g tein salee Jon
gane
managing racing
ging bnttie
particiiatinn
tion. To generate diverse predicates, age the GPT4Rol [70] model, which combines visual and linguistic data to generate detailed descriptions of object relationships within specified regions of interest. Although we annotate rela- tionship instances frame by frame, illustrated in Fig. 3, we easily create relationship tubes using the provided object tracking ID by connecting the same pair of objects with the same relationship predicate across consecutive frames. The annotation file includes object information (i.e., bounding boxes, category names, and tracking IDs) and relationships within each frame. Details on the quality control process and annotation examples can be found in Section A.2 of the Appendices.
urban
Figadd event
enjoyin ing
Gate sting
going ing
sur
rounded docking
Becking-mmitt Tiit
Celebrat
metertramt
Figure 4: Relationship word cloud AeroEyc dataset.
25000
Frames
Interactivities
Number lnteractvvtiess (bog
Nimber of Fromes
35000
30000
5000
Concert Constructng
Confeet
Cyclng
Modalide
Landslde
Flood
Swimming
Hanestatg
Basketball
Basting
Russrgg
Police Chatse
Relginn Actwvty
Party
Traffic Colferan
Traffic coggetoo
Fire
Car Racing
Evrtt
Ploagfarg
Parade Protest
Post Eanthouske
Buseball
Seccer
Park
Campus
Traffic Monitoreng
Harbour
Figure 5: Statistics for cach seene the AcroEye datasct.
5

---

Data Statisties. The AeroEye dataset is a collection of 2,260 videos with 261,503 frames, annotated with over 2.2 million bounding boxes across 56 object categories typically observed from aerial. oblique, and ground perspectives captured by drone. Specifically, our AeroEye dataset consists of 384 predicates divided into two relationship categories: 135 positions and 249 relations. The key strength of AeroEye is its richness in relationships. On average, each video in the dataset has 127 frames, providing moderate temporal depth for capturing detailed interactions. The average number of frames per scene is 8,970, indicating substantial variability and complexity. AeroEye is rich in relationships, with more than 43 million relationship instances. In Table I, we provide a detailed comparison with related datasets, while Fig. 5 presents statistics the AeroEye dataset. In addition, predicate definitions and additional statistics are discussed in Sections A. and A.4 of the Appendices.
4 The Proposed Approach
In this section, we present our CYCLO approach, including the Spatial Atlention Graph and the Cyclic Temporal Graph Transformer. The Spatial Attention Graph captures spatial dependencies and interactions between objects within the frame. In contrast, the Cyclic Temporal Graph Transformer models temporal relationships across frames, capturing short-term and long-term dynamics.
4.1 Problem Formulation
Given input video with T frames, construct dynamic scene graphs (Gi)T= that encode the relationships among objects within these frames. Each graph G,,, E,) captures static relationships, where node V consists of objects and edge & denotes the relationship between objects. Each object v E V has an object category of E C, and box coordinates of E [o, 1]². Each relationship ej E & represents the j-th triplet (sg. og. Py), where subject 8, and object 0g and predicate p, Cp-
4.2 Spatial Attention Graph
Self-attention mechanisms in one-stage object detectors [71, 19] model relationships between objects, allowing insights into the dynamics between entities without relying additional contextual infor- mation. For example, in an aerial parking lot video with cars, vans, and people, if the self attf-attnntion layer strongly connects the queries representing the person and the car, it suggests interaction, such as the person entering the vehicle. Inspired by [50], in our CYCLO approach, to construct the static utilize the DETR decoder to establish bidirectional connections among
graph in each frame object queries. In particular, we compute relational representations at each layer / by concatenating the query and key vectors, Q and K/ for every object pair. This process ensures the layer - output seamlessly transitions as input to layer I. We omit the superscript L related to the final layer to simplify the following discussion. At each frame, Ri, captures the dynamic interplay of relations at layer utilizing their query and key vectors. Furthermore, Rz, leverages object queries in the final layer for object detection. These relationships are formally defined in Eqn. (1).
R = Qlowy; Klowgl. R.,, [Z. 2.
(1)
Here, éw, and ow, are the linear transformations that process subject and object features, enabling the model to consider both object characteristics and their interrelationships comprehensively. In addition, gating mechanisms da,t and Is.t dynamically modulate the contributions from different layers. These gated representations from all layers are then integrated to construct relation matrix:
Da.: -RR,, De.t = =R....
(2)
Rs Σ e x Ri,e) + Ez.t x R.,
where &we denotes the linear weight applied during the gating process. Finally, the relation ma- trix is fed into the three-layer perception (MLP) with ReLU activation and a sigmoid function o, which predicts multiple relationships (p;) between pairs of objects 0z). Mathematically, G. o (ML)(ft.) is the graph representation frame 1-th, where G. N
Discussion. Transformer-basse approaches to VidSGG effectively capture interactions and temporal changes through self-attention mechanisms, creating detailed scene graphs that reflect video dynamic

---

relationships. However, these models often struggle to represent the directional and historical aspects of the relationship accurately. While effective at identifying token correlations, the scaled dot-product fails to consider their temporal or spatial ordering. This oversight is particularly critical in videos, where understanding the historical context of relationships is essential. For example, the sequence of relationships leading up to car crash, including speeding, lane changing, and passing, must be considered. Each interaction change provides crucial historical information that contextualizes the final relationships, vital for enhancing prediction and interpretation. In addition, traditional selft-attention [72] does not adequately capture this essential sequential, directional, and historical information [73]. It highlights the need for advancements in transformer architectures to more effectively integrate the direction and historical sequence of interactions within videos.
4.3 Cyelic Temporal Graph Transformer
We present the Cyelic Spatial- Graph Transformer to refine the spatial attention graph in each scene, capturing temporal dependencies via subject-object relationships across adjacent frames.
Cyclic Attention. As mentioned in Section 4.2, self-attention does not adequately capture directional and historical information. Therefore, we propose the cyclic attention (CA), defined as in Eqn. (3).
T-1
(Q((Ktt)))) mod T))
CA(Qu, K.)
ΣS <=0
(3)
dhead
In Eqn. (3), 17 is a shift term enabling cycli- cal indexing via mod T. The cyclical index- ing, illustrated in Fig. 6, allows for continuous sequence processing by connecting the end to the beginning, which is crucial for predicting movements dynamic interactions where past events influence future actions (e.g. car navi- gating a roundabout). Cyclical indexing differs from standard self-attention as it is permuta- tion equivariant without positional encodings. In contrast, cyclical attention is non- non-permutation cquivariant, which depends on the original se- quence order. This property is crucial for multi- object relationship modeling, where maintaining the chronological order of interactions is essen- tial. For instance, in a surveillance scenario, the sequence of a car stopping for a pedestrian must preserve the order of events, ensuring the vehicle stops before the pedestrian appears.
(Qg)
(K1)
Qaier (Qp)
Queriin (Q:)
Kays (K.)
(Ki)
O,K
TT
Queeries (Qg)
Keye
O.R A
Figure 6: Illustration of cyclic interactions in the Cyclic Spatial-Temporal Graph Transformer. Each frame, rep- resented by a colored block (where the first frame, and the last frame, T = 4), undergoes spatial attention to obtain queries (Q,) and keys (K.).
Temporal Graph Transformer. Our Temporal Graph Transformer refines spatial attention graphs, (T into a sequence of dynamic graphs {G. leveraging the temporal dynamics and spatial interactions of objects across video frames. Our approach employs series of cyelic attention blocks configured within multi-head attention to refine object representations by integrating features from adjacent frames. The core of our approach is the integration of cyclic attention into multi-head structure, which processes the sequence of input features 2 = (2,) represented as in Eqn. (4).
Zi - ow. ((ho(t); hs ht) = CA(dw; (2.), ow (2)), E (o.1....,e-11 1},
z' = (woo bii... :he a)),
(4)
where @w., owy. and øw: denote the linear transformations. Each head (L) computes the cyclic attention, integrating information across the video to enhance the temporal relationship at each frame. The outputs from various heads each frame are integrated into Z/. derived from concatenating all attention head outputs. These heads process features cyclically across different representation subspaces to capture the temporal evolution of relationships in the video. Then, Z is obtained by applying layer normalization (LN) and a skip connection to the aggregated features Z/, where Z = LN(Z' + 2). This step ensures that Z is stabilized and effectively integrated with the original features 2, thus dynamically updating the scene graph and ensuring temporal coherence.
In addition, Z, utilized to construct a new relation matrix R2 by applying the transformations in Eqn. (1) and (2). This updated matrix R; refines the relationship dynamics captured in static frames
7

---

Table 2: Our performance (%) on AyroEye with shift values (17 in Eqn. (3)) Recall (R) and mean Recall (mR).
Shin Value
R/ 20
R/RR @ 50
PredCis
R/mR6:100 100
R/mR6:20
R/mR@ 50
SGCls
6/mR6:100
R/mR 20
R/mR
SGDdt
R/ 100
56.20/ 19.23
61,62/
20.67
62.40/11119
54,157
16,22
59,59 18.20
60.37 18.38
43.53/ 13.29
47.93 13,69
48.94/ / 13,86
18.01
60.22/
61.03 19.53
$3.04
58.06
59.08
17.25
42.09 / 12.10
12.56
12.78
17.11
59.13/
57.19/
41.28 11.30
45 3030711
54.55 17.55
59.56/ 18.56
00.77 19:00
52,59/ 14.00
57.11 16.62
58.63 16.84
41.75 11.76
45.77/ 12:28
46.79 12.50
54.33/ 17.34
39.35/ 18.36
60.37 18.88
52.40/ 14.41
57.42/ 16.43
58, 58.44/ 16.65
41.50/ 11.51
12.03/12.03
46.55/ 12.26
Table 3: Our performance (%) AyroEye for varying frames per video Recall (R) and mean Recall (mR).
# Frume Discurded
20
PredCk
SGCls
10a101000
SGDet
kimR011000
56.20/ 19.25
61.02/ 20.67
62.40/
21.19
54.15/
54.15/16.22 16.22
59.59 18.20
60.37/
18.35
43.33 13.29
47.95
13.69
48.94/ 13.86
55 08 18.82
60. 39 20.26
61.15/ 20.76
53.07/ 15.90
58.40
17.84
59.16/ 18.01
42.66/ 13.02
46.97/
13,41
47.95/ 13.58
53.98/ 18,42
59.18/ 19.85
59.93/20.33 20.54
51.99/ 15.58
57.25 17.49
57,97/
17.05
41,81/ 12.75
46.05/
13.14
47,00/ 13.30
52.90 901 18,04
57.99/ 19.46
58.75/ 19:33
50.95 15.27
56.08 17.14
56.81 17.30
4097/ 12.40
45.11/ 12.87
46,06/ 13.03
51.84/ 17.60
56883 19,08
$7.55/
19.53
49.93/ 14.96
54.9 16,80
55,687
16.96
40.17 12.24
44.25/ 12,61
45.14/ 12.76
by correcting spurious or incomplete relationships and incorporating previously omitted ones using the temporal context from the frame sequence. As result, G. comprehensively represents persistent and transient interactions, including their direction and historical sequence within the video.
Loss Function. Visual object relationships involve predicates that may appear quite similar, such as "parking next to" and "stopping next to". Thereby, we utilize a multi-label margin loss, as in [22]]
Cp (T) PEP+ PEP-
Σ
Σ
max 6lr, + o(r,)))
In Eqn. (5), r represents subject-object pair, and p+ and P correspond to positive and negative predicates, respectively. The term ((r.p) measures the compatibility of the pair r with the predicate p. Additionally, object distributions are modeled using neural networks with ReLU activation and batch normalization. Cross-entrowe loss is applied during the learning process. The total loss is a combination of the multi-label margin loss Lp and the cross-entropy loss Lees defined as:
Listal
Lp + ALoe
(6)
where À is weight balancing the contribution of the cross-entropy loss Lee-
Experimental Results
In this section, we discuss the benchmark dataset evaluations and comparisons with SOTA methods.
5.1 Implementation Details
Dataset. We use 10-fold cross- volidaaali on the AeroEye dataset, including 1,797 videos for training and 463 videos for testing. We also evaluate our performance on PVSG [7] and ASPIRe [8] datasets.
Settings. We employ DINO [19] to extract the spatial attention graphs (in Section 4.2). DINO is trained with ResNet-50 backbone and 1500 queries on MAVREC, achieving 92.35 mAp on the validation set. The pre-trained detector is applied to baselines, and parameters are fixed during subsequent task training. Our model is trained on 8 A6000 GPUs using 12 epochs with Adaml/ optimizer (initial learning rate of le-5 and a batch size 1), gradient clipping (max norm of 5).
Evaluation Metries. We evaluate models on two standard tasks in image-based scene graph gen- eration followed by previous work [74, 7] that are predicate classification (PredCls). scene graph classification (SGCls), and scene graph detection (SGDer). While SGCIs predicts relationships given ground truth objects, SGDet involves detecting objects and predicting relationships. These tasks are evaluated using Recall (R @ K) and mean Recall (mR @ K), where K E {20, 50, 100).
5.2 Ablation Study
Semantic Dynamics in Cyclic Attention. By altering 7 (in Eqn. (3)), we consider the permutation or non-equivariance equivariance. If the predictions systematically adapt to the shifts induced by
8

---

Table 4: Comparison (mean 1 std) AeroEye against baseline methods in of Recall (R).
Method
200
PredCis Re50
Ri0100
R0020
SGCb R
RG110
R0020
SGDet
R0:90
Vauila (2a)
50.12 + 1,30
54.68
5.70
56.32 = 3.5
48.09 + 1.78
53.23 + 5.66
54.99
3.58
31,04
3.41
34.28
0.72
54.62
1.10
Transformr (2b)
53.23
1.71
59.35
4.12
6089
= 0.88
$1.12 = 1.66
57.32
+ 3.91
59.19
0.83
41,09
0.42
46.52
0.63
43.15
0.83
HIG (2e)
54.18
1.23
59.59
5:90
60.35
5.30
$2.03
+ 1.19
57.4
+ 5.87
58. 18.18- 5.22
57.28
0.50
38.59
2.37
50.27
1.49
CYCLO (2d Oursi
$6.30 1 0.70
61.62
2.90
62.40
1,88
54.15
1 0.7
59.59
1 2.2
00.37 _ 1.8
43,53
0,30
47,93 1 0.65
48,94 0.79
Table 5: Comparison (mean + std) on AyroEye against baseline methods in terms of mean Recull (mR).
PredCls
SGDet m0
1000100
mKG550
mR
Vanila (3a)
12.21 0.51
0.42
2.05
8.77
Trunsformer (2b)
11.12
13:32
1.53
0.35
12.31
HIG (3c)
18.37
0.68
0.79
0.37
17:08
2.66
17.69
1.70
13.12
13.79
CYCLO (2d Ours)
19,23 L 0.32
20.67 0.42
21.19
0.48
16.22 1 0.04
18.20
1.29
18.38
0.43
13,29
0.46
13,69 0.53
1386
0.55
different values, demonstrates a degree of permutation equivariance. Conversely, if the predictions change in ways that do not correspond systematically to these shifts, may indicate non-pen ponpenmutttioon equivariance. Table 2 shows decrease in performance, implying disrupted temporal patterns and non-per pon-eeenutttion invariance, indicated by unpredictable output changes relative to input shifts.
Cyclic Dependency. To further validate the cyclic dependency of our model, we discarded frames from every successive frame. Removing frames disrupts the temporal continuity of the sequence, which is crucial for maintaining the the cyclic nature of video. If the cyclic model presupposes that each frame has direct relationships with its adjacent frames circularly, then removing frames could sever these relationships, potentially diminishing the ability to leverage cyclical patterns effectively. Indeed, Table 3 informs decrease in Recall and mean Recall when frames were reduced.
5.3 Comparisons with Baseline Methods
Table 4 shows CYCLO outperforms other methods across metrics and tasks, surpassing Transformer by 2.95%, 3.03%, and 2.44% in the PredCIs, SGCIs, and SGDet R @@20, and maintaining its lead even at higher Recall thresholds. Moreover, Table shows significant improvements at mR @ 20, mR@50 @5, and mR @ 100, with CYCLO leading HIG by 0.86% in the PredCIs task at mR @20 and exceeding HIG by 1.12% and 1.32% in the SGCIs and SGDet tasks, respectively, at mR @ 20. In addition, the consistent performance and low standard deviation of the CYCLO model across Table and 5 demonstrate its robustness and overall superiority. Fig. 7 displays that CYCLO can capture the evolving relationships between objects by updating their positions and interactions.
5.4 Comparisons with State-of-the-Art Methods
We compare our performance on two recent benchmark datasets on VidSGG (i.e. PVSG and ASPIRe).
Table 7: Comparative performance (%) of our model and previous methods on the ASPIRe datasct, evaluated by Recall (R) and mean Recall (mR).
Table 6: Comparative performance (%) of our model and previous methods on the PVSG dataset, evaluated by Recall (R) and mean Recall (mR).
Interactisity
RimR@/20
RimR9 100
Vanilla
Podiion
10:220550
971/0.32
21.96/0.3
21,93 /0.55
/9.11/0.40
58,05 10.050.62
Model
R/R@@0 20
R/mR@50
R/mR@ 100
Handerafted (2b)
Postion
10,73/052
22,04/0.59
38.16/0.71
Vanilla (2a)
2.35 1.22
2.71/1.31
2.4/ 1.45
Rolution
9.2/0034
22.03
39.22/
Handcrafted (2b)
1D Convolation (2b)
2.56/ 1.24
2.79 1.24
2.78/
2.80/ 1.47
1.35
3.05 / 1.54
3.10/1.50 1.59
ID Convolation (2h)
Relatio
Audiion
1095/052
10.150334
22.05 /0.52
22.06/0.71
39.27/
28.31/0.76
HIG (2c)
Transformer (2b)
4.02 1.75
4.60/ 1.89
4.41 / 1.86
4.88/2.05
5.43/2.23
4.88/2.03
Trandormer (2b)
Relation
Pudtion
11,0/083
10,23/0.65
22.53
22,51 /0.71
/0.90
39,84/ 1,00
39,50/0.00
CYCLO (2d- Ours)
5.83/1.98
6.12/ 2.15
6.70/2334
HIG (3c)
Relaton
Puciion
13:02/009
10.26/0.29
23.73./0.34
24.50/1.31
42.35/1.19 1.12
41.47/0.30
CYCLO Ours)
Relation
Pusiiin
1..71/0.5
/
26.07/ 1.45
34.95/1.00
16.44/1552
43.94/149
Performance on PVSG. In Table 6, we report that the CYCLO approach achieves superior perfor- mance compared to other models on the PVSG dataset, particularly in terms of Recall and mean Recall metrics. In particular, CYCLO surpasses HIG and Transformer by 1.23% and 1.81 at R @ 20, respectively. In addition, it consistently outperforms both models at higher recall rates, including R @ 50 and @ 100, showcasing its effecttieneess across various thresholds. Our approach also slightly

---

pelle opproitinn
polor
Tweddgiing
Inverigingg
ploss
Scene graphs generated the CYCLO model on the Ayro dataset, illustrating dynamic relation- ships between objects agents across UAV-captured frames. (Best viewed in colors)
Figure 7:
improves the mean Recall, as the experimental results demonstrate. These results highlight the robustness of our CYCLO approach in recognizing and handling periodic actions, such as cooking, washing, cleaning, exercising, and other routine tasks frequently represented on the PVSG dataset.
Performance on ASPIRe. The ASPIRe dataset includes five distinct interactivity types. However, we focus on position and relation to ensure fair comparison. As shown in Table 7, CYCLO consistently outperforms existing models across multiple recall and mean recall thresholds. Notably, R @220, our CYCLO approach outperforms the HIG method, the second-bes, model, by 0.69% in position and more substantial 5.03% in relation. Additionally, CYCLO shows significant gains in mean Recall across all evaluated thresholds, demonstrating its effectiveness in tackling the long-tail distribution.
6 Conclusions
We have introduced CYCLO, novel approach that effectively captures periodic and overlapping relationships, handles extended sequences, and minimizes information loss, making it suitable for complex temporal modeling. In addition, we presented AeroEye, comprehensive and diverse dataset composed of drone-captured scenes, specifically designed to represent intricate object relationships and spatial positions in aerial videos. Through extensive experiments the AeroEye dataset and two large-scale in- mild datasets (i.e. ASPIRe and PVSG), we demonstrated the robustness and effectiveness of CYCLO in capturing dynamic interactions and evolving relationships over time.
Limitations. Although our CYCLO approach has achieved impressive performance, may reveal limitations when dealing with incomplete discontinuous videos. The periodic and cyclic attention mechanisms, crucial for capturing temporal and spatial object relationships, heavily rely on video continuity and completeness. Interruptions in the sequence, such missing discontinuous frames, disrupt the formation of accurate cyclical references, leading to inconsistent and incorrect predictions.
Broader Impacts. The proposed approach improves the capture of object interactions and temporal evolution in acrial and in-the-wild vidcos, which critical for surveillance, disaster response, traffic management, and precision agriculture applications. By modeling object interactions over time, CYCLO supports more informed decision-making lcading to safer and more sustainable practices. This advancement opens the door for future work developing surveillance systems that can model complex relationships from drone videos. However, it is important to recognize the potential risks associated with this approach, particularly the possibility of using it for unauthorized surveillance.
Acknowledgment. This work is partly supported by J.B. Hunt Transport Services (JBHunt), NSF Data Science and Data Analytics that are Robust and Trusted (DART), NSF SBIR Phase 2, and Arkansas Biosciences Institute (ABI) grants. We also acknowledge Thanh- Dat Truong for invaluable discussions and the Arkansas High- Performaree Computing Center (AHPCC) for providing GPUs.
10

---

References
[1] Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2:: Pyramid pooling transformer for scene understanding. IEEE transactions on pattern analysis and machine intelligence, 2022. 2
[2] Naga VS Raviteja Chappa, Pha Nguyen, Page Daniel Dobbs, and Khoa Luu. React: recognize every action everywhere all at once. Machine Vision and Applications, 35(4):102, 2024.
[3] Kha Gia Quach, Ngan Le, Chi Nhan Duong. Ibsa Jalata, Kaushik Roy, and Khoa Luu. Non-volume basserving-basel fusion to group-level emotion recognition on crowd videos. Pattern Recognition, 128:108646. 2022. 2
[4] Naga Venkata Sai Raviteja Chappa, Pha Nguyen, Thi Hoang Ngan Le, Page Daniel Dobbs, and Khoa Luu. Hatt-flow: Hierarchical attention-flow mechanism for group-activity scene graph generation in videos. Sensors, 24(1 1):3772, 2024. 2
[5] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, al. Openscene: scene understanding with open vocabularies. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 815-822 824, 2023. 2
[6] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236-10247, 2020. 2, 3, 5, 21
[7] Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Bi, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, et al. Panoptic video scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18675-18685, 2023. 2, 3,5, 8, 21
[8] Trong- Thuan Nguyen, Pha Nguyen, and Khoa Luu. Hig: Hierarchical interlacement graph approach to scene graph generation in video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2024. 2, 3, 5, 8, 21
19] Xinpeng Liu, Yong-Lu Li, Xiaoqian Wu, Yu- Winging Tai, Cewu Lu, and Chi-Keung Tang. Interactiveness field in human-object interactions. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 201 13-20122, 2022.
[10] Jeeseung Park, Jin Woo Park, and Jong-Seok Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152-17162, 2023. 2
[11] Hanhua Ye, Guorong Li. Yuankai Qi, Shuhui Wang, Qingming Huang, and Ming-Isuaa Yang. Hierarchical modular network for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17939-179999 17948, 2022. 2,3
[12] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10714-10726, 2023. 2,
[13] Wang Lin, Tao Jin, Ye Wang, Wenwen Pan, Linjun Li, Xize Cheng, and Zhou Zhao. Exploring group video captioning with efficient relational approximation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 15281-15290 2023. 2,
[14] Mahmoud Khademi and Oliver Schulte. Deep generative probabilistic graph neural networks for scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
11237-11245_ 2020. 2,
[15] Bruno Cesar Oliveira Souza, Marius Aasan, Helio Pedrini, and Adin Ramirez Rivera. Selfgraphvqa: self-supervised graph neural network for scene-based question answering. In Proceedings of the IEEF/CVF
International Conference on Computer Vision, pages 4640-4645, 2023. 2,
[16] Jian Ding, Nan Xue, Yang Long, Gui-Song Xia Xia, and Qikai La. Learning roi transformer for detecting oriented objects in acrial images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2,
[17] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 (4(11):7380-399 2021. 2,3
[18] Aritra Dutta, Srijan Das, Jacob Nielsen, Rajatsubhra Chakraborty, and Mubarak Shah. Multiview acrial visual recognition (mavrec): Can multi-view improve acrial visual perception? In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2, 3,
[19] Hao Zhang. Feng Li. Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. DINO: DETR with improved denoising anchor boxes for end andtto-end object detection. In The Eleventh International Conference Learning Representations. 2023. 2, 6, 8
11

---

[20] Chang Xu. Jian Ding, Jinwang Wang, Wen Yang, Huai Yu, Lei Yu, and Gui-Song Xia. Dynamic coarse-to- fine learning for oriented tiny object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7318-7328, 2023.
[21] Xiang Yuan, Gong Cheng, Kebing Yan, Qinghua Zeng, and Junwei Han. Small object detection via coarse-to-fine fonne proposal generation and imitation learning. In Proceedings of the IEEE/CVF international
conference on computer vision, pages 63 1317-6327, 2023. 2
[22] Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference computer vision, pages 16372 16382, 2021. 2, 3.4.8
[23] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017. 2,3, 18
[24] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[25] Yuanzhi Liang, Yalong Bai, Wei Zhang, Xueming Qian, Li Zhu, and Tao Mei. Vrr-vg: Refocusing visually-relevant relationships. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10403-10412, 2019.
[26] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6708 6709, 2019. 3
[27] Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European Conference on Computer Vision, pages 178-196. Springer, 2022. 3, 4, 21
[28] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In Proceedings of the 25th ACM international conference on Multimedia, pages 1300-1308
2017. 3
[29] Tao Wu, Runyu Hc, Gangshan Wu, and Limin Wang. Sportshhi: A dataset for human-human interaction detection in sports videos. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2,3
[30] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao. Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user seeerreerattdd vidcos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pages 279-287, 2019. 3
[31] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10668-10677, 2020.
[32] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Autonino Furnari, Evangelos Kazakos, Jian Ma. Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100 100. International Journal of Computer Vision, pages 1-23, 2022. 3
[33] Chang Xu, Jinwang Wang, Wen Yang, Huai Yu, Lei Yu, and Gui-Song Xia. Detecting tiny objects in aerial images: A normalized wasserstein distance and new benchmark. ISPRS Journal of Photogranmetry and Remote Sensing, 190:79-93. 2022. 3
[34] Gong Cheng, Jiabao Wang, Ke Li, Xingxing Xic, Chunbo Lang, Yanqing Yao, and Junwei Han. Anchor- free oriented proposal generator for object detection. IEEE Transactions on Geoscience and Remote Sensing, 60:1-11, 2022.
[35] Luigi Riz, Andrea Caraffa, Matteo Bortolon, Mohamed Lamine Mekhalfi, Davide Boscaini, André Moura, José Antunes. André Dias, Hugo Silva, Andreas Leonidou, et al. The monet dataset: Multimodal drone thermal dataset recorded in rural scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2545-2553 2023.
[36] Gong Cheng, Xiang Yuan, Xiwen Yao, Kebing Yan, Qinghua Zeng, Xingxing Xie, and Junwei Han. Towards large-scale small object detection: Survey and benchmarks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2,3
[37] Dawei Du, Yuankai Qi, Hongyang Yu. Yifan Yang. Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian. The unmanned aerial vehicle benchmark: Object detection and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 370-386 2018. 2,
12

---

[38] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding crowded scenes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterlam, The Netherlands, October 11-14, 2016, Proceedings. Purt VIII 14, pages 549-565.
Springer, 2016.
[39] Tung Minh Tran, N Vu, Tam V Nguyen, and Khang Nguyen. Uit-adrone: A novel drone dataset for traffic anomaly detection. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing. 2023.
[40] Lichao Mou, Yuansheng Hua, Pu Jin, and Xiao Xiang Zhu. Era: A data set and deep learning benchmark for event recognition in acrial videos [software and data sets). IEEE Geoscience and Remote Sensing Magazine, 8(4):125- 133, 2020. 3, 4
[41] Murari Mandal, Lav Kush Kumar, and Santosh Kumar Vipparthi. Mor-uav: A benchmark dataset and baselines for moving object recognition in uav videos. In Proceedings of the 28th ACM international
conference multimedia. pages 2626-266 2635, 2020. 3
[42] llker Bozcan and Erdal Kayacan. Au-air: A multi-modal unmanned acrial vehicle dataset for low altitude traffic surveillance. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8504-85 IEEE, 2020. 3
[43] Isha Kalra, Maneet Singh, Shruti Nagpal, Richa Singh. Mayank Vatsa, and PB Sujit. Dronesurf: Benehmark dataset for drone-based face recognition. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). pages 1-7. IEEE, 2019. 3
[44] Margherita Bonetto, Pavel Korshunov, Giovanni Ramponi, and Touradj Ebrahimi. Privacy mini-drone based video surveillance. 2015 11th IEEE international conference and workshops on automatic face and gexture recognition (FG), volume 4, pages 1-6. IEEE, 2015. 3
[45] Slim Hamdi, Samir Bouindour, Hichem Snoussi. Tian Wang. and Mohamed Abid. End-to-end deep one-class learning for anomaly detection in uav video stream. Journal of Imaging, 7(5)900, 2021. 3
[46] Ye Lyu, George Vosselman, Gui-Song Xia. Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing, 165:108-119,
2020. 2,
[47] Jingkang Yang. Jun Cen, Wenxuan Peng. Shuai Liu, Fangzhou Hong, Xiangtai Li, Kaiyang Zhou, Qifeng Chen, and Ziwei Liu. 4d panoptic scene graph generation. Advances in Neural Information Processing Systems, 36, 2024. 2
[48] Rongjie Li, Songyang Zhang, and Xuming He. Sgir: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19486- 19496, 2022. 3
[49] Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for an end-to-end scene graph generation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3
[50] Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3,
[51] Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 6292-6300 6301, 2023. 3
[52] Kibum Kim, Kanghoon Yoon, Jachycong Jeon, Yconjun In, Jinyoung Moon, Donghyun Kim, and Chany- oung Park. Llm4sgg: Large language model for weakly supervised scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3
[53] Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Composi- tional 3d scene synthesis from scene graphs. arXiv preprint arXiv:2312.0009: 2023. 3
[54] Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu. Target adaptive context aggregation for vidco scene graph generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13688-13697, 2021.
[55] Siddhesh Khandelwal and Leonid Sigal. Iterative scene graph generation. Advances in Neural Information Processing Systems, 35:24295 243088, 2022. 3,
[56] Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy-Chowdhury. Unbiased scene graph generation in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22803-22813, 2023. 3, 4
[57] Trong-Thue Thuan Nguyen and Minh-Trie Tran. A transformer based approach for dynamic referee assistance. In 2023 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), pages 1-6. IEEE, 2023. 3
13

---

[58] Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji. and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. pages 20090-23099, 2023. 3
[59] Chaolei Tan, Zihang Lin, Jian-Fang Hu, Wei-Shi Zheng, and Jianhuang Lai. Hierarchical semantic correspondence networks for video paragraph grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18973-18982, 2023. 3
[60] Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, and Wei-Shi Zheng. Collaborative static and dynamic vision-language streams for spatio-temporal video grounding. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 23100-23109, 2023. 3
[61] Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Moreney, Ruslan Salakhutdinov, and Ali Farhadi. Video relationship reasoning using gated spatio-temporal energy graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (0424-10433, 2019. 3
[62] Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao. Video relation detection with spatio-temporal graph. In Proceedings of the 27th ACM international conference on multimedia, pages 84-93, 2019. 3
[63] Xindi Shang, Yicong Li. Junbin Xiao, Wei Ji and Tat-Seng Chua. Video visual relation detection via iterative inference. In Proceedings of the 29th ACM international conference on Multimedia, pages 3654-3663 2021. 3,
[64] Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Luu. Type-to-track: Retrieve any object via prompt- based tracking. Advances in Neural Information Processing Systems, 36, 2024. 3
[65] Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Son Lam Phung. Ngan Le, and Khoa Luu. Multi-camera multi-object tracking on the move via single-stage global association approach. Pattern Recognition, 152: 154:1,, 2024. 3
[66] Kha Gia Quach, Pha Nguyen, Huu Le, Thanh- Dat Truong, Chi Nhan Duong, Minh-Trie Tran, and Khoa Luu. Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13784-13795 2021.
[67] Jingwei Ji, Ranjay Krishna, Fei-Fei Li, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020.
[68] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu, Memory-and -nttaiiiiipattionn transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference Computer
Vision, pages 13824-1 13835, 2023. 4
[69] Shugiang Cao, Weixin Luo, Bairui Wang. Wei Zhang, and Lin Ma. A circular window-based cascade transformer for online action detection. arXiv preprint arXiv:2208.1422) 2022.
[70] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on ongion-of-intersss arXiv preprint arXiv:2307.03660 2023. 17
[71] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision. pages 213-229. Springer, 2020.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you nced. Advances neural information processing systems, 30, 2017.
[73] Thanh-Dat Truong, Huy Bui, Chi Nhan Duong. Han Man-Scok Sco, Son Lam Phung, Xin Li, and Khoa Luu. Direeformer: directed attention transformer approach to robust action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20030-20044 2022. 7
[74] Cewu La, Ranjay Krishna, Michael Bemstten, and Fei-Fei Li. Visual relationship detection with language priors. ECCV, 2016.
[75] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Eision-ECCT 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pages 740-755. Springer. 2014. 18
[76] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a- judge with MT -T-bbecchae and chatbot arena. In Thirty-sevente Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. 18
14

---

[77] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023. 18
[78] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961 -269, 2017. 22
15

---

Appendices
A
The AeroEye Dataset
A.1 Relationship Definition
Our AeroEye dataset focuses on capturing the spatial positions and relationships between objects. including person-to-object and object-to-object interactions. Table A.8, A.9 provides comprehensive list the defined positions and relationships. Furthermore, Table A. offers guidance on the final relationship vocabulary specific to each scene category within the dataset.
Table A.8: list of the 135 position predicates defined on the AeroEye dataset.
above
at focal points at intersection at the ends of the pool
near playground equipment
above debris
safe exits
above water lines
near scoring zones
across different terrains
at the forefront of movements
near storage areas
across multiple lanes
at the helm
next to
across negofiation tables
at the junction at the starting line
on benches
across ploughed fields across the court
opposite
at the stern or sides at turning point
oriented
across the site adjacent to
outside
behind
outside danger zones
ahead in the race
behind agricultural machinery
outside of traffic flow over
ahead of competitors
behind the congregation
aligned on the track
below unstable hillsides
over open waters
along designated route
beneath
parallel
along cleared pathways
beside
parallel in lanes
along docks and aboard ships
beside bike lanes
surrounding the field
along monitored sections
beside debris
through city streets
along paths
beside merchandise stands
through debris
along pit stops
beside pool edges
through rows of crops throughout the fields to the left to the right together
along roads
beside the congregation
along the field
between
along the route
between buildings
along the sidelines
between conflicting parties
along the sides
between crop rows
under
along the sides of the pool
centered
undergoing
along viewing areas walkways
close by
underneath
alongside construction machinery
close to
within affected zones
amidst debris
closely positioned
within audience clusters
angled
clustered
within bottleneck points
around campus landmarks
distant
within cabin areas
around common areas
down
within crash sites
around damaged structures
facing
within crowd formations
around familiar objects
facing the audience
within dance areas
around heavy equipment
facing the stage
within debris piles
around penalty boxes
from vantage point
within designated lanes
around refreshment tables
from central points from control centers
within discussion circles
around ritual objects around stage areas around temporary aid stations around the deck around the fire around the vehicles around water features
within goal areas
in designated pit areas
within marked lanes
in front in lane indifferent inside
within moored boats
within open fields
within outfield areas
within reach of fire hydrants
interwoven
within spectator areas
safe distance at batting positions
near near debris
within surveillance zones
within the basketball court
16

---

Table A.9: A list of the 249 relationship predicates defined the AeroEye dataset.
acquaintances
dribbling
leading the way
ridiny
adhering to
drivers
learning
runners
alcrting
eucircled by
learning farming
runtung
oligned in trathic lane
encireling
living in coastal community
scoring
aligned
enclosed
loading
selling
aligning in lane
engaging in beach sports
loading cargo onto ship
selling merchandise
Bllnn sitlwall
engaging leisure activity
managing airport operations
setting up
Blong the lane
engaging in school sports
managing industrial operations
setting up temporary shelters
Bllng
engaging winier activilies
munaging urban
Sharing
blong the sidewalk
enjoying concent atmosphere
merging lane
sharing slogins
amidsting traffic
enjoying lakeaide park
mimicking
sharing excitemeot
appreciating exhibits
enjoying live music
moniloring
shopping
apgrosiching
enjoying ouidoor cinema
molivvaling
shopping in city
upproaching intersection
enjoying outdoor festivities
showcasing classic
MI center of intersecoio
enjoying spring lesiivilles
moving al intersecoion
showcasing local art
the conjunction
enjoying waterfmot gathering
moving towards
singing
at the stop sign
onjoying wilderness
navigating
Skating
attending games
enjoying
negotiating
skiing
boarding
enjoying winter sports
observing
soctalzing
boaters
enlertaining outdoor crowd
operating docking area
standing inside
bonllering
excavaling
operating market stalls
standing outsade
bunlding coustal protection
exchanging
opponents
Stopiig
bailding team colesion
exercising
overlooking
straddling lanes
building underwater structure
exploring forest trails
overtaking
surrounded by
bailding urban infrastructure
exploring mountain trails
owner
Swimmers
camping
exploring urban landscupes
painting
tackling
cupturing
exploring wildliiee habital
purallel
leammnats
capturing urhan night lights
exiernal to the huildingg
parking at
tending to agricultural tasks
catching
extinguishing
participating community
louning
celehrating
farmer and harvesting tool
participating in restivites
loward destlmation
celcbrating tostive atmosphere
participating maraihon
towing
celebrating public festival
fioding parking lot
participating in music camp
tracking
chamgang
lireighters
parlners
trading
Chasing
Anating
passing
Iraining
cheering
following
perforning
traveling
cleaning
forefronted
picnicking logether
Fraversing
close friends
players
trying to mediate resolutions
co-traveling
friends dancing together
playing
under observation
coaching
gathering
police
under surveillance
coincidental
graduating
practicing
undergoing
collaborating
handling emergency
preparing for emergencies
undergoing process
collecting crups
handling the plough
preserving beautiful moments
unloxding
collecting soil samples
harvesting
preserving natural resouroes
utilizing iveeside path
colliding
beading
protecting waterways
vacationing
communicating
beading towards
protesters
viewing
competing
helping
providing
visiting
conserving marine environment
biking
racers
visiting historical landmarks
constructing
hindering
racing adventure challenge
wailing light
contigaous
bomenwner and damaged property
racing through city streets
walking
cooperating
bosting lakeside wedding
racing through mountain trails
Watching
coordinated in lane
bosting ouidonr concert
misinn awareness and finds
Watching wildlife
coordinating
in conflict
rear
Within parking area
coordinating logistics
Inline
reconstructing
within traffic
srushing
in the middle
reenacting
worker and constraction tools
crossing
Independent
regulated
workers
Cyclissts
insitee constraction zone
relaxing
working
Nehatiin
Inspecting
repairing
worshipers
demonstrating
installing
resident and mudslide debris
serving food
directing along
interacting
ovacuating together
pitching
directing toward
Invessigating
responding
hosting rest stop
discussing
isllaed during flood
resting
displaying
kicking
retrieving
distriburing
revamping
A.2 Annotation Pipeline
To capture frequent and rapid changes in aerial videos while reducing redundancy, we annotate keyframes at 5FPS. At each frame, our annotation pipeline consists of two stages:
Stage 1: Object Localization and Tracking. We manually annotate bounding boxes with predefined categories, providing precise object localization and consistent tracking throughout the video.
Stage 2: Relationship Instance Annotation. To generate diverse predicates, we leverage the
GPT4Rol [70] model, which combines visual and linguistic data to generate detailed descriptions of object relationships within specified regions of interest. The process involves the following steps:
17

---

Table A.10: A hierarchical representation of the 249 relationship predicates on the AyroEye dataset, organized into 29 high-level semantic scene categories.
Scene
Relationship Predictes
Bascball
checring, watching, players, compatnng passing, scoring cheering. watching, players, competing. driblling, passing Boating, bouters, navi gating, enjoying waterfront guthering studying, engaging in school sports, learning. graduating racing through city streats, drivers, overtaking, moving
Basketball
Bonting
Campus
Car Racing
Concest
enjoying concert atmosphere, cheeringg watching, participating commanity event confliet, debaiing, trying mediatc resolutions, responding huilding infrasmmeture, inside consmnucron zone, worker and construction tools, workers, working, inspecting, installing moving, traveling. riding. eyclists, navigating, along the road extinguishing, responding. firefighters isolated during flood rescuing, responding, isolated during flood, flonting
Contlict
Consanectng
Cycling
Fire
Flood
Harhour
loading cargo onto ship, operating docking area. floating. navigating collecting crops. farmers, harvesiing, tending to agricultural tasks rescuing, responding. homesmmer and damaged property, isolated during floed, managing industrial operations rescuing, responding, homeommer and damaged property, undergoing process, managing industrial operations iollated during flood, observing. under surveillacee
Harvesting
Landslide
Mudslide
NonEvent
Parade Protest
demensttating, sharing slogans, in line. participating in community ovent picnicking logetherr enjoy ing lakeside park, engaping leisure aciiiity. walking celebrating, socializing, enjoying outdoor festivities dancing. singing tending to agricultural tasks, fammers, moving along the lane chasing, responding, following behind, confiict, moving at intersection rescuing,
Park
Party
Plooghing
Police Chase
Post Eartiquake
reconstructing. homeowner and damaged property wonshipers, engaging in leisure activity, paricipating community event moving, trvelling navigating runners, patticipating in marathon cheering, watching. players, compering, passing, scoring, kicking, marking enjoying lakeside park. enjoying waterfront gathering swinuners, floating, tackling appouching, emsshing responding, colliding, drivers, the mickle of intersection approaching. drivers, wailing at light, along the road, sharing lane appmoaching, monitoring, managing urban tralfic, observing, under surveillance
Religions Activity
Running
Soccer
Swimming
Traftic Collision
Trafiic Congestion
Traffic Monitoring
1. Text Generation: We leverage GPT4Rol integrateed instruction tuning with a large lan- guage model (LLM) to enhance interactions with regions of interest (Rol) within images. This model transforms bounding box references into language instructions, enabling detailed descriptions and reasoning about specific image regions, thus improving image under- standing granularity and accuracy. It utilizes a variety of transformed multimodal datasets, including COCO [75] and Visual Genome [23], to refine the alignment between visual and linguistic data, ensuring precise responses to spatial queries.
In particular, We input bounding boxes around objects with prompts such as, "What is the relationship between <object_ 1> in <region_ 1> and <object 2> in <region_2>> where obbject_i i> is a category name that labeled in Stage 1, and GPT4Rol replaces <region_ i> tags in these instructions with results from RolAlign, derived directly from the features of image. The model uses RolAlign to extract region-speciin features and combine them with language embeddings. The resulting multimodal embeddings are then interpreted by the Vicuna model [76], an instance of LLaMA [77].
2. Predicate Summarization and Selection: We employ a custom-destin -eeiigee filter to catego- rize the generated text into relationship types. This filter utilizes combination of keyword matching, dependency parsing, and semantic analysis to identify and classify the predicates accurately. The filter is designed to handle sentence structure and terminology variations, ensuring that the identified predicates are correctly mapped to their corresponding relation- ship types. Furthermore, the filter incorporates confidence scoring mechanism to prioritize high-quality predicates and filter out irrelevant or ambiguous ones. The final selection of predicates undergoes human oversight, where experienced annotators review and validate the filtered results. This manual verification step ensures the highest accuracy and relevance of the identified predicates, mitigating potential errors introduced by automated processing.
Quality Control To maintain the highest standard of annotation quality, we implement the following comprehensive measures:
Stage 1: Object Localization and Tracking:
All bounding box annotations are performed manually by skilled annotators with- out relying on automated detection or tracking models. This ensures precise object localization tailored to the specific characteristies of acrial vidcos.
18

---

We employ a rigorous double-checking process, where each frame in every video is carefully reviewed by a second annotator. This step helps identify and rectify any inaccuracies in bounding box placement dimensions.
In cases where object identities are inconsistent across frames due to occlusion, visual similarity, or other challenges, annotators meticulously correct the object numbers to maintain consistent tracking throughout the video.
Stage 2: Relationship Instance Annotation:
Annotators undergo extensive training using carefully curated examples from previous VidSGG datasets. This training familiarizes them with the intricacies of extracting predicates generated by the LLM and ensures a deep understanding of the annotation guidelines and best practices.
To minimize individual biases and ensure the robustness of annotations, we implement a repeated annotation process. Each video distributed multiple annotators, who independently extract and record the relationship instances. This redundancy allows for cross-validenn cross-validation and helps identify potential discrepancies or ambiguities.
In cases where annotators disagree on the extracted predicates or their categorization, a highly experienced meta-annotator is assigned review the conflicting annotations. The meta-annotator carefully examines the video content, considers the perspectives of the individual annotators, and makes the final decision the annotation record. This hierarchical review process ensures consistency and accuracy across the dataset.
By employing these rigorous quality control measures at each stage of the annotation pipeline, we ensure the highest level of accuracy, consistency, and completeness relationship instances.
A.3 Data Format
Our annotations are stored in JSON (JavaScript Object Notation) format organized as below:
[fta[{
"file_name": str,
"height": int, "width": int,
"image_id": int, "frame_index": int,
"video_id"; int, "metadata":
9
"id": int,
10
"cateeory_ide id"": int,
11
"iscrowd"; or 1, "area": int
13
}},
"annotations" [[
15
"bbox": [x, y, width, height],
"bbox_mode": 0 or 1,
17
"cteeoory_id": int,
"track_id"; int }}. 19
"positions": [[
21
id, metadata_id,
23
id id }},
25
"relations"; [[
metadata_id,
metadata_id,
28
relation_id ]]
30
}}.
"categories"; {
19

---

32
"id": int, "name": str
34
36
"id": int,
37
"name": str
39
{
"id" int, "name": str
41
Basic Image Information, This section contains the fundamental attributes of each image:
f ile_name (str): The name of the image file.
height (int): The height of the image in pixels.
width (int): The width of the image in pixels.
image_id (int): A unique identifier for the image.
frame_indes (int): The index of the frame within the video sequence.
video_id (int): An identifier for the video to which this image/frame belongs.
Metadata. This section includes the metadata key, which is list of segments within the image.
Each segment contains:
id (int): Unique identifier for the segment.
category_id (int): Identifier for the category of the object in the segment.
iscrowd (0 or 1): O for a single object and 1 for a cluster of objects.
area (int): The area covered by the segment in the image.
The annotations key contains a list of corresponding bounding boxes for each entry in metadata,
each tagged with a specific category
bbox (list): Xx_center, y- yenter,, width, height] of the bounding box.
bbox_modd (0 or 1): Bounding box mode.
category_id (int): Identifier for the object category in the bounding box.
track_id (int): Identifier to track the bounding box across different frames.
Relationship Attributes. This section encompasses lists of positions and relations for each segment, including two different metadata_ ids to represent the interactivity between two segments:
positions (list): List of position relations between segments, each containing:
metadata_id (int): Identifier for the first segment.
metadata_id (int):: Identifier for the second segment.
position_id (str): Identifier for position relation between the segments.
relations (list): List of other relations between segments, each containing:
metadata_id (int): Identifier for the first segment.
metadata_id (int): Identifier for the second segment.
relation_id id (str): Identifier for relationship between the segments.
These descriptors represent lists specifying various subject, object, and interactivity aspects for each bounding box within the annotations and metadata. For example, [3, O, 5] indicates that the third and first metadata segments share the relationship with an ID of 5.
20

---

Puatber of Intaa Oog msain giu Pottiar Predicate Number Objects (log scale)
Cal
parson Soetiii
Ligh
traffic light
boat Rol
hoop
ball
bus
tree
house
padlle
plcyclo
traffic sign
crane
ooal
bamee
free lifesva
barch
br talbos metoroppo
fonce
unbo eld
tram pee
spom
like.
pool
sho
other
lird
statue
hoss
wine turbine
scoreborre
stonag tank
score board
baoy stroler
Church
dam
street
besketballl
Figure A.8: Distribution of objects per category on the AcroEye dataset.
Figure A.9: Distribution of position predicates per category on the AcroEye datasct.
A.4
Additional Statistics
We present object, position, and relationship statistics per category in Fig. A.8, A.9, and A.10.
A.5 Data Samples
Fig. A.11 presents selected samples from AeroEye dataset, distinguished by its detailed bounding box annotations and meticulous relationship descriptions across various scenarios. As outlined in Section 3.2, each frame within AeroEye is annotated with precision and contextual relevance, ensuring clarity and avoiding the common ambiguities, such as generic or overlapping labels, prevalent in other datasets. A key feature of AeroEye is its categorization of relationships into positions and relations, as illustrated by the curved arrows representing positions and straight arrows denoting relations in Fig. 1. This multifaceted approach to annotation renders AeroEye uniquely comprehensive when compared to other datasets [6, 27, 7, 8J. The meticulous annotation process and the meticulously curated, information-rin rich nature of the AeroEye dataset firmly establish it as an invaluable resource, poised to catalyze significant advancements in relationship modeling within drone videos.
B Concepts
B.1 Progression
Relationship Representation. In each frame, a detector provides to a set of object features (6,
05 (,((), bounding boxes (N()) and category distributions {d)..... de 2010) for
21

---

Figure A.10: Distribution relationship predicates per calegory on the AeroEye dataset.
moving front of
person
Tredghtees
Banndn in the FIGHt lamv.
Aas upp pud ui Rumooa
toalighters
moving the right
moving
Indrghhers
moving the left nest
Figure A.11: Our AeroEye dataset includes diverse range scenarios, objects, and relationships.
the detected objects. These elements are used to define the relationships //.
at @ow, 2t, ow, ow (TT))) o
(B.7)
where [] denotes concatenation, denotes flattening operation, C represents element-wise addition, and the linear transformation matrices Φw,, ow., and Φw,, 0f implics the feature map of the union box, computed by the RolAlign [78], while f is function that converts the bounding boxes of the subject Ūū and object 76 into feature map with the same dimensions as oty. The semantic embedding vectors 3 and 8y are obtained from the object categories 2, and 25. respectively.
In the progressive approach, features of for each object are obtained using a detector such as Faster R- CNN in each frame. These features are utilized as specified in Eqn. (B.7) to compute the relationship features St. Finally, these features are processed through multilayer perceptions (MIPss) followed by softmax activation functions to classify the types of relationships between objects.
22

---

B.2 Batch Progression
In contrast to the Progressive approach, the Batch Progressive approach passes the relationship features through a Transformer architecture before classification.
A set of relationship features X. { 2 ......... _ is fed into the Transformer Encoder, which focuses on understanding the spatial context by inputting these relationship features into a sequence of identical self-attention layers, defined as:
n)) = Attenc (Q= K VV= V-X-=-1)) X(--1)
(B.8)
Each n-th layer receives the output from the (n-1)-th layer as its input, iteratively refining to enhance the representation of spatial relations embedded in the features, where denotes the layer number. The outputs are then processed by the Decoder, which captures temporal dependencies between frames, applying sliding window over the sequence of spatially contextualized representations:
2, = 1x.,..,,
E (T....,T)
(B.9)
where 7) is the window size, and T is number of frames. The positional encoding E, [ei, ... enl is embedded into the input to maintain sequence order:
Q= 2, + Ey. Z,, Z Attase. (Q, K, V).
(B.10)
Equation (B.10) enables the decoder to process each batch using self-attention layers, combining relation representations Z; with positional encodings Ey.
B.3 Hierarchical Graph
The hierarchical interlacement graph (HIG) method abstracts video content by representing temporal relationships using multi-level graph structure. Higher-level graph cells encompass broader seg- ments of video frames, allowing the HIG to efficiently capture and model the temporal dependencies and connections across different time scales within the video. In this method, object features of from individual frames are spatially and temporally fused to form graph nodes. These nodes are interconnected across successive frames, resulting in a series of interconnected frame-based graphs (V, E,)) where each G, is defined by its vertices Vi and edges E,. As the graph traverses, the total number of graphs decreases, ultimately resulting in singular graph representing the entire video. This hierarchical graph operates on predefined hierarchical levels L. At each level 7, the temporal scope is adjusted to T, T I + and new graph is generated that is specific that level and timeframe. Therefore, node features within each graph are dynamically updated through the computation and aggregation of messages from adjacent nodes, computed by weight matrices for each level and node pair (ū,, k,). This iterative refinement process is applied across all levels, resulting in a consolidated graph structure and updated feature set. Finally, relationship features between each node pair (ū,, 02) are fused and analyzed to classify the relationships.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Contributions in Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

While this paper does not include purely theoretical results, it does incorporate derivations of relevant formulas, along with their respective assumptions and complete proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see the Subsection 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our dataset being distributed at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
sinoiiicsyyycy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nipp.ccc//////// wttpccccppppcccceeeeeececcccclle htw.biii/ggdddessddeetttiiiiii
CCodeSubmie issiissi swwssssssnooiiiiiiieeeeeiiooeiie ceiiionoolii Policcy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Subsection 5.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Subsection 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eerpppeee/ee//ee cc hwtp///bll./ttccccccuuueeoeeedde
Answer: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Broader Impacts Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
27

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original papers that produced the code dataset are cited properly, including authors, title, publication venue, year, and link if available.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. wwwcco////o/oooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
28

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces a new dataset, which is discussed in Section 3 and Appendix A.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "TdwSmOvNBA", "reviewer": "Reviewer_Zf4k", "summary": "This paper proposes a video scene graph generation dataset called AeroEye on aerial videos and a framework called cyclic graph transformer to tackle the problem of video scene graph generation. The authors annotated the ERA and MAVREC dataset with keyframes at 5FPS. They manually annotated the frames for bounding box localization for each frames along with the tracking. The relationship annotations were done using a GPT4RoI model. The proposed cyclic graph transformer uses a cyclic attention mechanism which the author claims, are able to capture the direct and long-term temporal dependencies.", "questions": "1. Did you do evaluation on the 5 keyframes for each videos?\n2. point 5 in weakness section", "limitations": "yes, the limitations are addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed video scene graph dataset for aerial videos with a diverse set of predicates can offer more granular and nuanced understanding of dynamic interactions and relationships within aerial footage\n2. The dataset will be publicly available\n3. The proposed approach for circular attention for dynamic online scene graph generation seems promising", "weaknesses": "1. the paper is very difficult to follow. There are separate discussion sections which somewhat disrupts the flow.\n2. line 82 seems incomplete\n3.Line 118: 'no temporal edge is treated a boundary' can it not be a disadvantage as well since it does not takes into account for an event boundary?\n4. A very brief discussion of annotation procedure should be in the main paper such as manual annotation of bounding boxes and relationship annotations done by GPT4RoI model.\n5. For loss function section, it would be better to include  the overall loss equation. Does the object distribution refers to the object detection loss using the DETR?\n6. For table 2, you can refer to the shift value term from equation 3. It will be easy to follow. Table 2 refers to the ablation studies 'Semantic Dynamics in Cyclic Attention'. I think this section and the section followed by it should have a detailed explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer **Zf4k** for your thoughtful review. We appreciate the recognition of our novel dataset and approach. We acknowledge the need to provide clearer explanations.\n\n**Q1: The paper is very difficult to follow. There are separate discussion sections which somewhat disrupts the flow.**\n\nWe appreciate your feedback on the paper's readability. We are encouraged that multiple reviewers found our paper **clear** (Reviewer **nRfD**), **well-written**, and **easy to follow** (Reviewers **QFfh** and **5Gaf**). Nevertheless, we value all the feedback and will further enhance our paper's organization in the revision.\n\n**Q2: Line 82 seems incomplete 3.Line 118: 'no temporal edge is treated a boundary' can it not be a disadvantage as well since it does not takes into account for an event boundary?**\n\nWe have updated line 82. Regarding line 118, 'no temporal edge is treated **as** a boundary' is advantageous. CYCLO's continuous temporal information flow allows it to **capture gradual changes and complex dynamics**, such as the slow formation of traffic jams. This would be missed by models that frequently reset at event boundaries.\n\n**Q3: A very brief discussion of annotation procedure should be in the main paper such as manual annotation of bounding boxes and relationship annotations done by GPT4RoI model.**\n\nThank you for your suggestion. We will include a brief overview of the annotation procedure, with full details provided in **Appendix A.2**.\n\n**Q4: For loss function section, it would be better to include the overall loss equation. Does the object distribution refers to the object detection loss using the DETR?**\n\nWe will add the combined loss equation in line 242, where we mention that **the total loss combines these two losses**. 'Object distribution' is **not the detection loss** which refers to predicted **class probabilities** for DETR-detected objects.\n\n**Q5: Did you do evaluation on the 5 keyframes for each videos?**\n\nAs mentioned in Section 3.2, we **annotated keyframes at 5 FPS** and thus evaluate the model's performance on these keyframes. \n\n**Q6: For table 2, you can refer to the shift value term from equation 3. It will be easy to follow. Table 2 refers to the ablation studies 'Semantic Dynamics in Cyclic Attention'. I think this section and the section followed by it should have a detailed explanation.**\n\nWe appreciate your suggestion and will incorporate it into the revision. The shift term shift term ($\\eta$) in Eqn. 3 plays a crucial role in our model's temporal coherence and resolution. Our analysis reveals several key insights:\n1. Optimal temporal coherence: At $\\eta = 1$, the model achieves peak performance by effectively **capturing transitions between adjacent frames**.\n2. Information loss with increasing $\\eta$: Larger $\\eta$ values cause the model to **skip intermediate frames**, resulting in loss of crucial temporal information and consequently degraded performance.\n3. Maximized temporal resolution: $\\eta = 1$ allows the **capture of fine-grained dynamics**, which is essential for accurately predicting object interactions.\n4. Impact on sequence modeling: Higher $\\eta$ values impede the model's ability to **capture long-range dependencies**, affecting the integration of detailed temporal features.\n5. Frame order sensitivity: The model's non-permutation equivariance ensures **sensitivity to frame order**. For instance, the sequence of a person approaching a car, entering it, and then driving away must be captured correctly for the scene graph to represent the event accurately.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBVI3K4PhY", "reviewer": "Reviewer_nRfD", "summary": "This paper presents a new problem: modeling multi-object relationships from a drone's perspective. To address this, the authors propose the AeroEye dataset and introduce the Cyclic Graph Transformer (CYCLO) method. This method captures both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The authors not only validate the CYCLO approach on the AeroEye dataset but also test it on the PVSG and ASPIRe datasets, demonstrating the effectiveness of their method.", "questions": "1. The current model proposed by the authors shows low mR@K scores on the AeroEye dataset. However, from the figures in the supplementary material, the long-tail distribution of relations in this dataset does not appear to be very severe. What could be causing the low mR@K scores? If possible, I would like to see some typical bad cases.", "limitations": "It is recommended that the authors consider the limitations of this paper not only from a technical perspective but also from a societal standpoint. Given that relationship modeling from a drone's perspective may lead to widespread applications in surveillance and potentially significant impacts, this aspect requires special consideration.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors have introduced the problem of multi-object relationship modeling from a drone's perspective for the first time and constructed the AeroEye dataset, which effectively fills a gap in the field of multi-object relationship modeling and has significant application value.\n2. The CYCLO method proposed by the authors is not only useful for the dataset introduced in this paper, AeroEye, but is also a versatile method that can be applied to general Video SGG tasks. It has been tested on datasets like PVSG and achieved good performance.\n3. The structure of the paper is clear, the writing is standard and fluent, making it easy to understand.", "weaknesses": "1. The method proposed in this paper could benefit from a clearer network architecture figure, which would allow everyone to better understand the method presented.\n2. Is there an issue with non-differentiability in the Cyclic Attention described in Eq. 3? It would be helpful if the authors could provide further explanation of the Cyclic Attention mechanism.\n3. It is suggested to include some analysis of bad cases, which would help future researchers understand from which directions further optimizations can be made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to Reviewer **nRfD** for your recognition of the AeroEye dataset and the CYCLO method. We will enhance our paper by improving the architecture figure, addressing Cyclic Attention issues, expanding the failure case analysis, and elaborating on the suggested limitations.\n\n**Q1: The method proposed in this paper could benefit from a clearer network architecture figure, which would allow everyone to better understand the method presented.**\n\nWe have included a detailed network architecture figure in the attached file. \n\n**Q2: Is there an issue with non-differentiability in the Cyclic Attention described in Eq. 3? It would be helpful if the authors could provide further explanation of the Cyclic Attention mechanism.**\n\nEqn. 3 has **no non-differentiability** issues. The modulo operation (mod T) is used **only for key matrix indexing** and **does not affect gradient computation**. Gradients flow through differentiable components (dot product, softmax, summation), while the modulo operation implemented by a for-loop for indexing does not interfere with continuous gradient computations.\n\n**Q3: The current model proposed by the authors shows low mR@K scores on the AeroEye dataset. However, from the figures in the supplementary material, the long-tail distribution of relations in this dataset does not appear to be very severe. What could be causing the low mR@K scores? If possible, I would like to see some typical bad cases.**\n\nWhile the AeroEye dataset does not show a severe long-tail distribution of relationships, the low mR@K scores stem from the model's challenges in adapting to **rapidly changing relationships** within dynamic scenes in videos. In **fast-paced environments** like sport events or emergency responses, which were not well-investigated in OpenPVSG or ASPIRe, the model needs to work on keeping pace with swift changes in actions and interactions. An instance of bad cases is in the attached rebuttal file, the model need to quickly update its prediction of player interactions in soccer.\n\n**Ethics Review**\n\nIn line 124, we mentioned that we use **videos from the ERA [A] and MAVREC [B] datasets**, without including new videos. Videos are compliant with the European Union’s drone regulations [B]. It is always possible that some individual or an organization can use this annotation to devise a technique that can appear harmful to society. However, as authors, we are absolutely against any detrimental usage of our annotation and pledge not to support any detrimental endeavors concerning our data or the idea therein.\n\n**References**\n\n[A] Mou, L., Hua, Y., Jin, P., & Zhu, X. X. (2020). Era: A data set and deep learning benchmark for event recognition in aerial videos. IEEE Geoscience and Remote Sensing Magazine, 8(4), 125-133.\n\n[B] Dutta, A., Das, S., Nielsen, J., Chakraborty, R., & Shah, M. (2024). Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22678-22690).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fatObVvoE9", "reviewer": "Reviewer_QFfh", "summary": "This paper tackles an interesting task for understanding video scenes that focuses on modeling object relationships in aerial videos. Specifically, it introduces a new dataset, the AeroEye dataset, and proposes a novel approach, CYCLO, to better model the video object relationships. Experimental results demonstrate the effectiveness of CYCLO. CYCLO also achieves state-of-the-art performance on two scene graph generation benchmarks.", "questions": "- In the ablation study (line 265), when a frame is discarded, is it something like that the frame is set to some random noise?", "limitations": "I don't have concerns about the potential negative societal impact of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written, easy to follow, and presents many key points clearly.\n- Introducing the AeroEye dataset is a valuable contribution, as it fills an important gap in video scene graph generation datasets by providing a drone perspective relation dataset.\n- The design of CYCLO is both interesting and inspiring. The paper also demonstrates its superior performance compared to prior solutions.", "weaknesses": "- It would be great to report the inference cost of the proposed approach. \n- It is unclear how the proposed solution works in live (online) mode, e.g., video streaming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer **QFfh** for the positive feedback and constructive suggestions. We appreciate your recognition of the AeroEye dataset and CYCLO approach. We will carefully address your suggestions regarding inference cost and live video streaming applications.\n\n**Q1: It would be great to report the inference cost of the proposed approach.**\n\nIn the table below, we report FPS for models discussed in Section 2.2. CYCLO significantly **outperform these models in both recall and mean recall** with a slight trade-off in FPS.\n| **Method**       | **R/mR@20**       | **R/mR@50**       | **R/mR@100**      | **FPS**  |\n|------------------|-------------------|-------------------|-------------------|----------|\n| **Vanilla**      | 31.04 / 11.20      | 34.28 / 11.27      | 34.62 / 12.43      | **19.5** |\n| **Transformer**  | 41.09 / 11.88      | 46.52 / 12.31      | 47.15 / 12.91      | 17.8     |\n| **HIG**          | 37.28 / 11.98      | 38.59 / 13.12      | 39.27 / 13.29      | 15.3     |\n| **CYCLO**        | **59.59** / **13.29**  | **60.37** / **13.69**  | **43.53** / **13.86**  | 14.2     |\n\n\n**Q2: It is unclear how the proposed solution works in live (online) mode, e.g., video streaming.**\n\nCYCLO **online** processes video streams frame-by-frame, **continuously updating the scene graph** to reflect the latest object relationships. As new frames arrive, CYCLO dynamically adjusts object relationships and refines the scene graph based on the most recent data while leveraging historical context.\n\n**Q3: In the ablation study (line 265), when a frame is discarded, is it something like that the frame is set to some random noise?**\n\nNo, we **discard one frame out of every two successive frames** rather than replacing them with random noise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sqLwVsJKK", "reviewer": "Reviewer_5Gaf", "summary": "This work focuses on the VidSGG task. Specifically, it built a new UAV-based VidSGG dataset named AeroEye, and also further propose a CYCLO approach for using cyclic attention over the VidSGG task.", "questions": "Overall, while I tend to accept this work, I still have the following concerns and suggestions w.r.t. the current version of the submission:\n\n(1) It seems to me that, while the dataset focuses on the UAV scenario, the direct and long-range temporal relationships focused by the proposed method is not closely related to the UAV scenario. I appreciate if the connection (if any) can be more clearly explained.\n\n(2) Besides comparing with those existing UAV datasets, I suggest the authors to also include a statistical discussion (in a table format maybe) between AeroEye and those existing VidSGG datasets.\n\n(3) In line 43, the authors claim that \"They [5, 19] usually struggle with long-term dependencies due to the diminishing influence of inputs over time.\" This claim should be made with better support.\n\n(4) Moreover, I believe that the connectivity between the proposed CYCLO method and scene graph needs to be better discussed. This is important since the usage of \"cyclic attention\" solely in the video context seems to be not a new stuff already (e.g., Transformer Tracking with Cyclic Shifting Window Attention CVPR 2022). Thus, for authors to clearly indicate the novelty of their CYCLO method, from my perspective, it is necessary to (a) discuss how CYCLO is designed for scene graph more clearly and (b) discuss the difference between CYCLO and existing cyclic attention methods in the video context.", "limitations": "The authors have discussed their limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. From my perspective, a high quality dataset and large scale dataset is of critical imporantance over the VidSGG area. I thus appreciate the effort by the authors.", "weaknesses": "(See the questions section below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer **5Gaf** for the constructive feedback. Your suggestions on clarifying our method's UAV context, dataset comparisons, and CYCLO's scene graph relation have greatly improved this paper.\n\n**Q1: It seems to me that, while the dataset focuses on the UAV scenario, the direct and long-range temporal relationships focused by the proposed method is not closely related to the UAV scenario.**\n\nOur intention developing CYCLO **does not limit to UAV scenario**, but demonstrates **effectiveness across in-the-wild datasets** like OpenPVSG and ASPIRe as in experiments. The long-range temporal relationships crucial in UAV footage, such as tracking object movements over large areas and extended timeframes, are equally important in ASPIRe's street-level scenes. Here, understanding prolonged interactions between people and vehicles requires similar temporal analysis.\n\n**Q2: Comparison between AeroEye and existing VidSGG datasets.**\n\nWe've included a statistical comparison of AeroEye with both UAV and non-UAV VidSGG datasets in **Table A.11 of the Appendix**. This comparison will be moved to the main paper in the revision.\n\n**Q3: In line 43, the authors claim that \"They [5, 19] usually struggle with long-term dependencies due to the diminishing influence of inputs over time.\" This claim should be made with better support.**\n\n[5] employed **Transformers** with self-attention for modeling long-term dependencies in videos. However, Transformers lack inherent **temporal order** [A], and their attention mechanism struggles with **distant elements in long sequences** [B]. [9] utilized a **hierarchical graph-based** approach to model long-term dependencies by representing the video as a sequence of graphs, each capturing the evolving relationships among objects at different temporal and spatial scales. This method integrates temporal and spatial information by constructing and updating node and edge features at multiple hierarchical levels. However, this approach requires **observing the entire video** to build comprehensive hierarchical representations, which poses significant challenges for real-time or online processing.\n\n**Q4: Discuss how CYCLO is designed for scene graph more clearly.**\n\nCYCLO is specifically designed for video scene graph generation, building evolving graphs where nodes represent objects and edges represent relationships. The construction process involves:\n1. CYCLO first identifies objects in each video frame, creating nodes for each detected object.\n2. It then infers relationships between objects, establishing edges between nodes to represent these relationships.\n3. CYCLO employs a temporal graph transformer with cyclic attention across the video sequence. It processes the graph structure over time, updates node and edge features based on spatial and temporal context, and uses cyclic attention to connect information from the start to the end of the video.\n4. CYCLO cycles the attention mechanism between relationships over time. This process allows it to maintain continuity in object relationships across frames and enhance understanding of long-term dependencies. It also helps to recur patterns in object interactions and connect information from distant parts of the video.\n5. CYCLO integrates spatial information, such as object locations and current interactions, with temporal data, including past interactions and object trajectories, to preserve directional context. It also maintains historical context and learns how initial interactions influence subsequent actions.\n6. As the video progresses, CYCLO continuously updates the scene graph, refining object states, relationships, and the overall structure based on new information from each frame.\n\n**Q5: Discuss the difference between CYCLO and existing cyclic attention methods in the video context.**\n\nThanks for this suggestion. We will include this MS-CSWA [C] and its description in our final version. CYCLO differs from MS-CSWA [C] in several important ways:\n1. Spatio-temporal integration:\n   - CYCLO combines **spatial and temporal** dependencies in a graph-based framework, allowing dynamic refinement of spatial relationships over time.\n   - MS-CSWA focuses on maintaining **spatial** consistency within individual frames for object tracking but lacks temporal depth across frames.\n2. Operation:\n   - CYCLO uses **cyclical indexing and shifting** to ensure inter-frame temporal continuity, modeling how past interactions influence present and future relationships.\n   - MS-CSWA enhances spatial attention through only **cyclic shifts** but does not address temporal depth or graph-based updates across frames.\n3. Long-term dependencies:\n   - CYCLO captures **long-term dependencies and evolving interactions across frames**, preserving directional and historical information.\n   - MS-CSWA is limited to intra-frame consistency and is **unable to model long-term dependencies** across the video.\n\n**References**\n\n[A] Truong, T. D., Bui, Q. H., Duong, C. N., Seo, H. S., Phung, S. L., Li, X., & Luu, K. (2022). Direcformer: A directed attention in transformer approach to robust action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 20030-20040).\n\n[B] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.\n\n[C] Song, Z., Yu, J., Chen, Y. P. P., & Yang, W. (2022). Transformer tracking with cyclic shifting window attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8791-8800).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable feedback. Reviewers **nRfD** and **QFfh** recommend acceptance, praising our CYCLO approach to multi-object relationship modeling, the AeroEye dataset, and the versatility of our approach. Reviewer **5Gaf** leans towards acceptance with a **Borderline Accept**. Reviewer **Zf4k** suggests a **Borderline Reject**, noting to explain more details. We have updated the typos in our paper. We will clarify how our CYCLO approach applies to video scene graph generation with an illustration of the overall framework in the rebuttal PDF file. Individual responses for each reviewer are included below for specific concerns.

Meta-review: The paper received generally positive reviews during the initial review period. The reviewers appreciated the dataset as a valuable resource for the community and the quantitative performance of the proposed baseline. While some concerns were raised about limited discussion on the annotation procedure, discussion on lower mR@K scores under less pronounced in a long-tailed distribution setting, and computational complexity. The authors' response addressed these concerns, and I agree with the reviewers and recommend acceptance. The authors are asked to include the details from the rebuttal in the camera-ready version and make the dataset publicly available as recommended in the [NeurIPS 2024 Code and Data guidelines.](https://neurips.cc/public/guides/CodeSubmissionPolicy)
Decision: Accept (poster)