CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos

Authors: trong-thuan  nguyen
pha  nguyen
xin  li
jackson  cothren
alper  yilmaz
khoa  luu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 22:40:05

Abstract: Video scene graph generation (VidSGG) has emerged as a transformative approach to capturing and interpreting the intricate relationships among objects and their temporal dynamics in video sequences. In this paper, we introduce the new AeroEye dataset that focuses on multi-object relationship modeling in aerial videos. Our AeroEye dataset features various drone scenes and includes a visually comprehensive and precise collection of predicates that capture the intricate relationships and spatial arrangements among objects. To this end, we propose the novel Cyclic Graph Transformer (CYCLO) approach that allows the model to capture both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The proposed approach also allows one to handle sequences with inherent cyclical patterns and process object relationships in the correct sequential order. Therefore, it can effectively capture periodic and overlapping relationships while minimizing information loss. The extensive experiments on the AeroEye dataset demonstrate the effectiveness of the proposed CYCLO model, demonstrating its potential to perform scene understanding on drone videos. Finally, the CYCLO method consistently achieves State-of-the-Art (SOTA) results on two in-the-wild scene graph generation benchmarks, i.e., PVSG and ASPIRe.

Main paper: 
 introduction: 
 Visual scene understanding has shown significant progress in extracting semantic information from images and videos using deep learning algorithms [1, 2, 3] . Building upon the significant progress in visual scene understanding using deep learning algorithms [4, 5] , video scene graph generation (VidSGG) extends the concept of Scene Graph Generation (SGG) from static images to dynamic video, representing object relationships within a graph structure that evolves over time. VidSGG [6, 7, 8] focus on the temporal dimension by constructing a dynamic graph structure, encapsulating the spatial and temporal relationships among object interactions across frames. This helps in understanding human-object interactions [9, 10] , temporal events [11, 12, 13] , and reasoning [14, 15] . However, drone-captured videos present unique challenges due to larger image sizes and higher object density in Unmanned Aerial Vehicle (UAV) datasets [16, 17, 18] . Despite recent advances in tiny object detection [19, 20, 21] , current algorithms still need to effectively model object interactions and their temporal evolution in aerial videos, which have various applications in surveillance, disaster response.In this paper, we introduce AeroEye, the first dataset for Video Scene Graph Generation in dronecaptured videos featuring Aerial-Oblique-Eye views. AeroEye distinguishes itself by showcasing a rich tapestry of aerial videos and an extensive set of predicates describing the intricate relations and positions of multi-objects. To address multi-object relationship modeling in aerial videos from the AeroEye dataset, we propose the Cyclic Graph Transformer (CYCLO). This new approach can establish circular connectivity among frames and enables the model to capture direct and long-range temporal relationships. By continuously updating history across a ring topology, CYCLO allows the model to handle sequences with inherent cyclic patterns, facilitating the processing of object relationships in the correct temporal order. Furthermore, CYCLO provides several advantages to VidSGG, including the ability to model periodic and overlapping relationships, predict object interactions by reasoning from previous cycles, facilitate information transfer across frames, and efficiently utilize long sequences, addressing the limitations of prior methods [22, 8] . They usually struggle with long-term dependencies due to the diminishing influence of inputs over time.There are three main contributions to this work. First, we introduce a new AeroEye dataset for VidSGG in drone videos, augmented with numerous predicates and diverse scenes to capture the complex relationships in aerial videos. Second, we propose the CYCLO approach, utilizing circular connectivity among frames to enable periodic and overlapping relationships. It allows the model to capture long-range dependencies and process object interactions in the appropriate temporal arrangement. Finally, the proposed CYCLO approach outperforms prior methods on two large-scale in-the-wild VidSGG datasets, including PVSG [7] and ASPIRe [8] . Interestingly, using the same method (e.g., our CYCLO), the ratio of correct predictions to incorrect predictions (R/mR) on AeroEye is higher than PVSG (Tables 4 and 6 ), despite having more predicates (Table 1 ) and tiny objects. This suggests that our dataset is less visually ambiguous than PVSG. 
related work: 
 In this section, we review the existing datasets and benchmarks for Visual Scene Graph Generation, followed by a summary of the key challenges and issues related to Video Scene Graph Generation.Datasets. VisDrone [17] , DOTA [16] , and SODA-A [36] image datasets, along with UAVid [46] , UAVDT [37] , and MAVREC [18] video datasets, offer high-resolution UAV datasets that enable precise object detection in dynamic scenes. While these UAV datasets focus on object detection, the Visual Genome [23] pioneered image-based SGG, and the Action Genome [6] dataset extended this concept to capture dynamic interactions within videos. Recently, ASPIRe [8] and SportsHHI [29] emphasize diverse human-object relationships and sports-specific player interactions. Additionally, PSG-4D [47] expands the VidSGG to encompass the 4D domain, bridging the gap between raw visual data and high-level understanding. In Table 1 , we present a comparative overview of UAV-based and SGG datasets for images and videos, emphasizing their unique characteristics and advantages.Benchmarks. The existing benchmark focuses on Image Scene Graph Generation (ImgSGG) and Video Scene Graph Generation (VidSGG). ImgSGG identifies and categorizes relationships between objects within an image into predefined relational categories, including Transformer-based EQUATIONmethods [27, 48, 49, 50] and generative-based models [51, 52, 53] . VidSGG leverages the dynamic nature of object interactions over time to better identify relationships, as the temporal dimension of videos provides a richer context for understanding semantic connections within the scene. Current methods using hierarchical structures [54, 8] or Transformer architectures [22, 55, 56, 57] excel at capturing long-range dependencies and complex interactions, advancing video understanding in video captioning [11, 12, 13] , visual question answering [14, 15] , and video grounding [58, 59, 60] .VidSGG can be categorized into two main types based on the granularity of its graph representation.Video-level SGG represents object trajectories as graph nodes, capturing constant relations between objects for a video. Various methods have been proposed to address this problem, incorporating Conditional Random Fields [61] , abstracting videos [62] , and iterative relation inference techniques [63] on fully connected spatio-temporal. However, focusing primarily on recognizing video-level relations directly based on object-tracking [64, 65, 66] results and neglecting frame-level scene graphs results in a cumbersome pipeline highly dependent on tracking accuracy. In contrast, Frame-level SGG defines the graph at the frame level, allowing relations to change over time. The releases of the benchmark datasets [67, 7, 8] have prompted the development of VidSGG models. TRACE [54] , for instance, employs a hierarchical relation tree to capture spatio-temporal context information, while CSTTran [22] uses a spatio-temporal transformer to solve the problem. Recently, hierarchical interlacement graph (HIG) [8] abstracts relationship evolution using a sequence of hierarchical graphs.In this subsection, we conceptually compare our proposed approach with relationship modeling concepts discussed in Section 2.2 as illustrated in Fig. 2 . In addition, we highlight the advantages of our approach and discuss the properties that distinguish it from these existing methods.Concepts. The progressive approach fuses pairwise features between object pairs at each frame, encoding the object relationship at that specific step, followed by a fully connected layer to classify the predicate types. However, it processes frames independently without considering the temporal context. The batch-progressive approach employs a transformer block with positional embeddings on the fused query features. The hierarchical approach represents a video as a sequence of graphs, integrating temporal and spatial information at different levels. The node and edge features are updated at each hierarchical level based on the previous level to capture evolving object relationships.Limitations. While the batch-progressive approach considers temporal information, both these progressive and batch-progressive approaches have limitations in modeling the full complexity of temporal dynamics and dependencies in the video. In contrast, the hierarchical graph approach can capture complex interactions and relationships between objects by considering the temporal evolution of graphs at different granularity levels. However, the hierarchical graph requires analyzing the entire video before constructing the graph (i.e. offline method). These limitations underscore the need for more advanced approaches to efficiently model temporal dynamics, adaptively update memory to handle evolving video data, and accurately capture the intricate relationships between objects.Advantages of Our Design. Inspired by previous work [68, 69] , which processes temporal features through iterative feedback loops and circular updating, we propose the CYCLO approach that circularly incorporates an updated history of relationships. In contrast to these methods, which focus on frame-level updates influenced by global features, our approach constructs and refines scene graphs for each frame, capturing static spatial relationships between objects and their dynamic evolution over time. By leveraging circular connectivity, CYCLO establishes a continuous loop of temporal information, ensuring no temporal edge is treated as a boundary. It enables the Transformer to operate online and then capture and update relationships between objects more effectively, correcting erroneous connections. The theoretical properties are included in Section B of Appendices.We employ a custom-designed filter to categorize the generated text into relationship types. This filter utilizes a combination of keyword matching, dependency parsing, and semantic analysis to identify and classify the predicates accurately. The filter is designed to handle sentence structure and terminology variations, ensuring that the identified predicates are correctly mapped to their corresponding relationship types. Furthermore, the filter incorporates a confidence scoring mechanism to prioritize high-quality predicates and filter out irrelevant or ambiguous ones. The final selection of predicates undergoes human oversight, where experienced annotators review and validate the filtered results. This manual verification step ensures the highest accuracy and relevance of the identified predicates, mitigating potential errors introduced by automated processing.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the proposed aeroeye dataset: 
 In this section, we detail the AeroEye dataset annotation process and provide the dataset statistics.Data Preparation. We leverage videos from the ERA [40] and MAVREC [18] datasets to construct our AeroEye dataset. ERA consists of diverse scenes ranging from rural to urban environments in extreme conditions (e.g. earthquake, flood, fire, mudslide), daily activities (e.g. harvesting, plowing, party, traffic collision), and sports activities (e.g. soccer, basketball, baseball, running, swimming). MAVREC features sparse and dense object distributions and contains typical outdoor activities characterized by many vehicle classes, incorporating viewpoint changes and varying illumination. Relationship Classes and Instance Formulation. In Fig. 3 , we focus on two aspects of object relationships: positions (e.g. in front of, behind, next to) and relations, which consist of movement actions (e.g. chasing, towing, overtaking) and collision actions (e.g. hitting, crashing, colliding). These relationships are semantically complex and require detailed spatio-temporal context reasoning for recognition. Following previous PVSG benchmark datasets [6, 7, 8] , we define relationship instances at the frame level, considering the long-term spatial-temporal context. Each instance is formulated as a triplet <s, o, p>, where s and o denote the bounding boxes of the subject and object, and p represents the predicate (i.e. position and relation), included in Tables A.8, A.9 which are summarized in Fig. 4 . In addition, Fig. A.11 presents selected samples from our AeroEye dataset. Data Annotation. We annotate keyframes at 5FPS to capture frequent and rapid changes in positions and relations in aerial videos, reducing redundancy while keeping up with interaction changes. Our two-stage annotation pipeline first performs object localization and tracking and then relationship instance annotation. To generate diverse predicates, we leverage the GPT4RoI [70] model, which combines visual and linguistic data to generate detailed descriptions of object relationships within specified regions of interest. Although we annotate relationship instances frame by frame, as illustrated in Fig. 3 , we easily create relationship tubes using the provided object tracking ID by connecting the same pair of objects with the same relationship predicate across consecutive frames. The annotation file includes object information (i.e., bounding boxes, category names, and tracking IDs) and relationships within each frame. Details on the quality control process and annotation examples can be found in Section A.2 of the Appendices. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: While this paper does not include purely theoretical results, it does incorporate derivations of relevant formulas, along with their respective assumptions and complete proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
the proposed approach: 
 In this section, we present our CYCLO approach, including the Spatial Attention Graph and the Cyclic Temporal Graph Transformer. The Spatial Attention Graph captures spatial dependencies and interactions between objects within the frame. In contrast, the Cyclic Temporal Graph Transformer models temporal relationships across frames, capturing short-term and long-term dynamics.Given an input video with T frames, we construct dynamic scene graphs {G t } T t=1 that encode the relationships among objects within these frames. Each graph G t (V t , E t ) captures static relationships, where node V consists of objects and edge E denotes the relationship between objects. Each objectv i ∈ V has an object category v c i ∈ C v and box coordinates v b i ∈ [0, 1] 4 .Each relationship e j ∈ E represents the j-th triplet (s j , o j , p j ), where subject s j and object o j and predicate p j ∈ C p .Self-attention mechanisms in one-stage object detectors [71, 19] model relationships between objects, allowing insights into the dynamics between entities without relying on additional contextual information. For example, in an aerial parking lot video with cars, vans, and people, if the self-attention layer strongly connects the queries representing the person and the car, it suggests an interaction, such as the person entering the vehicle. Inspired by [50] , in our CYCLO approach, to construct the static graph in each frame t, we utilize the DETR decoder to establish bidirectional connections among object queries. In particular, we compute relational representations at each layer l by concatenating the query and key vectors, Q l t and K l t , for every object pair. This process ensures the layer l -1 output seamlessly transitions as input to layer l. We omit the superscript L related to the final layer to simplify the following discussion. At each frame, R l a,t captures the dynamic interplay of relations at layer l, utilizing their query and key vectors. Furthermore, R z,t leverages object queries in the final layer for object detection. These relationships are formally defined in Eqn. (1) .EQUATIONHere, ϕ Ws and ϕ Wo are the linear transformations that process subject and object features, enabling the model to consider both object characteristics and their interrelationships comprehensively. In addition, gating mechanisms g l a,t and g z,t dynamically modulate the contributions from different layers. These gated representations from all layers are then integrated to construct a relation matrix:EQUATIONwhere ϕ W G denotes the linear weight applied during the gating process. Finally, the relation matrix is fed into the three-layer perception (MLP) with ReLU activation and a sigmoid function σ, which predicts multiple relationships (p j ) between pairs of objects (s j , o j ). Mathematically, G t = σ(MLP( R t )) is the graph representation at frame t-th, where G t ∈ R N ×N ×|Cp| .Discussion. Transformer-based approaches to VidSGG effectively capture interactions and temporal changes through self-attention mechanisms, creating detailed scene graphs that reflect video dynamic relationships. However, these models often struggle to represent the directional and historical aspects of the relationship accurately. While effective at identifying token correlations, the scaled dot-product fails to consider their temporal or spatial ordering. This oversight is particularly critical in videos, where understanding the historical context of relationships is essential. For example, the sequence of relationships leading up to a car crash, including speeding, lane changing, and passing, must be considered. Each interaction change provides crucial historical information that contextualizes the final relationships, vital for enhancing prediction and interpretation. In addition, traditional selft-attention [72] does not adequately capture this essential sequential, directional, and historical information [73] . It highlights the need for advancements in transformer architectures to more effectively integrate the direction and historical sequence of interactions within videos.We present the Cyclic Spatial-Temporal Graph Transformer to refine the spatial attention graph in each scene, capturing temporal dependencies via subject-object relationships across adjacent frames.Cyclic Attention. As mentioned in Section 4.2, self-attention does not adequately capture directional and historical information. Therefore, we propose the cyclic attention (CA), defined as in Eqn. (3) . In Eqn. ( 3), η is a shift term enabling cyclical indexing via mod T . The cyclical indexing, illustrated in Fig. 6 , allows for continuous sequence processing by connecting the end to the beginning, which is crucial for predicting movements in dynamic interactions where past events influence future actions (e.g. a car navigating a roundabout). Cyclical indexing differs from standard self-attention as it is a permutation equivariant without positional encodings. In contrast, cyclical attention is non-permutation equivariant, which depends on the original sequence order. This property is crucial for multiobject relationship modeling, where maintaining the chronological order of interactions is essential. For instance, in a surveillance scenario, the sequence of a car stopping for a pedestrian must preserve the order of events, ensuring the vehicle stops before the pedestrian appears.EQUATIONTemporal Graph Transformer. Our Temporal Graph Transformer refines spatial attention graphs, { G t } T t=1 , into a sequence of dynamic graphs {G t } T t=1 , leveraging the temporal dynamics and spatial interactions of objects across video frames. Our approach employs a series of cyclic attention blocks configured within multi-head attention to refine object representations by integrating features from adjacent frames. The core of our approach is the integration of cyclic attention into a multi-head structure, which processes the sequence of input features Z = { Z t } T t=1 , represented as in Eqn. (4) .EQUATIONwhere ϕ Wc , ϕ W i q , and ϕ W i k denote the linear transformations. Each head h i (t) computes the cyclic attention, integrating information across the video to enhance the temporal relationship at each frame. The outputs from various heads at each frame are integrated into Z ′ t , derived from concatenating all attention head outputs. These heads process features cyclically across different representation subspaces to capture the temporal evolution of relationships in the video. Then, Z is obtained by applying layer normalization (LN) and a skip connection to the aggregated features Z ′ t , where Z = LN(Z ′ + Z). This step ensures that Z ′ t is stabilized and effectively integrated with the original features Z, thus dynamically updating the scene graph and ensuring temporal coherence.In addition, Z t is utilized to construct a new relation matrix R t by applying the transformations in Eqn. ( 1) and ( 2). This updated matrix R t refines the relationship dynamics captured in static frames by correcting spurious or incomplete relationships and incorporating previously omitted ones using the temporal context from the frame sequence. As a result, G t comprehensively represents persistent and transient interactions, including their direction and historical sequence within the video.Loss Function. Visual object relationships involve predicates that may appear quite similar, such as "parking next to" and "stopping next to". Thereby, we utilize a multi-label margin loss, as in [22] :EQUATIONIn Eqn. ( 5), r represents a subject-object pair, and P + and P -correspond to positive and negative predicates, respectively. The term ϕ(r, p) measures the compatibility of the pair r with the predicate p. Additionally, object distributions are modeled using neural networks with ReLU activation and batch normalization. Cross-entropy loss is applied during the learning process. The total loss is a combination of the multi-label margin loss L p and the cross-entropy loss L ce , defined as:EQUATION)where λ is a weight balancing the contribution of the cross-entropy loss L ce .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Please see the Subsection 5.1. 
experimental results: 
 In this section, we discuss the benchmark dataset evaluations and comparisons with SOTA methods.Dataset. We use 10-fold cross-validation on the AeroEye dataset, including 1,797 videos for training and 463 videos for testing. We also evaluate our performance on PVSG [7] and ASPIRe [8] datasets.Settings. We employ DINO [19] to extract the spatial attention graphs (in Section 4.2). DINO is trained with ResNet-50 backbone and 1500 queries on MAVREC, achieving 92.35 mAP on the validation set. The pre-trained detector is applied to baselines, and parameters are fixed during subsequent task training. Our model is trained on 8 × A6000 GPUs using 12 epochs with AdamW optimizer (initial learning rate of 1e -5 and a batch size of 1), gradient clipping (max norm of 5).Evaluation Metrics. We evaluate models on two standard tasks in image-based scene graph generation followed by previous work [74, 7] that are predicate classification (PredCls), scene graph classification (SGCls), and scene graph detection (SGDet). While SGCls predicts relationships given ground truth objects, SGDet involves detecting objects and predicting relationships. These tasks are evaluated using Recall (R@K) and mean Recall (mR@K), where K ∈ {20, 50, 100}.Semantic Dynamics in Cyclic Attention. By altering η (in Eqn. ( 3)), we consider the permutation or non-equivariance equivariance. If the predictions systematically adapt to the shifts induced by Cyclic Dependency. To further validate the cyclic dependency of our model, we discarded frames from every successive frame. Removing frames disrupts the temporal continuity of the sequence, which is crucial for maintaining the the cyclic nature of video. If the cyclic model presupposes that each frame has direct relationships with its adjacent frames circularly, then removing frames could sever these relationships, potentially diminishing the ability to leverage cyclical patterns effectively. Indeed, Table 3 informs a decrease in Recall and mean Recall when frames were reduced.Table 4 shows CYCLO outperforms other methods across metrics and tasks, surpassing Transformer by 2.95%, 3.03%, and 2.44% in the PredCls, SGCls, and SGDet at R@20, and maintaining its lead even at higher Recall thresholds. Moreover, Table 5 shows significant improvements at mR@20, mR@50, and mR@100, with CYCLO leading HIG by 0.86% in the PredCls task at mR@20 and exceeding HIG by 1.12% and 1.32% in the SGCls and SGDet tasks, respectively, at mR@20. In addition, the consistent performance and low standard deviation of the CYCLO model across Table 4 and 5 demonstrate its robustness and overall superiority. Fig. 7 displays that CYCLO can capture the evolving relationships between objects by updating their positions and interactions.We compare our performance on two recent benchmark datasets on VidSGG (i.e. PVSG and ASPIRe). Performance on PVSG. In Table 6 , we report that the CYCLO approach achieves superior performance compared to other models on the PVSG dataset, particularly in terms of Recall and mean Recall metrics. In particular, CYCLO surpasses HIG and Transformer by 1.23% and 1.81% at R@20, respectively. In addition, it consistently outperforms both models at higher recall rates, including R@50 and R@100, showcasing its effectiveness across various thresholds. Our approach also slightly improves the mean Recall, as the experimental results demonstrate. These results highlight the robustness of our CYCLO approach in recognizing and handling periodic actions, such as cooking, washing, cleaning, exercising, and other routine tasks frequently represented on the PVSG dataset.Performance on ASPIRe. The ASPIRe dataset includes five distinct interactivity types. However, we focus on position and relation to ensure a fair comparison. As shown in Table 7 , CYCLO consistently outperforms existing models across multiple recall and mean recall thresholds. Notably, at R@20, our CYCLO approach outperforms the HIG method, the second-best model, by 0.69% in position and a more substantial 5.03% in relation. Additionally, CYCLO shows significant gains in mean Recall across all evaluated thresholds, demonstrating its effectiveness in tackling the long-tail distribution.• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusions: 
 We have introduced CYCLO, a novel approach that effectively captures periodic and overlapping relationships, handles extended sequences, and minimizes information loss, making it suitable for complex temporal modeling. In addition, we presented AeroEye, a comprehensive and diverse dataset composed of drone-captured scenes, specifically designed to represent intricate object relationships and spatial positions in aerial videos. Through extensive experiments on the AeroEye dataset and two large-scale in-the-wild datasets (i.e. ASPIRe and PVSG), we demonstrated the robustness and effectiveness of CYCLO in capturing dynamic interactions and evolving relationships over time.Limitations. Although our CYCLO approach has achieved impressive performance, it may reveal limitations when dealing with incomplete or discontinuous videos. The periodic and cyclic attention mechanisms, crucial for capturing temporal and spatial object relationships, heavily rely on video continuity and completeness. Interruptions in the sequence, such as missing or discontinuous frames, disrupt the formation of accurate cyclical references, leading to inconsistent and incorrect predictions.Broader Impacts. The proposed approach improves the capture of object interactions and temporal evolution in aerial and in-the-wild videos, which is critical for surveillance, disaster response, traffic management, and precision agriculture applications. By modeling object interactions over time, CYCLO supports more informed decision-making, leading to safer and more sustainable practices. This advancement opens the door for future work developing surveillance systems that can model complex relationships from drone videos. However, it is important to recognize the potential risks associated with this approach, particularly the possibility of using it for unauthorized surveillance. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Please see Broader Impacts in Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original papers that produced the code or dataset are cited properly, including authors, title, publication venue, year, and a link if available.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Relationship DefinitionOur AeroEye dataset focuses on capturing the spatial positions and relationships between objects, including person-to-object and object-to-object interactions. Table A.8, A.9 provides a comprehensive list of the defined positions and relationships. Furthermore, Table A .10 offers guidance on the final relationship vocabulary specific to each scene category within the dataset. To capture frequent and rapid changes in aerial videos while reducing redundancy, we annotate keyframes at 5FPS. At each frame, our annotation pipeline consists of two stages:Stage 1: Object Localization and Tracking. We manually annotate bounding boxes with predefined categories, providing precise object localization and consistent tracking throughout the video.Stage 2: Relationship Instance Annotation. To generate diverse predicates, we leverage the GPT4RoI [70] model, which combines visual and linguistic data to generate detailed descriptions of object relationships within specified regions of interest. The process involves the following steps: [75] and Visual Genome [23] , to refine the alignment between visual and linguistic data, ensuring precise responses to spatial queries.In particular, We input bounding boxes around objects with prompts such as, "What is the relationship between <object_1> in <region_1> and <object_2> in <region_2>?", where <object_i> is a category name that labeled in Stage 1, and GPT4RoI replaces <region_i> tags in these instructions with results from RoIAlign, derived directly from the features of image. The model uses RoIAlign to extract region-specific features and combine them with language embeddings. The resulting multimodal embeddings are then interpreted by the Vicuna model [76] , an instance of LLaMA [77] .To maintain the highest standard of annotation quality, we implement the following comprehensive measures:• Stage 1: Object Localization and Tracking:-All bounding box annotations are performed manually by skilled annotators without relying on automated detection or tracking models. This ensures precise object localization tailored to the specific characteristics of aerial videos.-We employ a rigorous double-checking process, where each frame in every video is carefully reviewed by a second annotator. This step helps identify and rectify any inaccuracies in bounding box placement or dimensions. -In cases where object identities are inconsistent across frames due to occlusion, visual similarity, or other challenges, annotators meticulously correct the object numbers to maintain consistent tracking throughout the video. • Stage 2: Relationship Instance Annotation:-Annotators undergo extensive training using carefully curated examples from previous VidSGG datasets. This training familiarizes them with the intricacies of extracting predicates generated by the LLM and ensures a deep understanding of the annotation guidelines and best practices. -To minimize individual biases and ensure the robustness of annotations, we implement a repeated annotation process. Each video is distributed to multiple annotators, who independently extract and record the relationship instances. This redundancy allows for cross-validation and helps identify potential discrepancies or ambiguities. -In cases where annotators disagree on the extracted predicates or their categorization, a highly experienced meta-annotator is assigned to review the conflicting annotations.The meta-annotator carefully examines the video content, considers the perspectives of the individual annotators, and makes the final decision on the annotation record. This hierarchical review process ensures consistency and accuracy across the dataset.By employing these rigorous quality control measures at each stage of the annotation pipeline, we ensure the highest level of accuracy, consistency, and completeness in relationship instances.Our annotations are stored in JSON (JavaScript Object Notation) format organized as below:1 data [ {" file_name " : str ," height " : int ," width " : int ," image_id " : int ," frame_index " : int , Basic Image Information. This section contains the fundamental attributes of each image:• file_name (str): The name of the image file.• height (int): The height of the image in pixels.• width (int): The width of the image in pixels.• image_id (int): A unique identifier for the image.• frame_index (int): The index of the frame within the video sequence.• video_id (int): An identifier for the video to which this image/frame belongs.Metadata. This section includes the metadata key, which is a list of segments within the image. Each segment contains:• id (int): Unique identifier for the segment.• category_id (int): Identifier for the category of the object in the segment.• iscrowd (0 or 1): 0 for a single object and 1 for a cluster of objects.• area (int): The area covered by the segment in the image.The annotations key contains a list of corresponding bounding boxes for each entry in metadata, each tagged with a specific category_id:• bbox (list): [x_center, y_center, width, height] of the bounding box.• bbox_mode (0 or 1): Bounding box mode.• category_id (int): Identifier for the object category in the bounding box.• track_id (int): Identifier to track the bounding box across different frames.Relationship Attributes. This section encompasses lists of positions and relations for each segment, including two different metadata_ids to represent the interactivity between two segments:• positions (list): List of position relations between segments, each containing:-metadata_id (int): Identifier for the first segment.-metadata_id (int): Identifier for the second segment.-position_id (str): Identifier for position relation between the segments.• relations (list): List of other relations between segments, each containing:-metadata_id (int): Identifier for the first segment.-metadata_id (int): Identifier for the second segment.-relation_id (str): Identifier for relationship between the segments. These descriptors represent lists specifying various subject, object, and interactivity aspects for each bounding box within the annotations and metadata. For example, [3, 0, 5] indicates that the third and first metadata segments share the relationship with an ID of 5. We present object, position, and relationship statistics per category in Fig. A.8, A.9, and A.10. . This multifaceted approach to annotation renders AeroEye uniquely comprehensive when compared to other datasets [6, 27, 7, 8] . The meticulous annotation process and the meticulously curated, information-rich nature of the AeroEye dataset firmly establish it as an invaluable resource, poised to catalyze significant advancements in relationship modeling within drone videos.Relationship Representation. In each frame, a detector provides to a set of object features { v 1 t , . . . , v the detected objects. These elements are used to define the relationships x k t :x k t = ϕ Ws z t i , ϕ Wo z t j , ϕ Wu φ( u ij t ⊙ f ( b i t , b j t )), s i t , s j t (B.7)where [•, •] denotes concatenation, φ denotes a flattening operation, ⊙ represents element-wise addition, and the linear transformation matrices ϕ Ws , ϕ Wo , and ϕ Wu . u t ij implies the feature map of the union box, computed by the RoIAlign [78] , while f is a function that converts the bounding boxes of the subject b t i and object b t j into a feature map with the same dimensions as u t ij . The semantic embedding vectors s t i and s t j are obtained from the object categories c t i and c t j , respectively. In the progressive approach, features v i t for each object are obtained using a detector such as Faster R-CNN in each frame. These features are utilized as specified in Eqn. (B.7) to compute the relationship features x k t . Finally, these features are processed through multilayer perceptions (MLPs) followed by softmax activation functions to classify the types of relationships between objects.In contrast to the Progressive approach, the Batch Progressive approach passes the relationship features through a Transformer architecture before classification.t = { x 1 t , x 2 t , . . . , x K(t) t} is fed into the Transformer Encoder, which focuses on understanding the spatial context by inputting these relationship features into a sequence of identical self-attention layers, defined as:X (n) t = Att enc. (Q = K = V = X (n-1) t ) (B.8)Each n-th layer receives the output from the (n-1)-th layer as its input, iteratively refining to enhance the representation of spatial relations embedded in the features, where n denotes the layer number.The outputs are then processed by the Decoder, which captures temporal dependencies between frames, applying a sliding window over the sequence of spatially contextualized representations:Z i = [ X i , . . . , X T -η-1 ], i ∈ {1, . . . , T } (B.9)where η is the window size, and T is number of frames. The positional encoding E f = [e 1 , . . . , e η ] is embedded into the input to maintain sequence order:Q = K = Z i + E f , V = Z i , Z i = Att dec. (Q, K, V ). (B.10)Equation (B.10) enables the decoder to process each batch using self-attention layers, combining relation representations Z i with positional encodings E f .The hierarchical interlacement graph (HIG) method abstracts video content by representing temporal relationships using a multi-level graph structure. Higher-level graph cells encompass broader segments of video frames, allowing the HIG to efficiently capture and model the temporal dependencies and connections across different time scales within the video. In this method, object features v i t from individual frames are spatially and temporally fused to form graph nodes. These nodes are interconnected across successive frames, resulting in a series of interconnected frame-based graphs {G t (V t , E t )} T t=1 , where each G t is defined by its vertices V t and edges E t . As the graph traverses, the total number of graphs decreases, ultimately resulting in a singular graph representing the entire video. This hierarchical graph operates on predefined hierarchical levels L. At each level l, the temporal scope is adjusted to T l = T -l + 1, and a new graph is generated that is specific to that level and timeframe. Therefore, node features within each graph are dynamically updated through the computation and aggregation of messages from adjacent nodes, computed by weight matrices for each level l and node pair ( v i , v j ). This iterative refinement process is applied across all levels, resulting in a consolidated graph structure and updated feature set. Finally, relationship features between each node pair ( v i , v j ) are fused and analyzed to classify the relationships. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines: 


Decision: The paper received generally positive reviews during the initial review period. The reviewers appreciated the dataset as a valuable resource for the community and the quantitative performance of the proposed baseline. While some concerns were raised about limited discussion on the annotation procedure, discussion on lower mR@K scores under less pronounced in a long-tailed distribution setting, and computational complexity. The authors' response addressed these concerns, and I agree with the reviewers and recommend acceptance. The authors are asked to include the details from the rebuttal in the camera-ready version and make the dataset publicly available as recommended in the [NeurIPS 2024 Code and Data guidelines.](https://neurips.cc/public/guides/CodeSubmissionPolicy)

Reviews: review: paper_summary: paper_summary: This paper proposes a video scene graph generation dataset called AeroEye on aerial videos and a framework called cyclic graph transformer to tackle the problem of video scene graph generation. The authors annotated the ERA and MAVREC dataset with keyframes at 5FPS. They manually annotated the frames for bounding box localization for each frames along with the tracking. The relationship annotations were done using a GPT4RoI model. The proposed cyclic graph transformer uses a cyclic attention mechanism which the author claims, are able to capture the direct and long-term temporal dependencies.
strength_weakness: strength_weakness: 1. the paper is very difficult to follow. There are separate discussion sections which somewhat disrupts the flow.
2. line 82 seems incomplete
3.Line 118: 'no temporal edge is treated a boundary' can it not be a disadvantage as well since it does not takes into account for an event boundary?
4. A very brief discussion of annotation procedure should be in the main paper such as manual annotation of bounding boxes and relationship annotations done by GPT4RoI model.
5. For loss function section, it would be better to include  the overall loss equation. Does the object distribution refers to the object detection loss using the DETR?
6. For table 2, you can refer to the shift value term from equation 3. It will be easy to follow. Table 2 refers to the ablation studies 'Semantic Dynamics in Cyclic Attention'. I think this section and the section followed by it should have a detailed explanation.
questions: questions: 1. Did you do evaluation on the 5 keyframes for each videos?
2. point 5 in weakness section
limitations: limitations: yes, the limitations are addressed.
paper_summary: paper_summary: This paper presents a new problem: modeling multi-object relationships from a drone's perspective. To address this, the authors propose the AeroEye dataset and introduce the Cyclic Graph Transformer (CYCLO) method. This method captures both direct and long-range temporal dependencies by continuously updating the history of interactions in a circular manner. The authors not only validate the CYCLO approach on the AeroEye dataset but also test it on the PVSG and ASPIRe datasets, demonstrating the effectiveness of their method.
strength_weakness: strength_weakness: 1. The method proposed in this paper could benefit from a clearer network architecture figure, which would allow everyone to better understand the method presented.
2. Is there an issue with non-differentiability in the Cyclic Attention described in Eq. 3? It would be helpful if the authors could provide further explanation of the Cyclic Attention mechanism.
3. It is suggested to include some analysis of bad cases, which would help future researchers understand from which directions further optimizations can be made.
questions: questions: 1. The current model proposed by the authors shows low mR@K scores on the AeroEye dataset. However, from the figures in the supplementary material, the long-tail distribution of relations in this dataset does not appear to be very severe. What could be causing the low mR@K scores? If possible, I would like to see some typical bad cases.
limitations: limitations: It is recommended that the authors consider the limitations of this paper not only from a technical perspective but also from a societal standpoint. Given that relationship modeling from a drone's perspective may lead to widespread applications in surveillance and potentially significant impacts, this aspect requires special consideration.
paper_summary: paper_summary: This paper tackles an interesting task for understanding video scenes that focuses on modeling object relationships in aerial videos. Specifically, it introduces a new dataset, the AeroEye dataset, and proposes a novel approach, CYCLO, to better model the video object relationships. Experimental results demonstrate the effectiveness of CYCLO. CYCLO also achieves state-of-the-art performance on two scene graph generation benchmarks.
strength_weakness: strength_weakness: - It would be great to report the inference cost of the proposed approach. 
- It is unclear how the proposed solution works in live (online) mode, e.g., video streaming.
questions: questions: - In the ablation study (line 265), when a frame is discarded, is it something like that the frame is set to some random noise?
limitations: limitations: I don't have concerns about the potential negative societal impact of this work.
paper_summary: paper_summary: This work focuses on the VidSGG task. Specifically, it built a new UAV-based VidSGG dataset named AeroEye, and also further propose a CYCLO approach for using cyclic attention over the VidSGG task.
strength_weakness: strength_weakness: (See the questions section below)
questions: questions: Overall, while I tend to accept this work, I still have the following concerns and suggestions w.r.t. the current version of the submission:

(1) It seems to me that, while the dataset focuses on the UAV scenario, the direct and long-range temporal relationships focused by the proposed method is not closely related to the UAV scenario. I appreciate if the connection (if any) can be more clearly explained.

(2) Besides comparing with those existing UAV datasets, I suggest the authors to also include a statistical discussion (in a table format maybe) between AeroEye and those existing VidSGG datasets.

(3) In line 43, the authors claim that "They [5, 19] usually struggle with long-term dependencies due to the diminishing influence of inputs over time." This claim should be made with better support.

(4) Moreover, I believe that the connectivity between the proposed CYCLO method and scene graph needs to be better discussed. This is important since the usage of "cyclic attention" solely in the video context seems to be not a new stuff already (e.g., Transformer Tracking with Cyclic Shifting Window Attention CVPR 2022). Thus, for authors to clearly indicate the novelty of their CYCLO method, from my perspective, it is necessary to (a) discuss how CYCLO is designed for scene graph more clearly and (b) discuss the difference between CYCLO and existing cyclic attention methods in the video context.
limitations: limitations: The authors have discussed their limitations.

Comments: comment: Dear Reviewer **QFfh**,

The reviewer-author discussion deadline is nearing. We have yet to receive your final responses to our rebuttal. If you have any further questions, please let us know. We appreciate your invaluable input.

Best regards,

Authors
Dear Reviewer **Zf4k**,

The reviewer-author discussion deadline is nearing. We have yet to receive your final responses to our rebuttal. If you have any further questions, please let us know. We appreciate your invaluable input.

Best regards,

Authors
The author's response has addressed my concerns, I will maintain my original score.
Dear Reviewer **SrWK**,

Thank you for your thorough review and for recognizing the ethical considerations in our work. We appreciate your positive feedback.

Best regards,

Authors
Dear Reviewer **sKgV**,

Thank you for your comprehensive review and valuable feedback on our dataset's ethical considerations. We appreciate your recommendations.

Best regards,

Authors
Dear Reviewer **5Gaf**, 

We want to express our gratitude for your valuable time and constructive feedback.

Best regards,

Authors
Thank you for your response. I believe that most of my concerns have been well-solved and I thus increase from 5 to 6.