VrdONE: One-stage Video Visual Relation Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-18 08:32:12

Abstract: Video Visual Relation Detection (VidVRD) focuses on understanding how entities interact over time and space in videos, a key step for getting a deeper insight into video scenes beyond basic visual tasks. Traditional methods for VidVRD, challenged by its complexity, usually split the task into two parts: one for identifying what categories are present and another for figuring out their temporal boundaries. This split overlooks the natural connection between these elements. Addressing the need for recognizing entity independence and their interactions across a range of durations, we propose VrdONE, a streamlined yet efficacious one-stage model. VrdONE combines the features of subjects and objects, turning predicate detection into 1D instance segmentation on their combined representations. This setup allows for both category identification and binary mask generation in one go, eliminating the need for extra steps like proposal generation or post-processing. VrdONE facilitates the interaction of features across various frames, adeptly capturing both short-lived and enduring relations. Additionally, we introduce the Subject-Object Synergy (SOS) Module, enhancing how subjects and objects perceive each other before combining. VrdONE achieves state-of-the-art performances on both the VidOR benchmark and ImageNet-VidVRD, showcasing its superior capability in discerning relations across different temporal scales.

Main paper: 
 introduction: 
 Deep learning has propelled significant enhancements in visual video analysis for a variety of tasks such as object tracking [8, 22] , action classification [10, 37] , and action localization [10, 34, 36] . Despite the advancements, the increasing complexity of video data requires precise interpretation of spatial and temporal relationships among entities in videos. To address this challenge, Video Visual Relation Detection (VidVRD) has been introduced. VidVRD aims to detect all relational instances in a video, each represented by a triplet ⟨subject, predicate, object⟩. By harnessing rich semantic insights and interpretability, VidVRD is poised to enhance various These methods often overlook the spatiotemporal interactions between entities, thus failing to fully capture both transient and persistent relations. In contrast, our approach (d) utilizes a 1D temporal instance segmentation formulation that concurrently facilitates relation classification and framelevel relation mask generation for all relations in a single step, eliminating the need for additional post-processing. downstream applications, including video captioning [42] , video question answering [42] , and video visual grounding [15] .The VidVRD framework is divided into three sub-tasks: entity tracking, relation classification, and temporal boundary localization. As illustrated in Fig. 1 , the process begins with the identification of each entity's category and spatial location using pretrained video tracking models [6] . Traditional approaches to VidVRD typically treat the tasks of classification and temporal localization In front of :Inferencing × ×(a) Falsely merging relations. In front of :Inferencing × × Generated Proposals(b) Mis-splitting enduring relation. In front of :Inferencing × ×(c) Throwing all imperfectly matchable proposals. as distinct, processing them sequentially in either a classificationbased or localization-based manner. In classification-based strategies [30, 43] , relations are first identified on a clip-level (Fig. 1(a) ) or frame-level(Fig. 1 (b)), and relation periods are determined using heuristic temporal aggregation algorithms [5, 30] . Conversely, localization-based approaches (Fig. 1(c )) start with generating temporal proposals, which are then refined through a redundancy filtering mechanism before classification. However, existing methods do not coherently account for the spatiotemporal interactions between entities, resulting in suboptimal performance in both relation classification and localization. On one hand, the integration of clip-level and frame-level short-term relations primarily depends on locally extracted features. This can lead to ambiguous detections at the temporal boundaries of relations, such as mistakenly splitting a continuous relation into two disjoint ones (Fig. 2(a) ) or improperly merging temporally adjacent relations of the same category (Fig. 2(b) ). On the other hand, the use of generated proposals creates fixed-length temporal templates for video relations. As depicted in Fig. 2(c ), these templates often overlook potential relations that do not perfectly align with them during the inference stage, thereby constraining their effectiveness.In real-world scenarios, object interactions exhibit varied patterns across spatial and temporal dimensions. As shown in Fig. 3 , each type of video relation in the VidOR dataset [29] displays distinct spatiotemporal characteristics, including differences in duration and frequency. Furthermore, entities within these relations vary in aspects such as movement speed and range. For instance, the relation "in front of" and "shake hands" might occur simultaneously between two individuals during the same video segment. While "in front of" might persist throughout the segment, "shake hands" typically lasts only a few seconds and involves rapid movement. This diversity in spatiotemporal dynamics underscores the importance of accounting for these variations to accurately categorize relation types. Motivated by these observations, we aim to improve our model's performance in video relation detection by integrating richer spatiotemporal information.Building on this concept, we aim to integrate video relation classification and temporal boundary localization into a single holistic problem, reformulating it as a 1D temporal instance segmentation task (see Fig. 1(c) ). This unified approach allows for more precise relation classification and detailed relation boundary localization within a single inferencing step, benefitting from the improved supervision provided by temporal location binary masks.In this context, we introduce VrdONE, a spatiotemporal synergistic transformer designed for one-stage video visual relation detection. This model efficiently detects all relation instances between subject-object pairs in an untrimmed video. Initially, we capture the temporal and spatial features of all entities in the video sequence. For each subject-object pair, we align their features along the temporal dimension to enhance spatiotemporal interactions across various frames. We have developed the Subject-Object Synergy (SOS) module to improve mutual perception between the subjects and objects. Additionally, a Bilateral Spatiotemporal Aggregation (BSA) mechanism has been designed to effectively learn features that encapsulate both transient and persistent relations. These features are then processed by a relation encoder and directed towards the classification and temporal boundary localization branches. Both branches are concurrently trained in a single stage, supported by a relation identification loss and a mask prediction loss.In summary, our contributions are threefold: [14, 30, 31, 38, 41] have refined this method by enhancing classification accuracy using graph convolution networks [24, 38] or integrating multi-modal features [32, 38] . Innovations in association algorithms by Wei et al. and Su et al. [32, 38] have led to more precise temporal localization. However, clip-based approaches struggle with prolonged relations and are prone to errors from cumulative association steps. To better capture longrange relations, Chen et al. [5] introduced a multi-modal prototype learning approach that uses a 1D watershed algorithm [27] for frame-level classification and temporal localization. Concurrently, Gao et al. [11] and Zheng et al. [43] have explored parallel learning strategies for spatial and temporal relation metrics. Contrarily, Liu et al. [18] have attempted a new direction by generating numerous temporal proposals through sliding windows, filtered by template matching to pinpoint relation durations.Differing from these approaches, we reconceptualize the challenges of classification and temporal localization into a unified 1D instance segmentation task within a one-stage framework. Our method leverages interactions between subject and object features across frames to effectively capture both transient and persistent relations, significantly improving the precision of relation classification and localization. Spatiotemporal Synergistic Learning in Videos. Understanding vision tasks in videos requires a spatiotemporal synergistic approach. Initially, 3D convolutional neural networks were used to extract features across both spatial and temporal dimensions [4, 10] . More recently, transformer architectures have brought significant advancements in computer vision [9, 19, 35] . For instance, ViViT [1] integrates these architectures into video processing and sets new performance benchmarks, surpassing older 3D convolution-based methods. The Video Swin Transformer [20] adapts the Swin Transformer concept to video by expanding it into three dimensions, which enhances information capture from local to global contexts, improving efficiency in learning. Similarly, VideoMAE [34] and its successor, VideoMAE V2 [36] , leverage a Masked AutoEncoder approach in a self-supervised learning framework, applying consistent spatial masks across video clips to increase model robustness and effectiveness, thereby achieving notable performance improvements in various video processing tasks. Overall, integrating spatiotemporal elements is crucial for optimizing video processing across diverse applications. 
method 3.1 preliminaries: 
 Problem Setting. Given an untrimmed video 𝑉 of length 𝐿, which contains 𝑁 entities and 𝑀 possible relations, the goal of VidVRD is to learn a video relation detector G to generate all possible relations between the entities in 𝑉 and their corresponding durations, such thatEQUATIONwhere 𝑆 𝑖 and 𝑂 𝑗 denotes the subject and object when relation 𝑅 𝑘 happens, 𝑇 𝑠𝑡𝑎𝑟𝑡 and 𝑇 𝑒𝑛𝑑 denotes the begin and the end of 𝑅 𝑘 . In this case, 𝐹 = {𝑓 1 , 𝑓 2 , ..., 𝑓 𝑁 } represents the extracted features of the tracklets for all the objects. The feature for object 𝑖 is represented by 𝑓 𝑖 ∈ R 𝑙 𝑖 ×𝐶 , which is a set of feature vectors extracted for uniformly-sampled consecutive video frames, where 𝑙 𝑖 ≤ 𝐿 is the period that object exists in 𝑉 . Typically, the pipeline of VidVRD is divided into three sub-tasks: entity tracking, relation classification, and temporal boundary localization. After extracting 𝐹 from the results of entity tracking, previous works often treat the relation classification and temporal boundary localization of predicates separately. This procedure can be explained by Bayes's Formula, such that the distribution of the relation type 𝑅 𝑐 and its duration 𝑅 𝑑 are formulated either as:EQUATIONorEQUATIONHowever, the ignorance of the inherent connection between the two tasks and consequently deteriorates both the classification and localization performance. To fully mitigate the spatial and temporal features during the interaction of subjects and objects, we propose to reformulate the problem in a one-stage manner, i.e., directly estimating 𝑃 (𝑅 𝑐 , 𝑅 𝑑 |𝐹 ). Attention Mechanism in Transformers [35] has demonstrated its great ability to capture global information along an input sequence. Given the input query, key, and value, denoted by 𝑞 ∈R 𝑙 𝑞 ×𝐷 𝑞 , 𝑘 ∈ R 𝑙 𝑘 ×𝐷 𝑘 , 𝑣 ∈ R 𝑙 𝑣 ×𝐷 𝑣, the attention operation is calculated as:EQUATIONwhere typically 𝐷 𝑞 = 𝐷 𝑘 , 𝑙 𝑘 = 𝑙 𝑣 . Among the vanilla attention architecture, the self-attention proposes to generate the 𝑞, 𝑘, 𝑣 from the same input sequence 𝑒 ∈ R 𝑙 𝑒 ×𝐷 with three projection function:EQUATIONwhere restricting the perceptive fields on the sequence. Before applying the projection function, the input query, key, and value 𝑒 𝑞 , 𝑒 𝑘 , 𝑒 𝑣 will be separately divided into 𝐼 small segments 𝑒 𝑡 , 𝑡 ∈ [1, 𝐼 ], each segment is processed by an independent 1D convolutional layer ê𝑡 = Conv1D(𝑒 𝑡 ). The attention calculation is also performed segment-wise as:𝑊 𝑞 ∈ R 𝐷 ×𝐷 𝑞 , 𝑊 𝑘 ∈ R 𝐷 ×𝐷 𝑘 ,EQUATION)The results of 𝐼 segments will be further concatenated into one for the next attention layer. Based on this definition, we define the utilized local self-attention layer and local cross-attention layer used in our VrdONE asEQUATIONandEQUATIONrespectively.The goal of VrdONE is to build an efficacious one-stage video relation detector for simultaneously handling the relation classification and temporal localization. The overall pipeline of VrdONE is shown in Fig. 4 . Firstly, we apply a pretrained object detector [6, 26] to extract objects' features 𝐹 together with their spatial position Θ = {𝜃 1 , 𝜃 2 , ..., 𝜃 𝑁 }. For each subject-object pair, we process their features (𝑓 𝑠 , 𝜃 𝑠 , 𝑓 𝑜 , 𝜃 𝑜 ) using the Bilateral Spatiotemporal Aggregation (Section 3.3) for fully perceiving spatiotemporal interactions in the video. Specifically, we propose a subject-object synergy module for improving the mutual perception between the two entities. The In Bilateral Spatiotemporal Aggregation (BSA), we promote the mutual awareness of the subject and object features through mutual perception and ultimately encode them into a unified relational representation for later time dimensional segmentation. Given a pair of consecutive and untrimmed subject and object features, we generate subject-object pairs by enumerating all twoby-two combinations of triplet proposals, forming the set P = {(𝑓 𝑠 , 𝑓 𝑜 ) 𝑛 |1 ≤ 𝑛 ≤ 𝑁 * (𝑁 -1)}, where 𝑓 𝑠 , 𝑓 𝑜 , and 𝑁 denotes the subject feature, the object feature, and the number of detected entities. Subsequently, we crop the subject-object pairs with the overlapping time range to get the synchronized feature vectors 𝑓 𝑠 ∈ R 𝑙 𝑠𝑜 ×𝐶 and 𝑓 𝑜 ∈ R 𝑙 𝑠𝑜 ×𝐶 , with 𝑙 𝑠𝑜 denoting the length.To embed the detected spatial information into the features, we adopt an approach from [11] and employ absolute positional representations 𝜃 𝑎 ∈ R 𝑙 𝑖 ×8 for each entity. To be specific, the positional representations are comprised of the normalized bounding bbox coordinates and the offsets between two consecutive frames. Thereafter, the visual features 𝑓 and spatial features 𝜃 𝑎 are integrated into a general entity embedding with a multilayer perceptron (MLP), formulated as:EQUATIONwhere Concat(•, •) represents the concatenation along feature dimensions. Following the above process, the visual and spatial features of both subject and object are integrated into entity embeddings 𝑒 𝑠 and 𝑒 𝑜 , which are further fed into the Subject-Object Synergy Module to comprehend interactions. Subject-Object Synergy Module. The Subject-Object Synergy (SOS) module facilitates interaction between subject and object features to enhance mutual understanding. The SOS module is composed of an embedding layer and two Interactive Attention Blocks (IAB).The embedding layer shares the same structure as the encoder layer of a vanilla Transformer, consisting of a local multihead selfattention and an MLP. Specifically, the embedding layer of the 𝑙 𝑡ℎ SOS block is defined as:EQUATION)Accordingly, the subject and object features are embedded and are denoted as ē𝑙 and ê𝑙 .Within the SOS module, the Interactive Attention Block (IAB) enables information exchange between subject and object features to enrich their representations. Concretely, the Interactive Attention Block is composed of a self-attention layer and a cross-attention layer. For instance, to integrate object features into subject features, the aggregated subject representation is expressed as:EQUATIONLikewise, we augment the object features with the mutual information from the subject and finally obtain an aggregated object feature 𝑒 𝑙 𝑜 . After applying 𝐿 SOS layers, the enhanced subject and object features (𝑒 𝐿 𝑠 and 𝑒 𝐿 𝑜 ) capture comprehensive representations with innovative features from the interactions. We then fuse the subject and object features to form a unified representation for the subjectobject relation. To further facilitate positional awareness, we inject the relative position 𝜃 𝑟 𝑠𝑜 as follows:EQUATIONFinally, the subject feature, the object feature, and the relative position are projected to form the final representation of the subjectobject relation 𝑒 𝑠𝑜 using a two-layer MLP as:EQUATIONAfter obtaining the unified embedding 𝑒 𝑠𝑜 that contains rich spatiotemporal information, we further process it to achieve one-stage relation classification and temporal localization through the onestage relation detector. The one-stage relation detector is composed of a Relation Encoder 𝐸 𝑚𝑢𝑙 , relation decoder 𝐷 𝑟𝑒𝑙 , and temporal mask decoder 𝐷 𝑚𝑠𝑘 .Relation Encoder. We follow the design of feature pyramid network [16] and implement our relation encoder to capture multiscale features over varying temporal lengths. The relation encoder is stacked by a series of transformer blocks, which share a similar architecture with blocks defined in Eq. 10. Additionally, we propose to downsample the features before inputting them into each transformer block to perceive more long-range temporal information. By treating the unified features embedding 𝑒 𝑠𝑜 as the input of the first encoding block, the calculation of each block can be formulated as:â𝑙 -1 = 𝛿 (𝑎 𝑙 -1 ), ā𝑙 = LocalSA( â𝑙 -1 ) + ( â𝑙 -1 ),EQUATIONwhere 𝑎 𝑙 -1 is the output of the previous block and 𝛿 is the downsampling operation. In this way, multi-scale spatiotemporal features can be obtained from different layers of the relation encoder, forming a feature pyramid A = {𝑎 1 , 𝑎 2 , ..., 𝑎 𝐿 𝑒 }, where 𝐿 𝑒 is the number of transformer blocks.Relation Decoder and Temporal Mask Decoder. We employ a query-based transformer as our relation decoder and a feature pyramid decoder for temporal mask generation.For the relation classification, our relation decoder receives 𝑎 𝐿 𝑒 as its input to access high-dimensional semantic information. Specifically, the relation decoder consists of 𝐿 𝑟𝑒𝑙 transformer blocks with 𝑁 𝑞 learnable query embeddings 𝑞 ∈ R 𝑁 𝑞 ×𝑑 , which serve as template learners for all possible relation instances within a video. 𝑁 𝑞 and 𝑑 denote the number and dimension of query embeddings. The calculation can be formulated as:EQUATIONwhere 𝑒 1 𝑟𝑒𝑙 = 𝑎 𝐿 𝑒 . The final output of relation decoder 𝑒 𝑐𝑙𝑠 = 𝑒 𝐿 𝑟𝑒𝑙 𝑟𝑒𝑙 will pass through a classification head 𝐻 𝑐𝑙𝑠 to output the categories of the detected relations.For temporal relation localization, we generate a fine-grained mask using the temporal mask decoder for precise relation boundary detection in a per-frame mode. The temporal mask decoder contains a series of lateral connection layers for progressively upsampling the pyramid feature A. The number of lateral connection layers is the same as the number of transformer blocks in 𝐸 𝑚𝑢𝑙 . Concretely, the feature aggregation in the 𝑙-th layer isEQUATION)where 𝜂 denotes the upsampling operation, which performs linear interpolation on ã𝑙 -1 . The decoder's output 𝑒 𝑚𝑠𝑘 is recovered to the same length with 𝑒 𝑠𝑜 for better perception of temporal variations, and finally incorporates the classification embedding 𝑒 𝑐𝑙𝑠 through the localization head 𝐻 𝑙𝑜𝑐 to generate per-frame relation mask.Loss Functions. Similar to MaskFormer [7] , we employ a Bipartite Matching strategy to assign different queries to learn the corresponding instances. The matching cost for relation classification Table 1 : Comparison with state-of-the-arts on VidOR dataset. For detectors, FR, MG, and IE symbolize Faster R-CNN [26] , MEGA [6] , and Integrated Encoder, respectively. For extra features, L and M denote Language and Mask features, whereas I3D [4] and CLIP [25] denote visual feature extractor. For Social Fabric and our VrdONE, we represent the variants with extra features with a "-X" postfix. The best and second best performances are bolded and underlined.Relation Detection Relation Tagging mAP R@50 R@100 P@1 P@5 P@10 VRD-STGC [ EQUATIONwhere the classification cost -p𝑖 (𝑐 𝑔𝑡 𝑗 ) used in DETR [3] is replaced by Cross Entropy loss. This substitution is made perhaps due to the fact that -𝑝 𝑖 (𝑐 𝑔𝑡 𝑗 ) incurs a higher cost than cross entropy, potentially leading to premature overfitting in the training process, thereby hindering our model's learning. The L 𝑚𝑎𝑠𝑘 isEQUATIONwhich is a binary focal loss [17] and a dice loss [23] respectively. The overall loss function for training is given by:EQUATIONwhere 𝜎 (𝑖) denotes the index of the query matched to the ground truth with index 𝑖. Inference Phrase. During testing, we exhaustively enumerate all possible pairs to detect relations within the current video, resulting in 𝑁 × (𝑁 -1) potential subject-object pairs for inference. However, our model is capable of parallelly detecting all possible subject-object pairs and outputting all detection results in one step. For segmented frames, we consider those with a foreground probability greater than 0.5 as the detected relation range. Any post-processing to avoid isolated noisy positive points is ignored, as we have observed that our model demonstrates robustness in accurately identifying the temporal boundaries of relation instances. 
experiments: 
 Datasets. To evaluate our method, we conduct experiments on two datasets: ImageNet-VidVRD [31] and Video Object Relation (Vi-dOR) [29] . ImageNet-VidVRD comprises 1,000 videos sourced from the ILSVRC2016-VID dataset [28] , with a total duration of approximately 3 hours. It contains 35 Additionally, as depicted in Fig. 3 , the mean durations of relations in VidOR are typically much longer than those in ImageNet-VidVRD and vary across relation categories, posing additional challenges. Evaluation Metrics. We assess VrdONE's performance on two tasks: (1) Relation Detection (RelDet): This task involves detecting a set of visual relation triplets, and the corresponding tracklets of subject and object. A detected triplet is deemed correct if suffices both matching the ground-truth triplet and the detected tracklets manifest sufficient overlap with the ground-truth, e.g., 𝑣𝐼𝑜𝑈 > 0.5. We utilize mAP and Recall@K (R@K, K=50, 100) as the metrics for RelDet. (2) Relation Tagging (RelTag): This task only solely evaluates the precision of visual relation triplets and disregards the localization results of tracklets. Precision@K (P@K, where K=1, 5, 10) is employed as the evaluation metric for RelTag. Implementation Details. Following [11, 38] , we utilize the pretrained Object Detector MEGA [6] with backbone ResNet-101 [12] . Detection results are consolidated into object tracklets using deep-SORT [39] . We set the maximum length of overlapped subject-object durations as 512, otherwise cut out the outer length. The Multiscale Transformer Encoder incorporates 3 blocks, alongside the output from SOS, resulting in a 4-layer feature pyramid. With a downsampling ratio of 2, the feature pyramid comprises lengths of [512, 256, 128, 64] respectively. The decoder consists of 4 layers, [2] is implemented. Drop-out and Drop-path [13] rates are specified as 0 and 0.1. Training of VrdONE employs the AdamW [21] optimizer with a learning rate of 2 × 10 -4 . Warmup and Exponential Moving Average (EMA) techniques are employed to enhance and stabilize the training process.We conduct experiments on ImageNet-VidVRD and VidOR datasets and compare our VrdONE with the state-of-the-art methods on RelDet and RelTag tasks, as illustrated in Table 1 and Table 2 .On the VidOR dataset, we implement two versions VrdONE detector with the ordinary one obeys a traditional pipeline while the extra version incorporates features extracted by the CLIP [25] image encoder. For the ordinary implementation, our vanilla VrdONE achieves state-of-the-art performance on four metrics. In particular, VrdONE exhibits a noticeable improvement (+0.67%, +0.08%, and +0.87%) on all the RelDet metrics compared to the previous stateof-the-art [38, 43] , indicating a comprehensive enhancement in the temporal boundary localization by leveraging the spatiotemporal interaction. As for the implementation with additional CLIP features, method [11] and implementation [5] with extra features are also involved for a fair comparison. With extra CLIP [25] features integrated, VrdONE achieves the best or second-best performance across all six metrics. Specifically, VrdONE balances tasks between RelDet with RelTag, maintaining robust relation classification performance comparable to specialized models like Social Fabric [5] , which excel in perceiving category relationships. Notably, VrdONE demonstrates a significant advantage in temporal boundary localization performance, showing improvements of +0.96%, +0.36%, and +1.21% on mAP, R@50, and R@100, respectively.On the VidVRD dataset, VrdONE outperforms HCM by +1.17%, +0.23%, +0.16%, +2.00%, +2.00%, and +0.62% in terms of all the RelDet and RelTag metrics. By amalgamating the diverse metrics across both datasets, our VrdONE demonstrates exceptional performance, thereby validating the efficacy of the single-step methodology. Relation Detection Relation Tagging mAP R@50 R@100 P@1 P@5 P@10 w/o SOS 11. 28 𝑁 𝑞 Relation Detection Relation Tagging mAP R@50 R@100 P@1 P@5 P@10 Relation Tagging mAP R@50 R@100 P@1 P@5 P@10 [11] based on two-stage detection typically leverage a large query number (e.g., 𝑁 𝑞 = 100) to simultaneously detect the relations of all the subject-object pairs. Distinct from that, our work independently estimates the relation for each pair, therefore requiring fewer queries, as demonstrated in Table 4 . This can be extensively supported by a quantitative evaluation indicating that, on average, each subject-object pair in a single video clip in the VidOR training dataset is associated with 2.30 relations. Video length. Table 5 shows the impact of the input length of the video clips. Empirically, we truncate/pad the videos to a uniform length of 512 for best performance. Number of SOS modules. Table 6 illustrates the influence of the number of the Subject-Object Synergy layers. Accordingly, we set the number of layers as 2 in practice.In Fig. 5 , we present several visualization examples for comparison with BIG-C, VidVRD-II, and our VrdONE. The top part of Fig. 5 exhibits a sophisticated scene that features multiple humans and heavily occluded objects from VidOR dataset. Nonetheless, our Vr-dONE precisely captures the most of the relations. Specifically, our method can simultaneously consider spatial relations and action relations, e.g. "adult-play(instr)-guitar" and "guitar-in front of-adult", demonstrating that our method adequately considers spatiotemporal variance. In contrast, BIG [11] and VIDVRD II [30] are afflicted by the missed and wrong detections, especially in the case of the human and object interaction like "adult-play(instr)-guitar". In another easier case drawn from the VidVRD dataset, our VrdONE can produce diverse and confident detection results. It is worth mentioning that VrdONE accurately comprehends size and location relationships, affirming its advanced spatiotemporal understanding. Based on the results of the qualitative experiments above, we can fully demonstrate the superiority of our method and the effectiveness of spatiotemporal synergistic learning. 
conclusion: 
 In this paper, we reframe the Video Visual Relation Detection challenge as a 1D instance segmentation problem and unveil VrdONE, a pioneering one-stage detection model designed to curtail redundant heuristic post-processing. By leveraging the dynamic interplay between subject-object pairs, VrdONE enhances video representation, improving both temporal classification and localization tasks. The novel Subject-Object Synergy (SOS) module within VrdONE adeptly captures both transient and lasting relations by synthesizing mutual features. Comprehensive quantitative and qualitative assessments affirm that VrdONE achieves unparalleled performance in its field.Limitations. Despite VrdONE's advanced capabilities, it does exhibit certain constraints. Its effectiveness is partly dependent on the quality of the underlying pretrained video detection and tracking algorithms, as it utilizes processed tracklets for input. Additionally, VrdONE processes all possible subject-object pairs during inference without any preliminary filtering, potentially diminishing its overall efficiency. 
 
 Appendix: √ √ √ √ X 


Decision: This paper proposes addressing temporal localization in Video Visual Relation Detection (VidVRD) as a novel one-dimensional instance segmentation task, simplifying the complexity of the conventional two-stage approach. The proposed VrdONE method achieves good performance for VidVRD. The paper is well-written.

Most of the concerns previously raised by reviewers have been addressed in the rebuttal. Compared with state-of-the-art methods, the proposed method achieves the best performance with a moderate number of parameters and the least inference time. In addition to the existing ablation study, the authors have added an extra ablation study on 1D temporal instance segmentation in the rebuttal. The authors should also incorporate these changes into the paper upon acceptance.

Reviews: review: paper_summary: paper_summary: Summary: This paper introduces a framework for visual relation detection in videos. This framework is a one-stage method that does not require temporal proposal. The details of the models are well explained. The experiments were conducted on two datasets on both relation detection and relation tagging. The results show that the proposed framework achieves improved performance as to the previous models.
strength_weakness: strength_weakness: This paper introduces to address the relation detection method as a 1D segmentation problem. This enables the model to predict the relation temporal information in one feed forward propagation. The authors presented ablations of the model, showing the importance of each model component. The authors also analyzed the impact of different model parameters.
limitations: limitations: This paper claims that it addresses the challenge of complexity of the conventional methods which are a two-stage approach. The authors showed numerical results, however, it is unclear if the proposed framework can address the complexity/efficiency challenge. Additionally, from Table 1 it can be seen that the proposed framework shows similar performance as the Socal Fabric on the Relation Tagging task.

Many sentences used in this paper seem to be generated by AI technologies, many wordings are not natural languages and thus  are not easy to understand. The English usage of this paper needs significant improvement.
Many of notations used throughout the papers are not explained. Table 4 – Table 6 are the analysis of the model parameters, not the ablation of these parameters.
paper_summary: paper_summary: This paper introduces a novel perspective on video visual relationship detection (VidVRD) by conceptualizing temporal localization as a one-dimensional instance segmentation task. A new method named VrdONE is proposed, which integrates a bilateral spatiotemporal aggregation module and a one-stage relation detector. Experimental results on the VidVRD and VidOR datasets indicate that VrdONE achieves good performance.
strength_weakness: strength_weakness: 1. This paper proposes addressing temporal localization in VidVRD as a novel one-dimensional instance segmentation task.
2. The proposed VrdONE method achieves good performance for VidVRD.
3. The paper is written in a detailed and clear manner.
limitations: limitations: 1. The improvement brought by VrdONE is not highly significant.
2. The novelty of this paper primarily lies in the one-dimensional temporal instance segmentation; however, it lacks ablation studies or analyses to validate the advantages of this new paradigm.
3. For a one-stage method, it is recommended that the paper demonstrate its advantages in terms of inference time.
4. The presentation of Figure 5 is ambiguous and difficult to understand. It is recommended to use dividing lines or boxes to separate the top/bottom and detection/tagging areas.
paper_summary: paper_summary: This paper introduces VrdONE, a one-stage model for Video Visual Relation Detection. Unlike traditional methods that split category identification and temporal boundary detection, VrdONE integrates these processes by combining subject and object features into 1D instance segmentation. It also introduces the Subject-Object Synergy (SOS) Module, enhancing mutual perception before feature combination. VrdONE improves the performance of existing VidVRD models.
strength_weakness: strength_weakness: 1. This paper proposed a novel approach that simultaneous category identification and binary mask generation for video relations.
2. This paper introduces VrdONE, which is the first one-stage framework for VidVRD.
3. The proposed model outperforms VidVRD on the ImageNet-VidVRD and VidOR datasets.
limitations: limitations: 1. The writing of the paper is poor, and section 3 is not explained clearly.
2. The representation in the pipeline diagram appears unclear, please revise the diagram to align with the description provided in your methodology.
3. In the problem setting of section 3.1, it sets M possible relationships, but why is K used for the number of relationships in formula (1)?
4. BAS converts relational information into feature embeddings. What is its relationship with transient and persistent?
5. Formula (12) is not explained clearly.