Open-Vocabulary Video Scene Graph Generation via Union-aware Semantic Alignment

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:08:29

Abstract: Video Scene Graph Generation (VidSGG) plays a crucial role in various visual-language tasks by providing accessible structured visual relation knowledge. However, the requirement of annotating all categories of prevailing VidSGG methods limits their application in real-world scenarios. Despite the popular VLMs facilitating preliminary exploration of open-vocabulary VidSGG tasks, the correspondence between visual union regions and relation predicates is usually ignored. Therefore, we propose an Open-vocabulary VidSGG framework named Union-Aware Semantic Alignment Network (UASAN) to explore the alignment between visual union regions and relation predicate concepts in the same semantic space. Specifically, a visual refiner is designed to acquire open-vocabulary knowledge and the ability to bridge different modalities. To achieve better alignment, we first design a semantic-aware context encoder to achieve a comprehensive semantic interaction between object trajectories, visual union regions, and trajectory motion information to obtain semantic-aware union region representations. Then, a union-relation alignment decoder is utilized to generate the discriminative relation token for each union region for final relation prediction. Extensive experimental results on two benchmark datasets show that our UASAN achieves significant performance over existing methods, which also verifies the necessity of modeling union region-predicate alignment in the VidSGG pipeline. Code is available in supplementary material.

Main paper: 
 related work: 
 Open Vocabulary setting in SGG task. Scene Graph Generation (SGG) task [8, 17, 33, 34, 40, 41] aims to generate visual relation triplets in a given image or video, and provide the structure visual relation information for benefiting various downstream multimodal tasks. While existing SGG methods have achieved impressive performance on prominent public datasets, they are limited to predicting visual objects and their relations within a closed-set environment. The aforementioned constraint significantly hinders the practicality of SGG methods in real-world scenarios, due to they rely on training with known classes of objects and relations.Recently, the closed-set SGG has been extended to unseen classes through efforts made in the zero-shot setting [23, 42] , where the triplets for inference are not seen in the training set. Moreover, He et al. [12] proposes a more challenging open-vocabulary setting for image-based SGG task (Ov-SGG). To be specific, in Ov-SGG, the model is trained only utilizing the objects from a predefined set of seen object categories (or base categories), subsequently predicting relationships among unseen object categories (or novel categories). Both seen and unseen sets are subsets of the open-vocabulary object class set. It means that not only the object combinations but also the object categories themselves may not be seen during model training. Therefore, [12] utilize among of region-caption pairs for pre-training a visual-relation model, and finetune the relation model with prompt learning strategy. Besides, [12] also define a more challenging setting named general Ov-SGG, in which the predicate categories set are divided into novel set and base set. The former contains novel predicates during model inference that are not seen in training. Given the advantages conferred by pre-trained visual-language models (i.e., CLIP and GLIP), there is an inclination towards tapping into the ability of these VLMs for better visual or semantic relevant works. Inspired by the trend, Zhang et al. [47] explore the pre-trained visual semantic space (VSS) and propose a novel SGG model named 𝑉 𝑆 3 to transfer the language-image knowledge for benefiting Ov-SGG. Besides, due to VLMs (e.g., CLIP) struggle to distinguish between different relation types, Li et al. [21] integrate LLMs (e.g., GPT [1] ) into their model to generate detailed composition descriptions based on a chain-ofthought strategy for a better relation prediction.Different form the aforementioned SGG frameworks that are based on image, Gao et al. [9] is the first to explore the Openvocabulary Video Scene Graph Generation setting, and propose a novel framework named Relation Prompt Learning framework (Re-Pro), where compositional prompt is utilized to capturing complex spatial-temporal information for predicate representation learning. Although RePro have make preliminary attempts, the potential benefit from the correspondence between visual union regions and relations is still not considered. Moreover, some recent scene graph generation works [50] Video Scene Graph Generation. Video Scene Graph Generation (VidSGG) task [2, 4, 8, 24, 26, 27, [33] [34] [35] [36] [37] , aiming to detect and localize the visual relationships between different entity trajectories in a given video, has been widely used in various visual comprehensive task. Shang et al. [34] is the first to propose the VidSGG task with releasing a dataset named ImageNet-VidVRD, and propose a novel VidSGG framework named VidVRD. Inspired by Vid-VRD, Qian et al. [27] and Tsai et al. [37] focus on exploring the spatio-temporal information with a graph structure for relation prediction. Moreover, [36] design a Target Adaptive Context Aggregation Network to capture context information for each subjectobject pair. Different from them, Su et al. [35] propose a novel Multiple Hypothesis Association framework, which pays more attention to maintains the constructed multiple relations for selecting accurate ones. Moreover, Gao et al. [8] decompose the VidSGG pipeline, and propose a classification-then-grounding framework assisted by a video temporal grounding module for triplet localization. Although existing VidSGG approaches have achieved great performance, they are still limited to a closed-set training process, which prevents them from being applied in real-world scenarios. Therefore, in this paper, we propose an open-vocabulary VidSGG framework to improve the generalization of our model for recognizing unseen categories. Then we explore the alignment between union region and relation predicate concept. Specifically, we design a visual refiner to transfer such knowledge and ability from the bridge encoder into our framework by distillation. Relying on the ability bridging different modalities distilled from bridge encoder, the object trajectory features extracted by visual refiner also contains textual semantic characteristic. Therefore, we concatenate the representations of objects, union region, and motion information into a sequence to simulate textual relation triples, and a context semantic-aware encoder is designed to achieve comprehensive semantic understanding among these triplets. Then we propose a union-region alignment decoder to generate union-aware relation tokens. Finally, we incorporate the relation tokens with the union and motion information for relation predicate prediction. 
preliminary: 
 Problem Definition. Given a video 𝑉 , video scene graph generation (VidSGG) aims to detect the visual entities and their relationships in the form of relation triplet < 𝑠, 𝑝, 𝑜 >, where 𝑠 and 𝑜 are the class labels of subject and object, and 𝑝 is the class label of predicate. To conduct open-vocabulary VidSGG setting, following [9] we divide the categories of the objects and the predicates collected from all the annotations into base split and novel split. Specifically, we denote the base and novel object categories as: O 𝑏𝑎𝑠𝑒 with the number of 𝑁 𝑂 𝑏𝑎𝑠𝑒 and O 𝑛𝑣𝑜𝑒𝑙 with the number of 𝑁 𝑂 𝑛𝑜𝑣𝑒𝑙 , respectively, as well as P 𝑏𝑎𝑠𝑒 and P 𝑛𝑜𝑣𝑒𝑙 denoted as the base and novel predicate categories. We train our method with the triplet samples only containing base object and predicate categories, while evaluating it on both novel and all categories. Object Trajectory Generation and Feature Extraction. We utilize the same features as it in [9] . Specifically, a pre-trained object trajectory detector is employed to generate sets of classagnostic trajectories from a given video. The detected object trajectories are denoted as T = {𝑇 𝑖 } 𝑁 𝑖=1 . For each trajectory 𝑇 𝑖 , it consists of a bounding box sequence𝑇 𝑖 = {b 𝑗 } 𝑀 𝑗=1, where b 𝑗 is the bounding box, and b 𝑗 ∈ R 4 . For each trajectory 𝑇 𝑖 , we use a pretrained ViT [5] for feature extraction, and denote it as f 𝑖 ∈ R 𝑑 . Note that a video is typically cut into short video segments at first for computational simplicity during model training. To explore the alignment between visual union regions and predicate concepts, we extract union regions for the subject-object pairs in each video segment, where we construct 𝑁 𝑃 pairs totally, where 𝑁 𝑃 = 𝑁 * (𝑁 -1). The union region is the union of the bounding boxes of subject and the ones of object for each pair, and we also use pre-trained ViT to extract union region features, which is denoted as U ∈ R 𝑁 𝑃 ×𝑑 . Moreover, we also collect motion information for each subject-object trajectory pair < 𝑇 𝑠 ,𝑇 𝑜 > as the same in [30, 33] to make our model notice the relative position and the motion trend between objects in relation prediction, Specifically, following [9] we first allocate the category labels to the detected trajectories according to the Intersection over Union (IoU) with the ground-truth trajectories, considering only base object categories. Then, we bridge the visual and textual modalities as follows:EQUATIONEQUATIONwhere WordEmb(•) means a word embedding project. Moreover, the probability of classifying trajectory 𝑇 𝑖 to class 𝑐 ∈ 𝑂 𝑏𝑎𝑠𝑒 is as:EQUATIONwhere o 𝑐 is the text embedding of the object category 𝑐.According to [9] , we define the trajectories with allocated labels as positive samples, while others are negative samples, and calculate the classification loss L 𝑡𝑟𝑎 𝑗 .In consideration of the previous Ov-SGG methods excessively relying on the object-level alignment provided by pre-trained VLMs, we then explore the semantic correspondence between the visual union region and the relation predicate concept in this section. Knowledge Distillation Learning. At first, we design a visual refiner to transfer the open-vocabulary knowledge from our pretrained bridge encoder into our framework through a distillation strategy, which can facilitate our model paying more attention to visual-relevant semantic information. Specifically, we design a sequence of learnable visual concept tokens to obtain visual-aware concept features, and the tokens are denoted as Q = {q 1 , q 2 , ..., q 𝐿 }, where 𝐿 is the length of the sequence. To avoid the interference of noise contained in the detected object trajectories and reduce the costs during distillation, we utilize a small amount of manually annotated object trajectories, and also use pre-trained ViT to extract features, denoted as F 𝑀 = {f 𝑚 } 𝑀 𝑚=1 . To obtain visual-aware representations for each object trajectory, Q is extended to Q ′ ∈ R 𝑀 ×𝐿×𝑑 , and a attention module [38] , denoted as 𝐴𝑡𝑡𝑛(•, •, •), is utilized for modeling the dependencies of the visual concept tokens, denoted asV 𝑀 = Attn(Q ′ , Q ′ , Q ′ ).Then, a cross-attention-based module is used for comprehensive interaction between object trajectories and visual concept tokens, denoted as F 𝑣 = Attn(V 𝑀 , F 𝑀 , F 𝑀 ). After that, we use knowledge distillation strategy to transfer the knowledge into our framework. The distillation loss is as follows:EQUATIONwhere F ′ 𝑀 = BridgeEnc(F 𝑀 ). Semantic-Aware Context Encoder. As similar as generating F 𝑣 , we also obtain the visual-aware trajectory features F and union region features Û, where F ∈ R 𝑁 ×𝐿×𝑑 and Û ∈ R 𝑁 𝑃 ×𝐿×𝑑 . Relying on the ability bridging different modalities of our designed visual refiner, the object trajectory features and union region features also contain textual semantic characteristic. Therefore, we utilize a semantic-aware context encoder (SACEncoder) to conduct comprehensive semantic interaction within the triplet to obtain semantic-aware representations. Specially, according to the constructed 𝑁 𝑃 subject-object pairs before, we can obtain the features of the subject and object trajectories, denoted as F 𝑆 and F 𝑂 , where F 𝑆 , F 𝑂 ∈ R 𝑁 𝑃 ×𝐿×𝑑 . Then, we concatenate the subject trajectories, object trajectories, the union region representation, and two types of motion features for simulating relation triplets, which also consist of subject, object, and predicate. Then, to reduce the influence of the order of the components in our sentence, we utilize a attention-based module to conduct interaction among each of them with a learnable token denoted as f 𝑟 , and obtain the encoded "triplet" representations as follows:EQUATIONEQUATIONwhere 𝜙 𝑒𝑛𝑐 means a MLP, and Norm means normalization project. F 𝑚𝑜𝑡 ∈ R 𝑁 𝑃 ×𝐿 𝑚𝑜𝑡 ×𝑑 is the extracted object motion features for all subject-object pairs. Union-relation Alignment Decoder. After aggregating the context semantic from objects and motion information, we then need to generate the union-aware relation tokens for final relation prediction. Therefore, we design a union-relation alignment decoder (URADecoder), which is a multi-head attention module [38] :EQUATIONEQUATIONwhere MHAttn is a multi-head attention module, and 𝜙 𝑑𝑒𝑐 means a MLP. With the assistance of the decoder, we can generate the union-aware relation tokens F 𝑑𝑒𝑐 for further relation prediction.Prompt-based Union-relation Embeddings and Relation Predictor. Different from traditional classification tasks, the number of categories to be predicted is uncertain during training and inference. Open-vocabulary model predicting relying on the preextracted predicate category embeddings. Therefore, we introduce the prompt learning strategy into our framework for accessing predicate category embeddings. Specifically, we utilize a sequence of learnable word embedding tokens as the prompt tokens for each predicate category, and extract predicate category embeddings by our pre-trained bridge encoder as follows:EQUATIONEQUATIONwhere w 𝑙 (𝑙 ∈ 1, 2, ..., 𝐿 𝑟𝑒𝑙 ) is a learnable word embedding. c is the predicate word embedding for class 𝑐 ∈ P 𝑏𝑎𝑠𝑒 and r 𝑟𝑒𝑙 𝑐 is the category embedding.After obtaining the union-aware tokens F 𝑑𝑒𝑐 , we can indeed use them for relation prediction. Additionally, to comprehensively consider the initial and semantic-aware union region information, we aggregate the multiple union region features, motion information, and union-aware tokens into a relation predictor for for a more robust predicate classification:EQUATIONwhereF [𝑈 ]𝑒𝑛𝑐 is the encoded union region representation obtained from F 𝑒𝑛𝑐 and 𝜙 𝑝𝑟𝑒 means a MLP. For the each < 𝑇 𝑠 ,𝑇 𝑜 > pair, the relation feature of them is denoted as r 𝑠,𝑜 ∈ R. We compute the probability of relation predicate 𝑐 similar to Eq.( 3), where we replace f ′ 𝑖 with r 𝑠,𝑜 and replace the object category embeddings with the relation predicate embeddings r 𝑟𝑒𝑙 𝑐 , where 𝑐 ∈ 𝑃 𝑏𝑎𝑠𝑒 . The probability is denoted as 𝑝 𝑟𝑒𝑙 𝑠,𝑜 (𝑐), and the relation prediction loss is similar to the classification loss in [9] , denoted as L 𝑢𝑛𝑖𝑜𝑛 .Model Training. To complement union-level relation prediction process, we also take fine-grained subject-object pairs into relation prediction by a conventional strategy [9] . Specifically, we concatenate the subject and object trajectory features as the relation features, and leverage prompt learning strategy for generating predicate embeddings as similar to Eq.( 9), denoted as c 𝑠𝑜 𝑐 , 𝑐 ∈ P 𝑏𝑎𝑠𝑒 , for final relation prediction. The loss is denoted as L 𝑓 𝑔 .Therefore, the final loss L for optimizing our framework is as follows:EQUATION)Model Inference. During model inference, we only utilize the detected class-agnostic object trajectories for trajectory classification and relation prediction. Note that during relation prediction, all object categories are utilized, while either novel or all predicate categories are used for prediction. 
experiments: 
 In this section, we conduct extensive experiments on public datasets and demonstrate the effectiveness of our proposed UASAN framework. Please refer to Supplementary Material for the code, trained model, detailed parameter settings and more experimental results.Datasets. We evaluate our proposed method on the VidVRD [34] and VidOR [32] We compare our model with three baseline models on object trajectory classification task: ALPro [19] , RePro [9] , and BLIP-2 [20] .VidVRD-all R@5 R@10 R@5 R@10 R@5 R@10 ALPro 41. 38 1 . We directly input the detected object trajectories and the textual classes into the encoders of two pre-trained VLMs, i.e., ALPro and BLIP-2, and calculate the similarity for classification. Different from them, RePro chooses to distill knowledge from ALPro with a MLP module, and achieves better classification performance than ALPro on all splits. We only finetune our bridge encoder, and achieve the SOTA performance. Specifically, our model outperforms BLIP by gains of 8.80% in terms of R@5 on novel split on VidVRD dataset, while it also achieves (23.10%, 13.88%) improvements on all split. When compared with RePro, though RepPro surpass our model by gains of 0.66% on R@5 on base split, we still outperform it on novel and all splits with averages of 15.93%. Note that RePro and our method are both trained only with base categories, our finetuned bridge encoder achieves better performance on novel split while maintains the performance on base split. It demonstrate that the finetune strategy promotes our model better adopt to specific situation, thereby improving the classification performance of our model for both novel and base object categories.For generating a video sense graph in open-vocabulary setting, we separate the training processes of object trajectory classification and visual relation prediction. Therefore, the trajectory classification results are fixed during relation prediction.Setting. We compared our proposed method with following methods: MHA [35] , VRD-SGTC [26] , IVRD [24] , BIG-C [8] , and Re-Pro [9] . And the comparison results are shown in Table 2 . From the results in Table 2 , we have the observe that our proposed model achieves better performance on most of metrics, though only basesplit data is used for model training. When compared with those trained conventionally methods (i.e., all object and predicate categories are seen both in training and inference stages), we outperforms BIG-C by gains of (5.90%, 6.27%, 7.94%) in terms of mAP, R@50 and R@100 on RelDet task, and we also achieve an improvement with an average of 6.37% on RelTag task. Moreover, we also Our proposed UASAN consistently achieves the best performance on all metrics compared to RePro [9] . Specifically, we outperform RePro by gains of (2.24%,2.98%,3.29%) on mAP, R@50 and R@100 metrics on RelDet task. In addition, when evaluated on RelTag task, UASAN also surpasses RePro with an average of 7.60%. Moreover, to facilitate a fair comparison the relation prediction performance of our model with RePro, we design a RePro variant (denoted as RePro* in Table 2 ) in which the object classifier is replaced by the classifier pre-trained in our framework, aiming to mitigate the influence of the performance of open-vocabulary object trajectory classification. We can observe that the variant achieves comparable performance with RePro. Specifically, RePro outperforms RePro* by gains of (1.67%, 0.32%) on mAP and R@50 metrics on RelDet task, while the variant achieves improvements of (1.50%,2.81%,3.27%) on P@1, P@5 and P@10 metrics. The comparison results investigate that the relation prediction performance of our framework is more relevant to our designed alignment strategy between union regions and predicate concepts, and our proposed modules (i.e., visual refiner, semantic-aware context encoder and union-relation alignment encoder) also facilitate our model towards a comprehensive and robust relation prediction.Comparison in the open-vocabulary setting, we train our model with base-split, and evaluate it with novel and all-split. The results on VidVRD dataset are summarized in Table 3 . From Table 3 we can observe that our proposed method already achieves the best results on almost all metrics. Specifically, when evaluated on novelsplit, we outperform VidVRD-II by gains of (7.48%, 4.79%, 5.96%) on SGDet task. We also surpass it with an average of 10.32% on mAP, R@50 and R@100 metrics on SGCls task. When compared with Re-Pro, our proposed method surpasses it with (4.95%, 0.50%, 1.83%) in terms of mAP, R@50 and R@100 on SGDet task on novel-split. Moreover, when evaluated on SGCls and PredCls tasks, our proposed framework outperforms RePro with improvements of (4.18%, 3.97%, 5.30%) and (4.88%, 3.81%, 2.65%) for novel-split, respectively. For all-split, UASAN also achieves clear margin gains on SGDet (e.g., 23.57% vs. 21.33% on mAP and 15.90% vs. 12.92% on R@50). The superior performance of our model demonstrates the necessity of exploring the alignment between visual union regions and relation predicate concepts, which facilitates the generalization ability on unseen categories while maintaining the recognition ability for seen categories during model training.To investigate the effectiveness of each component of our model, we conduct ablation studies on VidVRD dataset. We implement three variants of our model as follows: (1) A conventional pipeline is implemented, where we concatenate subject and object features as the relation tokens and directly conduct predicate classification. This variant is denoted as Model A make it our baseline. (2) We introduce the union regions and the object motion information of the trajectories into our framework to generate relation tokens for predicate classification, denoted as Model B. (3) We add the designed context semantic-aware encoder to Model B to achieve comprehensive context semantic understanding, and the encoded representations are utilized for relation prediction. This variant is denoted as Model C. And the model using all designed modules is denoted as full-model. Table 4 presents the performance of each model variant. Exploring Facilitation of Pre-trained Models. To explore the influence brought by the employed pre-trained models (ALPro utilized in RePro and Bridge Encoder utilized in our framework), we implement the Model A, which only considers the alignment between predicate concept embeddings and the subject-object pair features, which are the concatenated subject and object features as in RePro. In other words, we replace the ALPro in RePro with pretraind Bridge Encoder. According to the results in Table 4 we can find that Model A achieves comparable performance with RePro. Specifically, when evaluated on novel-split, RePro outperforms Model A by gains of (2.31%, 1.31%) on mAP, R@50 and R@100 on SGDet task, while Model A achieves improvements of (0.63%, 1.00%) on R@50 and R@100 metric under all-split. Effectiveness of Exploring Union Region Modeling. We develop the Model B to verify the necessity of modeling the alignment between union regions and relation predicate concepts. In Model B, we simply design a sequence of learnable word tokens, and generate union-level predicate embeddings. Then, the union by gains of (0.96%, 1.32%, 2.32%) on SGDet task under novel-split, which demonstrates the designed decoder brings our model stronger generalization to novel categories compared with only utilizing semantic-aware representations. In addition, full-model also achieves great improvements by gains of 1.08% on mAP metric when evaluated with the all-split. Additionally, clear improvements are also achieved on all metrics of SGCls and PredCls tasks. The results indicate the effectiveness of our designed decoder structure, which has the capability to integrate multiple representations and generate discriminative union-aware relation tokens for final robust relation prediction. 
conclusion: 
 In this paper, we propose an open-vocabulary video scene graph generation framework named Union-Aware Semantic Alignment Network (UASAN), which explores the alignment between visual union regions and relation predicate concepts for more comprehensive and robust relation prediction. Specifically, we design a visual refiner to generate visual-aware representations for detected object trajectories and their union regions. Then we design a semanticaware context encoder to obtain semantic-aware representations.After that we utilize a union-relation alignment decoder to generate discriminative union-aware relation tokens for final relation prediction. Extensive experimental results on VidVRD and VidOR benchmarks demonstrate our proposed UASAN outperforms almost all SOTA methods on Ov-VidSGG task. In the future, we aim to explore a lighter framework while maintaining comparable performance and application in various downstream tasks, such as VQA and video caption.  
: 
 role in various visual comprehensive tasks, such as visual question answering [3, 15, 16] , video retrieval [6, 7] , and video captioning [25, 48] , by furnishing structured knowledge to enhance video understanding.Despite achieving impressive performance, existing VidSGG frameworks [8, 24, 26, [32] [33] [34] 49] remain constrained to recognizing objects and predicting visual relations within closed-set scenarios, which entails that the categories of objects and relation predicates are pre-defined and manually annotated. However, such the closedset model training process prevent current VidSGG frameworks from being employed in real-world scenarios, due to their inclusion of various visual object or relation concepts that do not appear or are unseen in the model training set. When encountering these novel categories, current VidSGG methods are likely to fail to recognize or classify them into known categories as shown in Figure 1 . Meanwhile, the movement of objects in real world scenarios over time makes the relationship between visual objects blurred and complicated, which also makes annotating more difficult. Moreover, due to the expensive and time-consuming labor costs for annotating, collect all categories from a real-world scenario is also not accessible. Therefore, it is crucial to address how to imbue a VidSGG model with the generalization ability to recognize novel categories when only being trained on limited categories.To reduce the need for annotating novel categories and improve the generalization of the models, open-vocabulary learning has been explored in the object detection field, named open-vocabulary object detection (OVD) [10, 11, 39, 44] . Specifically, the aim of open-vocabulary learning is to train a model with annotations on a part of classes (i.e., base classes), and generalize it to unseen classes (i.e., novel classes) during inference. Inspired by this, there have also been some preliminary studies on the open-vocabulary scene graph generation task (Ov-SGG) [9, 12, 47] in recent years.Benefiting from the encyclopedic knowledge of popular visionlanguage models (VLMs) [20, 22, 28] , such as CLIP [28] , existing Ov-SGG methods [9, 12, 47] can easily recognize novel objects and explore novel relationships through the object-level alignment between visual objects and textual category labels. However, these Ov-SGG methods excessively rely on the aligned visual and textual object features provided by pre-trained VLMs for model learning, while ignoring the alignment between visual relation regions and relation predicate concepts. The visual relation region is represented as the union region of two objects, and is usually regarded as a type of assistant complement for relation prediction. In fact, most existing methods simply fuse such union region representations with object features, such as concatenating them with object features [47] , for relation classification, which is shown in Figure 2(a) . It is obvious that only using subject-object pair features (e.g., woman and house in Figure 1 ) to be aligned with textual predicate embeddings (e.g., near by in Figure 1 ) is insufficient for relation prediction. Though some recent closed-set scene graph generation works [50] have perceived the correspondence between visual union region information and visual relation concepts, they do not yet explicitly model the correspondence between them and relation concepts and still directly conduct alignment in the semantic space, which results in a lack of comprehensive interaction and correspondence between visual semantic information. As indicated in previous research [13, 43] , VLMs (e.g., CLIP [28] ) still encounter challenges in performing compositional scene understanding, such as inter-object relation, which results in incomplete and unreliable alignment between subject-object pair representations and relation predicate representations and leads to an ambiguous relation prediction.Therefore, we propose a novel Ov-VidSGG framework named Union-Aware Semantic Alignment Network (UASAN) to explicitly model the alignment between visual union regions and relation predicates in the same semantic space for joint feature fusion, and the framework is shown Figure 2(b) . Specifically, we design a visual refiner guided by a bridge encoder to transfer the open-vocabulary knowledge and the ability to bridge the modality gap into our framework. It facilitates our model paying more attention to visual-relevant semantic information, which results in a sequence consisting of visual-aware subject, object, union region, and motion information representations. After that, we design a semantic-aware context encoder to achieve comprehensive interaction within the constructed sequence for obtaining the semanticaware representations. Then we propose a union-relation alignment decoder to generate union-aware tokens based on the encoded sequence for final relation predicate prediction.In summary, our contributions in this paper are as follows:(1)We propose an open-vocabulary video scene graph generation method named Union-Aware Semantic Alignment Network (UASAN), which explicitly explores the alignment between the object trajectory union regions and the relation predicates for a more comprehensive relation prediction.(2) Our proposed UASAN consists of three cooperative components: (1) A visual refiner is designed to transfer the knowledge and ability into our framework for obtaining visual-aware representations; (2) a semantic-aware context encoder is utilized to achieve comprehensive semantic-aware interaction based on the representations from our visual refiner; and (3) a union-relation alignment decoder is proposed to integrate semantic-aware representations for generating discriminative relation tokens for final prediction.(3) Extensive experiments on two benchmark datasets, i.e., Vid-VRD and VidOR datasets, demonstrate the effectiveness of our proposed framework. 
 
 Appendix:  


Decision: This paper initially received relatively positive reviews (2 Weak Accept, 1 Borderline Accept, 1 Borderline Reject). The main concerns are about the: 1) The presentation needs to be further improved (such as figures, experimental settings); 2) More comprehensive ablations on each component; 3) Insights behind the designs. After the rebuttal, most of these concerns have been addressed, and the Borderline Accept has been raised to Weak Accept. In total, we think this submission is meaningful for our community and we recommend Accept.

Reviews: review: paper_summary: paper_summary: The paper introduces the Union-Aware Semantic Alignment Network (UASAN), a  framework to tackle the challenges in Video Scene Graph Generation (VidSGG). UASAN aims to bridge this gap by exploring the alignment between visual union regions and relation predicates in the same semantic space.
strength_weakness: strength_weakness: 1. It proposes UASAN to explicitly model the alignment between visual union regions and relation predicates, improving relation prediction.
2. Extensive experiments on VidVRD and VidOR datasets demonstrate significant performance improvements over existing methods.
limitations: limitations: 1. As the open-vocabulary setting is not first proposed by this work, it is not suitable to illustrate the open-vocabulary setting in Figure 1.
2. While the paper demonstrates that a major contribution is that the UASAN consists of three cooperative components, I wonder what is the contribution of each component? 
3. It would be better to provide some qualitative results to show that the model can really detect the visual union. 
4 It should be highlighted why visual union can contribute to the open-vocabulary case.
paper_summary: paper_summary: This paper introduces an open-vocabulary VidSGG method that improves the alignment between visual union features and relation concepts. The authors finetune a Q-former (from BLIP-2) to bridge the visual-textural modalities for subject-object pairs and relation predicate categories. They also design a distillation strategy to transfer the pre-trained knowledge from Q-former to a visual refiner, and introduce semantic-aware context encoder with union-relation alignment decoder to enhance the feature alignment. Extensive experiments on VidVRD and VidOR datasets show the state-of-the-art performance and verify the effectiveness of each component.
strength_weakness: strength_weakness: -	The novelty and technical contributions are considerable. 
 - This paper is the first one to introduce the semantic alignment between visual relation regions and predicate concepts for open-vocabulary VidSGG
 - This paper finds a novel and effective approach to improve the model’s open-vocabulary detection ability, i.e., finetuning the Q-former form BLIP-2 to bridge the visual-textural modalities for subject-object pairs and predicate categories.
-	The proposed UASAN model shows state-of-the-art open-vocabulary VidSGG performance on both VidVRD and VidOR benchmarks. 
-	The experiments and ablation studies are comprehensive and all the modules of UASAN are verified to be effective.
limitations: limitations: -	Lack of insights for the union-aware semantic alignment mechanism in the open-vocabulary setting.
 - How does the union-aware semantic alignment prevent the model overfitting to the base classes? 
 - Why the context semantic-aware and union-relation alignment are especially needed in the open-vocabulary settings? Does this also work well in conventional VidSGG setting?
-	The technical part (Sec.3) is a bit complicated. Some technical details are unclear. For example, 
 - What is the detailed structure of visual refiner? Does it share the same structure as the Q-former and trained from scratch?
 - The queries $Q$ (Line 446) and $Q’$ (Line 453) are not illustrated in Figure 3. It’s a bit confusing whether they are queries for the Bridge Encoder or the Visual Refiner.
-	Some experiment settings are confusing. 
 - In Sec.4.4, Line 497, the authors try to explore the influence of pretrained models (i.e., the ALPro in RePro vs. the Bridge Encoder in this paper). They say “we replace the ALPro in RePro with pre-traind Bridge Encoder”. However, the performance scores in Table 4 (e.g., 13.38% R@50 of SGDet) are exactly the same as those in the RePro’s paper (which is implemented with ALpro). Then, it’s unclear whether they re-implement RePro with their Bridge Encoder or implement their method with the ALPro in RePro.

-	Suggestions for the presentation of Figure 2. 
 - For conventional VidSGG in sub-figure (a). In the closed setting, they typically train a classifier without using text encode. So, they do not suffer from the semantic alignment issue. The current sub-figure (a) might be misleading. A more appropriate comparison in (a) would be existing open-vocabulary VidSGG methods rather than conventional closed-set VidSGG methods
paper_summary: paper_summary: The paper proposes an open-vocabulary video scene graph generation (VidSGG) framework named Union-Aware Semantic Alignment Network (UASAN). It aims to align visual union regions with relation predicates for better relation prediction in videos. The framework includes a visual refiner, a semantic-aware context encoder, and a union-relation alignment decoder. Experimental results on two benchmark datasets demonstrate the effectiveness of UASAN in comparison to existing methods.
strength_weakness: strength_weakness: 1. The paper addresses the important challenge of extending video scene graph generation to an open-vocabulary setting, enabling recognition of unseen object and predicate categories. This improves the practical applicability compared to closed-set approaches.
2. The proposed UASAN framework explicitly models the alignment between visual union regions and relation predicate concepts, which the authors argue is important for robust relation prediction but under-explored in prior work.
3. Extensive experiments on VidVRD and VidOR datasets show significant performance improvements over state-of-the-art methods.
limitations: limitations: 1. The novelty is limited since the proposed method relies heavily on pre-trained models like ViT, BLIP-2. The use of Knowledge Distillation seems like the engineering improvement, which lacks deep insights. While leveraging pre-training is common practice, it would be good to analyze the framework's robustness to different backbone choices. How much does performance degrade with smaller, more efficient backbones?  How much does performance degrade with multimodal large language models as backbones?
2. Related to the previous point, computational complexity and inference speed should be analyzed in the Experiment section. Runtime benchmarks and potential optimizations would strengthen the paper.
3. The experimental comparison focuses on open-vocabulary methods, but it would be good to include closed-set state-of-the-art results (even if not directly comparable) to contextualize the absolute performance.
4. Error analysis could yield additional insights - on which types of objects, relations, or videos does the approach still struggle? Qualitative visualizations of success and failure modes would help build intuition.
5. The paper does not sufficiently discuss how the model performs in varied real-world scenarios outside the benchmark datasets.
6. Please ensure consistent use of terminology throughout the paper to avoid confusion. For example, the different words of "visual refiner" and "visual feature extractor" in Figure 2 and Figure 3 could be confused.
paper_summary: paper_summary: This paper addresses the alignment problem of visual regions and relationship predicates in the feature space for video scene graph generation. To achieve this alignment, the paper designs a semantic-aware context encoder and a union-relation alignment decoder. Extensive comparative experiments were conducted on the VidVRD and VidOR datasets.
strength_weakness: strength_weakness: 1. This paper focuses on the alignment problem of visual regions and relationship predicates, which is crucial for scene graph generation and visual understanding. Compared to current object-based alignment methods, this paper provides a richer semantic understanding of visual information.
2. The figures in the paper are very clear and easy to understand, especially Figure 3, where the arrow indicators for the selection of Predicate category base are interesting.
3. There is a significant performance improvement on the VidVRD dataset.
limitations: limitations: 1. The paper focuses on video scene graph generation, but the model structure does not include specific designs for understanding temporal content. Could this network be equally applicable to image scene graph tasks?
2. I suggest that the authors consider including the experiments on the vidor dataset in the main text, as the text mentions several times that experiments were conducted on the VidOR dataset, but all experimental tables in the later sections are based on VidVRD, which might confuse readers. I think many of the detailed explanations in the implementation details could instead be placed in the supplementary materials or the open-source code.
3. According to the experimental results, this method achieves significant performance improvements on the VidVRD dataset, but the improvements on the VidOR dataset are very limited. I am concerned whether this is due to the characteristics of the dataset's bias or an overfitting issue?
4. The motivation of this paper emphasizes the alignment of visual regions and relationship predicates. I suggest that the authors add visualization experiments on the alignment design. Comparing with object-only alignment methods could more intuitively demonstrate the importance of aligning visual regions and relation predicates.
5. I would like to know if the authors have explored the impact of different splits for base and novel categories.