Towards Flexible Visual Relationship Segmentation

Authors: fangrui  zhu
jianwei  yang
huaizu  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 00:33:43

Abstract: Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG),  and referring relationships(RR) tasks. 
Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner.
In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. 
FleVRS leverages the synergy between text and image modalities, 
to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding.
Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD,  +4.7 $mAP$ on unseen HICO-DET.
Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.

Main paper: 
 introduction: 
 An image is not merely a collection of objects. Understanding the visual relationships between different entities at pixel-level through segmentation is a fundamental task in computer vision, which has broad applications in autonomous driving [28, 58] , behavior analysis [65, 67] , navigation [10, 15, 22, 27] , etc. Furthermore, segmenting relational objects extends beyond mere detection, playing a 38th Conference on Neural Information Processing Systems (NeurIPS 2024).Promptable Open-vocabulary One/Two-stage Model HOI SGG RLIPv2 [92] ✓ ✓ ✗ ✓ Two UniVRD [101] ✓ ✓ ✗ ✗ Two SSAS [38] ✗ ✗ ✓ ✗ One GEN-VLKT [47] ✓ ✗ ✗ ✓ OneFleVRS (Ours) ✓ ✓ ✓ ✓ OneTable 1 : Comparisons with previous representative methods in three aspects of model capabilities. To the best of our knowledge, our FleVRS is the first one-stage model capable of performing standard, promptable, and open-vocabulary visual relationship segmentation all at once. crucial role in improving visual understanding and providing a more comprehensive abstraction on the visual contents and interactions among them.Ideally, a visual relationship segmentation (VRS) model should demonstrate flexibility across three key dimensions. 1) Capability of segmenting various types of relationships, including both human-centric and generic ones. These relationships are defined as triplets in the form of <subject, predicate, object>. Human-object interaction (HOI) detection [4, 18] , which we adapt into HOI segmentation in our work, exemplifies this capability, such as <person, ride, horse> in Fig. 1 . Panoptic scene graph generation (SGG) [55, 81, 85] , captures generic spatial or semantic relationships among pairs of objects in an image, e.g., bench on pavement in Fig. 1 . A unified model that can handle these tasks concurrently is essential, as it eliminates the need for separate designs and modifications for each specific task. 2) Grounding of relational subject-object pairs with different prompts. Given various textual prompts, the model should output the desired entities and relationships, facilitating a more natural and intuitive user interface. For instance, it should be able to detect just the person in an image or all possible interactions between a person and a horse, as illustrated in Fig. 1 . 3) Open-vocabulary recognition of visual relationships. In realistic open-world applications, the model should generalize to new scenarios without requiring annotations for new concepts not seen during training. This capability includes detecting novel objects, relationships, and their combinations.Existing models in visual relationship segmentation (VRS) have targeted aspects of the desired capabilities but fall short of providing a comprehensive solution, as detailed in Tab. 1. Models have typically focused on tasks like human-object interaction (HOI) detection [23, 35, 47, 57, 72, 95, 97, 108] and panoptic SGG [50, 71, 81, 85, 93, 104] . Although models such as [92, 101] have attempted to unify VRS under a single framework, they need additional pretraining on HOI datasets (Tab. 1) and lack features such as promptable segmentation, which allows for dynamic entity and relationship generation based on textual prompts, as well as capabilities for open-vocabulary promptable segmentation. Efforts to detect instances referred to by textual prompts have been made [21, 38, 75, 107] , but these models fail to capture all desired entities or relationships comprehensively and struggle with classifying multi-label interactions between the pairs, limiting their effectiveness in complex scenarios. Although recent vision-language grounding models like [29, 52, 83] and multimodal large language models such as [2, 5, 76, 78, 88] exhibit enhanced capabilities in grounding instances specified by free-form text and show strong generalization over novel concepts, they still do not generate the required pairs in the format of segmentation masks. Furthermore, these models require significant computational resources and additional vision models for precise localizations. For open-vocabulary VRS, existing works [47, 91, 92] leverage textual embeddings to transfer knowledge. However, models [91, 92] fall short in grounding diverse prompts, while [47] is exclusively designed for HOI detection, not generic VRS.To address the limitations in existing models, we introduce FleVRS, a flexible one-stage framework capable of performing standard, promptable, and open-vocabulary visual relationship segmentation simultaneously. Our approach integrates human-centric (HOI segmentation) and generic VRS (Panoptic SGG) by adopting SAM [36] to unify different types of annotations into segmentation masks and using a query-based Transformer architecture that outputs triplets in the format <subject, predicate, object>. The model enhances its interactive capabilities by accepting textual prompts as inputs. These prompts are converted into textual queries that assist the decoder in accurately identifying and localizing objects within the relationships. Additionally, we unify the labels from different datasets into a shared textual space, transforming classification into a process of matching with a set of textual features. Leveraging textual features from the CLIP model [64] , we enable the effective matching of visual features with textual knowledge of novel concepts. This design In standard VRS, without textual queries, the latent queries perform self-and cross-attention within the relationship decoder to output a triplet for each query. For promptable VRS, the decoder additionally incorporates textual queries Q t , concatenated with Q v . This setup similarly predicts triplets, each based on Q v outputs aligned with features from the optional textual prompt Q t .In our FleVRS, the promptable VRS task goes beyond mere identification; it involves outputting segmentation masks for both subject and object pairs along with categorizing their relationships. This capability is essential for understanding and interpreting complex relational dynamics.Vision and language models Recent advancements in large-scale pre-trained vision-language models (VLM) [39, 64, 68, 69, 82, 94] and multimodal large language models (MLLM) [2, 5, 76, 78, 88] have demonstrated impressive performance and generalization capabilities across a variety of vision and multimodal tasks [36, 109, 110] . However, these models primarily focus on entity-level generalization, with open-vocabulary VRS receiving less attention. While recent efforts in zero-shot HOI detection [47, 61, 80, 92] often utilize CLIP [64] for category classification, their open-vocabulary capabilities lack the flexibility needed for prompt-driven input. Although current VLMs and MLLMs are adept at grounding novel concepts from text, they require significant computational resources and additional visual models, and cannot directly generate comprehensive segmentation masks for subject and object pairs. In contrast, our FleVRS provides a lightweight solution that effectively supports various types of open-vocabulary VRS, enabling category classification and the integration of novel concepts from prompts.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the abstract and section 1 introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Standard VRS. Given an image I, the goal of standard visual relationship segmentation (VRS) is to detect all the visual relationships of interest, either human-centric (i.e., HOI detection) or the generic ones (SGG), in terms of triplets in the form of <subject, predicate, object> (masks and object categories of subject and object, and the predicate category). The subject is always human in HOI detection, whereas it can be any type of object in SGG (may or may not be human). We consider the panoptic setting [85] of SGG, where a model needs to generate a more comprehensive scene graph representation based on panoptic segmentation rather than rigid bounding boxes, providing a clear and precise grounding of objects. To produce fine-grained masks, we convert existing bounding box annotations from HOI detection datasets [4, 18] into segmentation masks using the foundation model SAM [36] , as illustrated in Fig. 2 . We employ a filtering approach based on Intersection over Union (IoU) to filter out inaccurate masks. Details are in Appendix.Promptable VRS. Our FleVRS optionally accepts textual prompts as inputs, enabling users to specify visual relationships for promptable VRS. It accommodates three types of structured text prompts: a single element (e.g., <?, predicate, ?>), any two elements (e.g., <subject, predicate, ?>, <subject, ?, object>), or all three elements. Consequently, the model outputs only the triplets that match the specified elements in the prompt, as depicted in the right column of Fig. 1 . Without textual prompts, it functions as a standard VRS model, exhaustively generating all possible triplets, illustrated in the left part of Fig. 1 .Open-vocabulary VRS. In practice, it's essential for a VRS model to adapt to new concepts, including new categories of entities (i.e., subject and object), predicates, and their various combinations. Expanding these concept vocabularies to encompass a wider range is particularly challenging due to the vast potential combinations and the long-tail distribution of these categories. Thus, our goal is to equip the model to operate in an open-vocabulary setting, where it can effectively handle these diversities. It it important to note that the above three capabilities are complementary; for example, the text prompts in promptable VRS can include novel object or predicate categories.To this end, we propose integrating the above three aspects into a single unified framework. Since these settings are complementary, a general-purpose model should be capable of performing various combinations of these three functions. Additionally, their inherent similarities make it more intuitive to consolidate them within a flexible, unified approach.Inspired by the success of Transformer-based segmentation models [6, 109] , we design a dual-query system for our VRS model, illustrated in Fig. 3 . Latent queries, a set of learned embeddings, generate triplets (which may be empty) to formulate output masks and relationship categories. For promptable VRS, textual queries derived from input prompts are incorporated. We employ an image encoder and a pixel decoder to extract visual features, coupled with a relationship decoder that processes <subject, object> pairs and their interrelations. For open-vocabulary VRS, our approach shifts from traditional classification to a matching strategy that aligns visual and textual features for both object and predicate categories, enhancing the model's adaptability to new concepts. Each component of this architecture is elaborated further below.Image Encoder. Specifically, given the image I ∈ R H×W ×3 , it is first fed into the image encoder Enc I to obtain multi-scale low-resolution featuresF = F s ∈ R C F × H s × W s, where the stride of the feature map s ∈ {4, 8, 16, 32}, and C F is the number of channels.Pixel Decoder. A Transformer-based pixel decoder Dec p is used to upsample F and gradually generate high-resolution per-pixel embeddings P. P is then passed to the relationship decoder Dec R to compute cross-attention with query features.Textual Encoder. When a text prompt is provided for promptable VRS, we use the textual encoder Enc T to encode it into a set of textual queries Q t ∈ R Nt×Cq , where N t is the number of tokens in the textual queries, and C q denotes the channel number of query features. In practice, we use the textual encoder from CLIP [64] as Enc T . The format of the text prompt can be a single item (e.g. "<p>predicate</p>"), two of them ("
subject
<p>predicate</p>"), or all three of them, where "predicate" and "subject" denote category names of predicate and subject, respectively. "
", "<o>", "<p>" are used as separate tokens between subject, predicate and object in the text prompt. We could use natural language as the textual prompt instead of using a structured format. However, collecting the textual VRD data is not trivial, and we leave it as an extension of our model in future work.Relationship Decoder. The relationship decoder Dec R , based on a Transformer decoder design, processes pixel decoder outputs P and latent queries Q v to generate all possible triplets for standard VRS. Inside, masked attention [6] utilizes masks from earlier layers for foreground information. Each Q v output feeds into five parallel heads: two mask heads for subject and object masks (M s , M o ), two class heads for their categories (C s , C o ), and another class head for relationship prediction (C p ) During training, Hungarian matching aligns predicted triplets with ground truth. For standard VRS inference, triplets above a confidence threshold are considered final predictions. For promptable VRS, Enc T transforms text prompts into textual queries Q t that are concatenated with Q v and input into Dec R . This process, which uses self-and cross-attention mechanisms, generates <subject, predicate, object> triplets, similar to standard VRS. An additional matching loss during training ensures the model predicts triplets as specified by the text prompt. During inference, we calculate similarity scores between the textual query feature (last token's feature of Q t ) and the latent query outputs. We then select entities and relationships specified in the textual prompt from the top k triplets for the final outputs.To enable open-vocabulary VRS, our FleVRS uses the CLIP textual encoder [64] to match visual features with candidate textual features for object and predicate categories. We convert these categories into textual features using prompt templates, such as "A photo of [predicate-ing]" for HOI segmentation and "A photo of something [predicate-ing] (something)" for panoptic SGG. 1 The model computes matching scores between predicted class embeddings and these textual features, allowing classification beyond the fixed vocabulary of the training set and facilitating open-vocabulary VRS. Textual prompts are similarly encoded, and their features are used to calculate similarity scores for promptable VRS inference.We use Hungarian matching during training to find the matched triplets with ground truth ones. For standard VRS, we compute focal losses L s b , L o b and dice losses L s d , L o d on subject and object mask predictions, cross-entropy losses L s c , L o c , L p c on subject, object, and predicate category classifications, which can be written asEQUATIONwhere λ b , λ d , and λ c are hyper-parameters for adjusting the weights of each loss. λ s c , λ o c , λ p c are different classification loss weights for subject, object, and predicate. For promptable VRS, we adopt an additional matching loss L g between the matched triplet class embedding and the textual query feature (the last token feature of Q t ), which is in the form of cross-entropy loss. The final training loss is written asEQUATIONwhere λ g controls the weight of L g . L c depends on the text prompt. For example, given <subject, predicate>, there will not have L s c and L p c terms in Eq. ( 2), with subject and predicate categories being given. See the appendix for the concrete values of loss weights.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Do not have societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Datasets For HOI segmentation, we utilize two public benchmarks: HICO-DET [4] and V-COCO [18] . To fit Our FleVRS, we use SAM [36] to transform box annotations into masks and apply Non-Maximum Suppression (NMS) to remove overlapping masks with an IoU threshold greater than 0.1. We omit no_interaction annotations from HICO-DET due to incomplete annotation, leaving 44,329 images (35, 801 training, 8,528 testing) with 520 HOI classes from 80 objects and 116 actions. 2 V-COCO is built from COCO [49] , comprising 10,396 images (5,400 training, 4,964 testing), featuring 80 objects and 29 actions, and includes 263 HOI classes. Both datasets align with COCO's object categories. For panoptic SGG, we use the PSG dataset [85] , sourced from COCO and VG [37] intersections, containing 48,749 images (46,572 training, 2,177 testing) with 133 objects and 56 predicates.Data Structure for open-vocabulary HOI segmentation Following prior studies [3, 23] , we evaluate HICO-DET under three scenarios: (1) Unseen Composition (UC), where some HOI classes are absent despite all object and verb categories being present; (2) Unseen Object (UO), where certain object classes and their corresponding HOI triplets are excluded from training; and (3) Unseen Verb (UV), where specific verb classes and their associated triplets are similarly omitted. In UC, the Rare First (RF-UC) approach targets tail HOI classes, while Non-rare First (NF-UC) focuses on head categories. Originally, UC included 120/480/600 categories for unseen/seen/full sets, which reduces to 115/405/520 after removing no_interaction annotations. For UO, we select 12 unseen objects from 80, resulting in 88/432 unseen/seen HOI categories.Evaluation Metric For standard HOI segmentation, we convert the predicted masks to bounding boxes to compare with current methods, and follow the setting in [4] to use the mean Average Precision (mAP) for evaluation. We also turn the outputs of other methods into masks and report For standard panoptic SGG, following [85] , we use R@K and mR@K metrics, which calculate the triplet recall and mean recall for every predicate category, given the top K triplets from the model. A successful recall requires both subject and object to have mask-based IoU larger than 0.5 compared to their GT masks, with the correct predicate classification in the triplet.Implementation Details Following [6, 109] , we use 100 latent queries and 9 decoder layers in the relationship decoder. We adopt Focal-T/L [84] for the Image Encoder and DaViT-B/L for the pixel decoder. We use the textual encoder from CLIP to encode input text prompt and subject, object, and predicate categories. During training, we set the input image to be 640 × 640, with batch size of 64. We optimize our network with AdamW [54] with a weight decay of 10 -4 . We train all models for 30 epochs with an initial learning rate of 10 -4 decreased by 10 times at the 20th epoch. To improve training efficiency, we initialize Our FleVRS using the pre-trained weights from [109] . For all experiments, the parameters of the textual encoder are frozen except its logit scales. The loss weights λ b , λ d , λ c and λ grd (superscript omitted) are set to 1,1,2, and 2. More details are in the appendix.We evaluate our method on three benchmarks, i.e. HICO-DET [4] , VCOCO [18] for HOI segmentation, and PSG [85] for the panoptic SGG. [11] EfficientDet-d3 56.1 / --/ -CDN [95] ResNet-101 63.9 / 61.3 65.8 / 63.2 RLIP [91] ResNet-50 61.9 / 61.3 64.2 / 64.0 GEN-VLKT [47] ResNet-101 63.6 / 61.8 65.9 / 64.0 ERNet [48] EfficientNetV2-XL 64.2 / --/ -Ours Focal-L 65.2 / 66.5 66.5 / 67.9Table 3 : Quantitative results on V-COCO. We report both box and mask mAP .The best score is highlighted in bold, and the second-best score is underscored. '-' means the model did not release weights and we cannot get the mask mAP . Due to space limit, we show the complete table with more models in the appendix.performance over current single-stage methods in terms of box and mask mAP on HICO-DET. We also achieve competitive performance on VCOCO [18] , as shown in Table 3 . The advantages of Our FleVRS come from: (1) one-stage Transformer-based design with fine-grained training supervision for VRS. With subject and object masks, the model has more accurate supervision, compared with box annotations that contain redundancy [85] . (2) good language-visual alignment with the large-scale pretrained model [64] . Our FleVRS achieves competitive results without additional training on large-scale detection datasets [101] . Among one-stage HOI methods, our approach is simpler and able to tackle different datasets without modifications to the structure.Panoptic SGG From Table 4 , Our FleVRS can achieve competitive results in terms of R@50 and R@100 without elaborated designs for PSG, compared with most of previous work. Our FleVRS is not superior to HiLo [104] , which is mainly due to the long-tail distribution of the dataset and the Further experiments with various backbones demonstrate performance enhancements with larger models. Additionally, incorporating a box head for supervision alongside mask supervision enhances performance, which is attributed to the masked attention mechanism inspired by [6] . Exploring the potential synergies of training across multiple datasets, we find that while unified training improves VCOCO's performance due to its smaller size, HICO-DET and PSG show limited gains. This disparity is likely due to the different predicate categories used in PSG compared to HICO-DET and VCOCO.Comparison with previous works. UniVRD uses a two-stage approach, where the model first detects independent objects and then decodes relationships between them, retrieving boxes from the initial detection stage. In contrast, our method employs a one-stage approach, where each query directly corresponds to a <subject, object, predicate> triplet. This transition improves time efficiency from O(MxN) to O(K), where M is the number of subject boxes, N is the number of object boxes, and K is the number of interactive pairs. Our approach also provides greater flexibility by learning a unified representation that encompasses object detection, subject-object association, and relationship classification in a single model. [55] .In terms of training data, we use much fewer training data (x50 less, without using VG [37] and Ob-jects365 [66] ) and Our FleVRS with the Focal-L [84] backbone is much smaller than UniVRD [101] (164M vs 640M) with LiT(ViT-H/14), we achieve comparable results(37.4 vs 38.1 on HICO-DET). While our method does not match RLIPv2 [92] in performance, this is due to different design philosophies and goals. RLIPv2 is a two-stage approach optimized for large-scale pretraining and relies on separately trained detectors. Our FleVRS, however, is not designed for pretraining and does not include a separately trained detector. Our focus is on enhancing the flexibility of the VRS model without directly training on extensive curated data(x50 more, VG and Objects365). Thus, the differences in performance are attributed to the scale and design objectives. We further discuss the FLOPs and the number parameters of the backbone compared to previous works in the Appendix.We evaluate the ability of promptable VRS on the VRD dataset [55] , to compare with [38] . As in Table 5 , Our FleVRS can locate entities given flexible text query inputs and performs better localizing subjects and objects. Our FleVRS gets particular better results on localizing subjects (0.568 vs 0.335, 0.556 vs 0.334), which is mainly because there are fewer categories in subjects compared with objects and lots of subjects are humans, making it easier to segment subjects. We further evaluate our promptable VRS approach on HICO-DET and PSG, as they contain rich relationship labels. Since there are no previous baselines, we show qualitative results in Fig. 4 . We visualize the subject and object masks with the highest matching score for each example. We can see that the model is able to localize subject and object masks and predict their relationships given the structured textual prompt. We further perform postprocessing way to search triplets from standard VRS output, which serves as another baseline to show the effectiveness of our method. Please refer to section E in the appendix for results and discussions of fair comparison.Difference with standard REC tasks The referring expression comprehension (REC) tasks on benchmarks like RefCOCO [30] , RefCOCO+ [59] , and RefCOCOg [90] are designed to detect objects based on free-form textual phrases, such as "a ball and a cat" or "Two pandas lie on a climber."In contrast, the promptable VRS task in our work focuses on detecting subject-object pairs within a structured prompt format, such as <?, sit_on, bench> or <person, ?, horse>, as illustrated in Fig. 1 of the main paper. Our FleVRS is designed to encode and compute similarity scores for each of these elements separately. Our primary focus is on relational object segmentation based on a single structured query, which differs significantly from the objectives of REC benchmarks.We conduct open-vocabulary experiments following the defined zero-shot HOI detection setting [23, 25, 26, 47] on HICO-DET. As shown in Table 6 , Our FleVRS surpasses previous single-dataset methods across all settings, with its open-vocabulary capabilities stemming from the knowledge transferred from CLIP [64] . GEN-VLKT [47]  
conclusion: 
 In this work, we present a novel approach for visual relationship segmentation that integrates the three critical aspects of a flexible VRS model: standard VRS, promptable querying, and open-vocabulary capabilities. Our FleVRS demonstrates the ability to not only support HOI segmentation and panoptic SGG but also to do so in response to various textual prompts and across a spectrum of previously unseen objects and interactions. By harnessing the synergistic potential of textual and visual features, our model delivers promising experimental results on existing benchmark datasets. We hope our work can serve as a solid stepping stone for pursuing more flexible visual relationship segmentation models. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Do not have such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Citations are complete. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Deploying our model in real-world scenarios requires specialized pretraining data for relationship understanding, which is notably scarce. The lack of automated annotation pipelines and dependence on the CLIP model pose scalability challenges due to specific resource requirements. Ideally, we aim for a single general-purpose framework that can be trained on multiple datasets and enhance performance across various tasks and benchmarks. However, achieving this remains a challenge with our current model. We leave the exploration of how to synergize different datasets and develop effective training strategies to future work. While integrating free-form text inputs is more natural as large language models evolve, it necessitates additional preprocessing to align with our framework. Furthermore, the absence of comparable methods for promptable VRS makes complete fair benchmarking difficult.We use Focal T/L [84] network as the image encoder Enc I . Given the image I ∈ R H×W ×3 , we pass it to Enc I and obtain multi-scale features of different strides and channals F = {F s |s = 4, 8, 16, 32}, where s is the stride.Then, the pixel decoder Dec p gradually upsample F to generate high-resolution per-pixel embeddings P = {P i |i = 1, 2, 3, 4}, where i is the layer number and different P i s have the same channel number but different resolutions. P will then input to Dec R .Under standard VRS, Dec R takes latent queries Q v and P as inputs. Under promptable VRS, we use the textual encoder Enc T to encode the textual prompt into a set of textual queries Q t and concatenate Q t with the latent queries Q v , and input them to Dec R . Inside Dec R , cross-and self-attention are computed among queries and per-pixel embeddings, where masked attention [6] is adopted to enhance the foreground regions of predicted masks.On top of latent queries output Q v o ∈ R Nv×Cq (N v is the number of latent queries, C q is the channel number), there are five heads, producing predictions in parallel. They are two mask heads f Ms (•), f Mo (•) for predicting subject and object masks (M s , M o ), and two class heads g Cs (•), g Co (•) for predicting their object categories (C s , C o ). Another class head g Cp is used to predict relationships C p for this <subject, object> pair. Detailed operations can be written asEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONwhere Up[•] denotes the upsampling operation, T s , T o and T p denote candidate textual features of subject, object and predicate categories that are encoded by CLIP [64] . The mask embeddingsf Ms (Q v o ) and f Mo (Q v o )compute the dot products with the last layer's per-pixel embedding P 4 , respectively, and upsample to the original resolution as final mask predictions.Training. We employ Hungarian matching to align predicted triplets with ground truth, calculating mask and category classification losses on these matches. For promptable VRS, we introduce a matching loss to assess the similarity between the matched triplet embedding and the textual prompt's feature, formulated as a cross-entropy loss. The triplet embedding combines class embeddings from class heads. For the textual prompt's feature, we use the last token's feature from textual queries Q t . For instance, with a textual prompt like <subject, predicate, ?>, where the subject and predicate are specified, the similarity measurement utilizes the summation of their class embeddingsg Cs (Q v o ) + g Cp (Q v o ).Inference. Under standard VRS, we compute the confidence score of each triplet, which comes from the product of subject, object, and predicate classification scores. We take top k s (k s = 100) triplets to compute mean average precision for HOI segmentation and mean recall for panoptic SGG. Under promptable VRS, we compute similarities between the textual prompt's feature and triplet embeddings and choose k f (k f = 10) as the final predicted triplets. We set the input image size to be 640 × 640, with batch size as 64. The model is optimized with AdamW [54] with a weight decay of 10 -4 . We set N v = 100 and N v = 200 for Focal-T and Focal-L backbones, respectively, and C q = 512. The structure of pixel decoder Dec p is a Transformer encoder with 6 encoder layers and 8 heads. The structure of relationship decoder Dec R is a Transformer decoder with 9 decoder layers.Standard HOI segmentation The model only takes the image as input without textual prompts. Since the subject class is always "person" in HOI segmentation, we omit the subject class head. The model is trained with 30 epochs, with an initial learning rate of 10 -4 (10 -5 for the image encoder) decreased by 10 times at the 20th epoch. The loss weights λ b , λ d , λ o c and λ p c are set to be 2, 1, 1, 2. Promptable HOI segmentation To enable promptable HOI segmentation, we build three types of textual prompts: 1) "
person
<p>predicate</p>"; 2) "<p>predicate</p><o>object</o>"; 3) "
person
<o>object</o>". We evaluate the model on HICO-DET [4] since it contains richer human-object interactions than VCOCO [18] . During training, we randomly sample various types of text prompts and simultaneously train different objectives using distinct loss terms. To prevent the model from learning shortcuts, we select one ground truth triplet per training image and pair it with a randomly chosen textual prompt type. This approach ensures a balanced distribution of labeled training data for promptable VRS across different prompt types.Standard panoptic SGG We use the subject class head to predict the subject category and the model does not have textual prompts as inputs. The model is trained with 60 epochs, with an initial learning rate of 10 -4 (10 -5 for the image encoder) decreased by 10 times at the 40th epoch. The loss weights λ b , λ d , λ s c , λ o c and λ p c are set to be 2, 1, 1, 1, 2. Promptable panoptic SGG Similar to promptable HOI segmentation, there are three types of textual prompts: 1) "
subject
<p>predicate</p>"; 2) "<p>predicate</p><o> object</o>"; 3) "
subject
<o>object</o>". Similarly, during training, we randomly We set the weight of grounding loss λ g to be 2, while other weights are the same as standard panoptic SGG.Open-vocabulary promptable VRS We adopt the zero-shot setting from [47] for open-vocabulary HOI segmentation. To streamline our approach, we integrate the open-vocabulary promptable setting within the broader context of open-vocabulary VRS. In this setting, 'open-vocabulary' refers to handling both seen and unseen categories in the input textual prompts. We randomly exclude object and predicate categories during training and assess our model on these as well as on seen categories. Given the absence of existing benchmarks for this specific challenge, we present qualitative results demonstrating our model's proficiency in open-vocabulary promptable VRS.We show qualitative results of promptable and open-vocabulary VRS on HOI segmentation and panoptic SGG by giving the model different types of structured textual prompts. For simplicity, we show examples of omitting only one component of the triplet. We can see that our model is able to localize the correct subject and object and complement the missing element corresponding to the given textual prompt, e.g. <person, ?, dining_table> in Fig. 5 (c) and <truck, on, ?> in Fig. 6(b) . The model can also predict multiple interactions for the same subject-object pair, as shown in Fig. 5(c ) and Fig. 6(c ). We further trained two versions by removing unseen objects and unseen predicates, respectively. We show that our model can detect novel objects and predicates by feeding unseen concepts in textual prompts, as in the rightmost columns of Fig. 5 and Fig. 6 . Fig. 6 (b) shows the model outputs multiple instances in one subject mask due to similar patterns occurring in the training set. Note that the flexible VRD task is more difficult on the PSG [85] dataset due to its complexity of scenes, while we make the first attempt and our model is still able to show promising grounding results.Grounding ability compared with prompt-based vision-language models. Although promptable VRS is similar to vision-language models like GLIP [40] and MDETR [29] in grounding capabilities, it has distinct objectives. Unlike these models, which focus on entities, promptable VRS outputs triplets, making direct comparisons infeasible. Previous models are not equipped to handle the promptable relationship understanding task directly. To explore this, we modify our structural design to incorporate multiple text prompts as inputs, which are individually processed with their matching scores aggregated for classification. This experimental setup, however, results in reduced performance, increased inference time (26s vs. 5s), and higher GPU memory usage (5G vs. 3G). Thus, we argue that the proposed structure is suitable for tackling promptable VRS.Clarifications of choosing segmentation masks We firstly illustrate the importance of choosing segmentation masks over boxes in Fig. 7 . Traditional bounding boxes often include overlapping and ambiguous information, leading to redundancy. Segmentation masks, by accurately delineating object boundaries, provide a more precise and clear representation, reducing such redundancy, which is also illustrated in [85] and [86] . Besides, segmentation masks provide enhanced visual understanding and comprehensive contextual analysis. Additionally, object detection models often struggle to precisely extract foreground objects, which is why they are typically combined with segmentation models like SAM for fine-grained image tasks. Our model, however, presents a unified model that can localize both subjects and objects, along with their corresponding segmentation masks.Figure 7 : Illustration of the importance of using masks instead of bounding boxes.We show examples where one object is occluded by other objects. We show both bounding box annotations and masks generated with SAM, where only the masks can correctly locate the pure object.Noise handling in using masks generated by SAM. To address potential noise and inaccuracies in masks generated by SAM, we employ a filtering approach based on Intersection over Union (IoU). We compute the IoU between the generated masks and the original box annotations. Masks with an IoU score below a threshold of 0.2 are considered to have significant deviations from the ground truth and are filtered out. This threshold is chosen to balance the trade-off between including sufficient mask data and excluding those with substantial inaccuracies. The chosen IoU threshold helps ensure that only masks with a reasonable overlap with the ground truth annotations are retained. This threshold is set based on empirical evaluation and aims to minimize the impact of masks that are too noisy or incorrect, while still retaining as much useful data as possible. After using this strategy, we conduct analysis on 200 samples. We tested various thresholds and found this gets the best balance between denoising and data retaining(95% valid data retraining).More visualizations of generated masks. We have included additional visualizations in Fig. 8 to illustrate the fine-grained masks generated from the bounding box annotations of existing HOI detection datasets. These visualizations indicate that converting to masks significantly reduces the redundancy in the box annotations. Additionally, as shown in Fig. 8 NeurIPS Paper ChecklistOmit "something" for spatial relationships.interchangeable with "verb", "predicate". 


Decision: This paper was reviewed by four experts in the field. The authors did a great job in the rebuttal, and reviewers raised their ratings after the rebuttal (unanimously agreed to accept the paper). The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not higher: Reviewer k2zH raised some valuable concerns, including more discussions on the unified framework for VRS tasks and using SAM-based models to compare with other models trained on much smaller datasets.

Reviews: review: paper_summary: paper_summary: This paper propose a model to handle multiple visual relationship tasks, like HOI detection and Scene Graph Generation.

The proposed model is based on vision-language models similar to CLIP.

It handles different formulations like standard close-set, open-vocabulary, and prompted setting.
strength_weakness: strength_weakness: - Why not detection, rather than segmentation? What's the necessity for segmentation-style output rather than traditional detection style tasks? I would like more clarification on this.
- Also, (if not using the segmentation setting,) this method could be compared with more methods. Currently, the compared methods seems still not sufficient.
- This general contribution and pipeline is similar to UniVRD, except (1) this model is mask-based rather than box-based, and (2) a prompted setting is further supported. Thus, the contribution seems incremental.
- The performance is basically the same as UniVRD and is not as good as RLIPv2.
questions: questions: - What's the essential difference between this model and UniVRD?
- What's the necessity for segmentation rather than prediction?
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: This work proposes an approach for visual relationship segmentation that integrates the three aspects of a VRS model: standard VRS, promptable querying, and open-vocabulary capabilities. The idea of the article is very good, but the performance seems to be lacking.
strength_weakness: strength_weakness: 1-Previous work mostly uses ResNet series backbones, while the authors use a Focal-L Backbone. How do the FLOPs and parameter counts of this backbone compare to ResNet? I am concerned there may be an unfair comparison.

2-The performance of the proposed method does not seem to be particularly superior, which is a significant drawback. For example, in Tables 2 and 3, UniVRD, PViC, and RLIPv2 significantly outperform the proposed method in terms of box mAP metrics. Can the authors analyze this situation?

3-In line 208, the authors mention using SAM to generate masks. How do they handle the noise in these masks? Some masks might significantly deviate from the ground truth.

4-There are some typos, such as in line 25: "textttperson".
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: This work presents a novel approach for visual relationship segmentation that integrates the three critical aspects of a flexible VRS model: standard VRS, promptable querying, and open-vocabulary capabilities. By harnessing the synergistic potential of textual and visual features, the proposed model delivers promising experimental results on existing benchmark datasets.
strength_weakness: strength_weakness: 1. The experimental comparison suffers from potential unfairness, contrasting the proposed method with others using Focal-L backbone against those employing ResNet backbone seems somewhat unreasonable.
2. Since this paper includes referring expression comprehension (REC) tasks in abstract, it is unreasonable not to report experimental results on corresponding benchmarks like RefCOCO, RefCOCO+, and RefCOCOg.
3. While the research perspective of this paper is reasonable and innovative, the overall architecture design of Flex-VRS lacks novelty compared to previous related works.
4. More visualizations are needed for demonstrating the produced fine-grained masks generated by converting existing bounding box annotations from HOI detection datasets.
questions: questions: Please refer to Weakness Section for more details.
limitations: limitations: The authors have thoroughly discussed the limitations of this work.
paper_summary: paper_summary: This paper proposed a flexible framework that can effectively handle human-object interaction (HOI) detection, scene graph generation (SGG), and referring expression comprehension (REC) tasks. The proposed method further addressed the problem of promptable visual relationship segmentation and enabled the capability for open-vocabulary segmentation. Its main idea is to leverage the pretrained vision-language models for grounding textual features to visual relationship inside images. Experimental results show that the proposed framework is able to achieve SOTA performance on standard, promptable and open-vocabulary visual relationship detection/segmentation tasks.
strength_weakness: strength_weakness: 1. According to Table 7, the major performance improvement of the proposed method comes from the supervision of the mask head, which is provided from SAM and never used by previous works. When only the box head is used, the results of the proposed method is behind the current SOTA methods notably on HICO-DET and VCOCO. This makes the benefit of the proposed unified framework questionable if such large performance drops are observed.

2. Apart from introducing the segmentation head, the major technical contribution of the proposed method is unclear to me. The model architecture and losses are very similar to those previously proposed dual-decoder architectures in GEN-VLKT, RLIP, etc., and this part needs further clarification.

3. In Table 7, from the last row, we can see that the benefit brought by adding PSG is minor. Hence, the results on PSG dataset should also be reported so that the readers could have a better understanding on how unifying the framework could benefit each individual task.
questions: questions: Please refer to the Weakness section.
limitations: limitations: The authors have widely discussed the limitation of the proposed work and also its challenges in benchmarking.

Comments: title: Response to Authors
Summary of the author-reviewer discussion
comment: The authors have addressed most of my concents. After checking the peer review comments and the author's responses, I decided to raise the given score for this work.
We would like to express our sincere thanks to all the reviewers for their detailed and insightful feedback and suggestions. We also extend our gratitude to the ACs for organizing the review process for submission 3507. After rebuttal, we receive positive ratings from all reviewers.

We propose a flexible model that seamlessly integrates standard, promptable, and open-vocabulary visual relationship segmentation, delivering promising experimental results and demonstrating a high degree of adaptability. We would also like to highlight how we address the reviewers' concerns at the conclusion of the discussion period.

For reviewer LLLd, we resolve the concerns by: 1) clarifying the advantages of segmentation over detection; 2) clarifying how we compare with previous works under different settings; and 3) discussing the differences with previous works.

For reviewer W5TQ, we address the concerns by: 1) providing backbone comparisons; 2) discussing performance comparisons; and 3) elaborating on the noise handling of generated masks. Reviewer W5TQ acknowledged that our rebuttal addressed their concerns and subsequently raised the score.

For reviewer EDLE, we tackle the concerns by: 1) providing backbone comparisons; 2) clarifying our architectural design; 3) discussing the differences between our task and REC tasks, and we will change the term in the revised version to avoid confusion; and 4) showing additional visualizations of the generated masks.

For reviewer k2zH, we address the concerns by: 1) providing additional results for baseline comparisons; 2) providing further results on PSG along with clarifications; and 3) discussing the training data volume of our model compared to previous methods. Reviewer k2zH acknowledged that our rebuttal addressed the major concerns and subsequently raised the score.

Overall, our model integrates various functionalities into a single framework, and its effectiveness is validated under different settings, including standard, promptable, and open-vocabulary VRS, as well as using different supervisions, losses, and datasets for fair comparisons.

We will incorporate all suggestions from the reviewers into our revised version.

Best regards,

Authors of Paper 3507
Dear Reviewer k2zH,

We appreciate your prompt feedback. Thank you for your suggestion; we will definitely include a discussion on the mutual benefits of the unified model in the limitations and future work section of our revised version. We agree that SAM has been trained on significantly larger-scale data compared to the training datasets of the detectors. 

However, we would like to clarify that in our comparison, we focus on the number of images used directly to train our model, not the training data used to obtain the model(SAM) for pseudo-labeling. Additionally, we directly utilize the SAM model to generate pseudo-mask labels, whereas other methods rely on both off-the-shelf detectors and human labor for box labeling as the teacher model. We hope this helps clarify the third point further.

Thanks again for your time and dedication!

Authors of Paper 3507
Dear Reviewer EDLE,

Thank you again for your time to review our paper. Could you please check if our rebuttal has addressed your concerns at your earliest convenience? The deadline of the discussion period will end very soon. Thank you!

Best regards,

Authors of Paper 3507
Dear Reviewer LLLd,

Thank you again for your time to review our paper. Could you please check if our rebuttal has addressed your concerns at your earliest convenience? The deadline of the discussion period will end very soon. Thank you!

Best regards, 

Authors of Paper 3507
Dear Reviewer W5TQ,

We sincerely appreciate your thorough and professional review of our paper. We're pleased that our rebuttal addressed your concerns, and we're grateful for your recognition and timely response.

Best regards, 

Authors of Paper 3507
The author has addressed my concerns, and I will increase the rating to 5
1. I thank the authors for the additional clarification. The results in the current setup make more sense and convincing, which addressed my major concerns on the performance side.

2. I appreciate the authors' efforts on designing a unified framework for VRS tasks. Ideally, one of the major benefits/motivations of unification is that a unified framework usually enables training with more data in a flexible way and lead to performance improvement across different benchmarks. I suggest the authors to have a more extensive discussion on this perspective in the limitation and future work sections of the revised paper.

3. I have to point out that the detectors (e.g., DETR, Faster R-CNN, and YOLO) used by previous work are usually trained on small scale datasets like COCO, while SAM is trained with far larger-scale data than even the combination of COCO, VG and Objects365. Therefore, the two configurations here are not directly comparable.

Since my major concerns have been addressed, I would like to increase my rating to 5.
Thanks for the response. We'd like to clarify those concerns. 

1. We want to clarify that the suggested direct baseline — training previous SOTA models with the same SAM data used in our paper — is **exactly** what we provided in the rebuttal PDF, which we have also attached below. It shows that SAM-labeled data can hurt previous models. Thus, the major performance improvements of our work are due to both the SAM-labeled data and our architectural design.
| Model    | Trained with original boxes | Trained with SAM masks |
|----------|-----------------------------|------------------------|
| CDN      |             31.4            |          28.5          |
| STIP     |             32.2            |          29.7          |
| GEN-VLKT |             35.6            |          32.1          |


2. We argue that the unified model presented here integrates standard, promptable, and open-vocabulary VRS tasks into a single system, offering broader flexibility than previous methods, without requiring elaborate designs for various tasks. We did not claim that these tasks are mutually beneficial. Thus, we explain why the potential synergies of different datasets are not mutually beneficial in the rebuttal and on Lines 279-285 of the main paper. 


3. SAM is treated as an off-the-shelf tool to generate mask annotations, similar to how off-the-shelf detectors like DETR, Faster R-CNN, and YOLO are used to generate box annotations. We did not use SAM training data in our model training. However, previous models incorporate VG and Objects365 into their training processes. These models use box annotations generated by off-the-shelf detectors, but the training data for those detectors is also not counted as part of their models' training data.

If you have any further concerns or questions, please bring it so we can answer it promptly as it is close to the end of the discussion period. 

Thanks again,

Authors of Paper 3507
I thank the authors for their responses. However, my major concerns still remain.

1. Regarding Response 1, I thank the authors for providing the additional results. These results suggest that the proposed method and previous SOTAs have different focus, but do not address my concerns on whether the major performance improvements of the paper are brought by training models with SAM labelled data. More direct baselines should be training previous SOTA models with the same SAM data as used in the paper, or training the proposed method with the same segementation data used in the mask heads of previous methods.

2. Regarding Response 3, from the newly added results, we could observe that the performance gains brought by training with different datasets within a unified framework is very margnial. This make the true benefits of unification unconvincing.

3. The authors claim that the proposed method uses much fewer training data (x50 less, without using VG and Objects365) than UniVRD and RLIPv2, but I think this claim does not hold: the proposed model relies on the SAM model which is trained with far more labelled data than VG and Objects365.

Given the above concerns, I could not recomnad acceptance for the current version of this paper.
Dear reviewer, a reminder to take a look at the author's rebuttal and other reviews. Did the rebuttal address your concerns?
Dear reviewer, a reminder to take a look at the author's rebuttal and other reviews. Did the rebuttal address your concerns?
Dear reviewer, a reminder to take a look at the author's rebuttal and other reviews. Did the rebuttal address your concerns?
Dear reviewer, a reminder to take a look at the author's rebuttal and other reviews. Did the rebuttal address your concerns?
Dear Reviewer EDLE,

We sincerely appreciate your valuable feedback and suggestions. In our rebuttal, we have provided explanations regarding the experimental comparisons, differences from REC tasks and previous works, and have included additional visualizations.

We kindly inquire whether these results and clarifications have adequately addressed your concerns. Please feel free to let us know if you have any further questions.

Thank you very much!

Best regards,
Authors of Paper 3507
Dear Reviewer LLLd,

We sincerely appreciate your valuable feedback on our paper. In our rebuttal, we have discussed the differences between our work and previous studies, as well as provided clarifications regarding the segmentation of relational objects.

We kindly inquire whether these clarifications have adequately addressed your concerns. Please feel free to let us know if you have any further questions.

Thank you very much!

Best regards,
Authors of Paper 3507
Dear Reviewer W5TQ,

We sincerely appreciate your recognition and valuable feedback on our paper. In response, we have analyzed the backbone issue and provided detailed comparisons and evaluation clarifications of our model against previous works. Additionally, we have clarified the handling of noise in masks and will ensure this is clearly outlined in our revised version.

We kindly inquire whether these clarifications have adequately addressed your concerns. Please feel free to let us know if you have any further questions.

Thank you a lot!

Best regards, 
Authors of Paper 3507
Dear Reviewer k2zH, 

We sincerely appreciate your recognition and valuable feedback on our paper. In response, we have conducted additional experiments regarding the mask head and the PSG task. We have also provided a detailed clarification of our contributions compared to previous works.

We kindly inquire whether these results and clarifications have adequately addressed your concerns. Please feel free to let us know if you have any further questions.

Thank you a lot!

Best regards,
Authors of Paper 3507