UniQ: Unified Decoder with Task-specific Queries for Efficient Scene Graph Generation

Authors: xinyao  liao
dangyang  chen
yuanyuan  fu
wei  wei
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:14:52

Abstract: Scene Graph Generation(SGG) is a scene understanding task aims at identifying object entities and reasoning their relationships within a given image. In contrast to prevailing two-stage methods based on a large object detector (e.g., Faster R-CNN), one-stage methods integrate a fixed-size set of learnable queries to jointly reason relational triplets <subject, predicate, object>. This paradigm demonstrates robust performance with significantly reduced parameters and computational overhead. However, the challenge in one-stage methods stems from the issue of weak entanglement, wherein entities involved in relationships require both coupled features shared within triplets and decoupled visual features. Previous methods either adopt a single decoder for coupled triplet feature modeling or multiple decoders for separate visual feature extraction but fail to consider both. In this paper, we introduce UniQ, a Unified decoder with task-specific Queries architecture, where task-specific queries generate decoupled visual features for subjects, objects, and predicates respectively, and unified decoder enables coupled feature modeling within relational triplets. Experimental results on the Visual Genome dataset demonstrate that UniQ has superior performance to both one-stage and two-stage methods.

Main paper: 
 detr and its variants: 
 DETR [1] drops hand-designed components like spacial anchors [34] and non-maximal suppression [38] by adopting transformer encoderdecoder architecture [42] and a one-to-one assignment strategy [26] . But DETR suffers from low convergence issues and requires 500 training epochs while Faster R-CNN [39] only needs 12 epochs in the COCO dataset [35] . Researchers [28, 63] find out that the instability of bipartite matching leads to slow convergence. They accelerate DETR training by using a denoising training module.{predicate} {object} {predicate} {object} Decoder Q t Y {<subject, object, predicate>} Decoder Q s Y {subject} Q o Q p Decoder Q s Y {subject} Q o Q p Decoder Decoder (a) {predicate} {object} {predicate} {object} Decoder Q t Y {<subject, object, predicate>} Decoder Q s Y {subject} Q o Q p Decoder Q s Y {subject} Q o Q p Decoder Decoder (b) {predicate} {object} {predicate} {object} Decoder Q t Y {<subject, object, predicate>} Decoder Q s Y {subject} Q o Q p Decoder Q s Y {subject} Q o Q p Decoder Decoder (c)Other works [4, 16, 67] demonstrate that the one-to-one assignment strategy generates too few positive samples, which hinders the model from learning discriminative features. In this paper, we follow Group DETR [4] to enable one-to-many assignments while training and speed up training efficiency. 
formulation: 
 The task of SGG takes an image I as input. And then generates a scene graph G that consists of a set of relational triplets < S, P, O >, where S = {𝑠 𝑖 } 𝑖 ⩽𝑛 denotes the set of subjects, O = {𝑜 𝑖 } 𝑖 ⩽𝑛 denotes the set of objects, P = {𝑝 𝑖 } 𝑖 ⩽𝑛 denotes the set of predicates, n denotes the number of triplets, (𝑠 𝑖 , 𝑝 𝑖 , 𝑜 𝑖 ) represents the 𝑖-th triplet in G. Therefore, the SGG task is equivalent to modeling the conditional distribution 𝑃 𝑟 (G | I). The two-stage methods decompose it into a product of conditionals as Equation (1) . The first step aims to model 𝑃 𝑟 (S, O | I) distribution, which means detects all entities of the given image and exhaustively pair them into subject-object pairs. 𝑃 𝑟 (P | S, O, I) means that predicate prediction is conditioned on the subject-object pairs detected and paired from the first stage. This formulation significantly relies on the ability of object detector due to the entities omitted in the first stage have no opportunity to serve as subjects or objects.EQUATIONIn contrast, one-stage methods jointly optimize both object detection and predicate prediction by an end-to-end architecture to circumvent the cascading errors that is commonly associated with two-stage processes. However, while two-stage methods explicitly predict predicate classes conditioned on subject-object pairs, one-stage methods call for finding a balance between decoupled features demanded by object detection and coupled features needed by predicate prediction.We summarize two kinds of decoder structures in previous methods and propose a new formulation for better prediction.Single decoder with task-agnostic queries (STA) baseline [56] uses a fixed set of task-agnostic queries as input for a single decoder as shown in Figure 2 (a) . Compared to the formulation in Equation ( 1), it predicts not only subject-object pairs but also predicates accordingly at the same step as Equation ( 2). However, the subject, object, and predicate of a triplet occupy separate regions. Relying solely on a learned positional query may not be sufficient to precisely identify every component of triplets.EQUATIONThree decoders with task-specific queries (TTS) baseline [23, 24, 30] as shown in Figure 2 (c) devises three kinds of task-specific queries and input them into three decoders to predict subjects, objects, and predicates separately. As STA models subject, object, and predicate representations in the same query, coupled features shared within triplets are inherently contained in task-agnostic queries. For a fair comparison, we add an MLP before each decoder layer to model features shared among task-specific queries in the TTS and STS baselines.Single decoder with task-specific queries (STS) baseline proposed by this paper is shown in Figure 2 (b). We unify three decoders into a decoder while taking three kinds of task-specific queries as input. This formulation not only enables generating triplets in parallel as Equation ( 2) but also models features specific to subjects, objects, and predicates. Our formulation of one-stage SGG can achieve competitive results with fewer parameters as shown in Figure 1 . 
our approach: 
 Our proposed approach is composed of two major components: an image feature extractor implemented by a CNN backbone and a transformer encoder (Section 4.1), a relational triplet predictor implemented by a shared decoder with task-specific queries (Section 4.2). Like DETR [1] , the image feature extractor first maps the original image to a condensed feature space with lower dimensions. The triplet predictor takes image features as input and directly predicts a set of triplets < S, O, P >, eliminating the graph assembling module used in SGTR [30] . The approach follows the set prediction tradition as DETR [1] for end-to-end training and utilizes a one-tomany assignment strategy for faster convergence during training (Section 4.3). The framework of UniQ is represented in Figure 3 .For a given image I ∈ R 3×𝐻 0 ×𝑊 0 that has 3 color channels, 𝐻 0 pixels in height and 𝑊 0 pixels in width, a conventional CNN backbone (eg.ResNet [12] ) maps it into high-level image features X ∈ R 𝐶 ×𝐻 ×𝑊 , where 𝐶 denotes the number of feature map channels, and 𝐻 , 𝑊 correspond to the spatial dimensions of the lower-resolution map.A transformer encoder then flattens the spatial dimensions and extracts more compact features Y ∈ R 𝑑 ×𝐻𝑊 with positional encodings PE ∈ R 𝑑 ×𝐻𝑊 added to each layer. A 1 × 1 convolution reduces the dimension from 𝐶 to 𝑑. Equation (3) represents how the image feature extractor functions.EQUATIONOur Relational Triplet Predictor employs decoder architecture similar to Transformer [42] to decode the predicted relational triplets.The decoder takes three fixed-size sets of task-specific queries, i.e., subject queries Q s ∈ R 𝑁 ×𝑑 , object queries Q o ∈ R 𝑁 ×𝑑 , and predicate queries Q p ∈ R 𝑁 ×𝑑 as input, and harnesses a parameter-shared decoder to generate task-specific representations all at once, denoted by Z 𝑥 ; 𝑥 ∈ {𝑠, 𝑜, 𝑝}. The final step is to generate a set of triplet estimations < Ŝ, Ô, P > by input task-specific representations into feed-forward networks (FFN). The whole procedure is depicted as Equation (4) .EQUATIONThe relational triplet predictor contains three components: (1) Relation-aware Task-specific Queries: This component firstly generates three sets of task-specific queries for subjects, objects, and predicates respectively, then fuse each relational triplet to let the task-specific queries be aware of which relations they belong to (Section 4.2.1). ( 2 Relation-aware Task-specific Queries. We devise three types of task-specific queries for detailed triplet representation, They are three sets of learned embeddings Q s , Q o , Q p , each set has 𝑁 queries with size 𝑑 of representations. Since < q 𝑠,𝑖 , q 𝑜,𝑖 , q 𝑝,𝑖 > represents the 𝑖-th triplet, each task-specific query necessitates to know which relationship it belongs to before separate decoding. We adopt a multi-layer perceptron (MLP) concatenating queries output from the previous (𝑙 -1)-th decoder layer to make them aware of their global relational contexts. This module can be formulated as:EQUATIONwhere [; ] denotes the concatenate operation, 𝑙 denotes the 𝑙-th decoder layer. object queries Q 𝑜 and predicate queries Q 𝑝 can also be updated as Equation (5).Triplet Self-Attention. This module aims to capture the detailed dependencies among triplet components. For instance, when provided with a relational triplet such as <human, ride, horse>, the spatial location of 'human' may assist in locating the object 'horse'. At a particular decoder layer 𝑙, Q 𝑙 𝑠 , Q 𝑙 𝑜 , Q 𝑙 𝑝 has the size of (𝑏𝑠, 𝑁 , 𝑑), where 𝑏𝑠 is the number of batch size. To organize each triplet as a sequence, we reshape task-specific queries to the dimensions of (1, 𝑏𝑠 × 𝑁 , 𝑑), subsequently concatenating them into triplet queries Q 𝑙 𝑡 with dimensions (3, 𝑏𝑠 × 𝑁 , 𝑑). After that, the self-attention mechanism successfully operates within each triplet, and explicitly models interactions of spatial and semantic information between different sub-tasks. As the transformer architecture maintains permutation invariance, positional encodings PE 𝑙 𝑠 , PE 𝑙 𝑜 , and PE 𝑙 𝑝 , which share the same shape as queries, undergo a similar reshaping process as queries to form PE 𝑙 𝑡 and then added into the input of each attention layer. This module can be formulated as,EQUATIONwhere 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(.) denotes a multi-head attention operation, 𝑑 denotes the feature dimension of Q, K, and V, Q 𝑙 𝑡 on the left side of equation denotes the triplet queries embedded with mutual information among subjects, objects, and predicates. Q 𝑙 𝑡 then transits back to Q 𝑙 𝑠 ∈ R 𝑏𝑠 ×𝑁 ×𝑑 , Q 𝑙 𝑜 ∈ R 𝑏𝑠 ×𝑁 ×𝑑 , and Q 𝑙 𝑝 ∈ R 𝑏𝑠 ×𝑁 ×𝑑 , following the construction method in reverse.After embedding coupled triplet features for each task-specific query, this module adheres to the conventional architecture of the transformer decoder. It performs selfattention and cross-attention operations over task-specific queries in parallel and predicts subjects, objects, and predicates respectively. To facilitate parallel decoding, task-specific queries Q 𝑙 𝑠 , Q 𝑙 𝑜 , and Q 𝑙 𝑝 are concatenated along the batch size dimension. The above implementation enables task-specific queries to separately reason about the visual representations they require while using shared parameters. This module can be represented as,EQUATIONwhere Y denotes image features output from the encoder as Equation (3), PE 𝑦 denotes the position encodings of the image feature map Y. Q 𝑙 𝑜 and Q 𝑙 𝑝 are concurrently updated with shared parameters, similar to Q 𝑙 𝑠 .Inspired by DETR [1] , we redefine SGG as a set prediction task to facilitate end-to-end training, where we aim to generate a fixed-size set of 𝑁 triplet predictions represented as {( ŝ𝑖 , ô𝑖 , p𝑖 )} 𝑖 ⩽𝑁 , with ŝ𝑖 = { ĉ𝑠 , b𝑠 }, ô𝑖 = { ĉ𝑜 , b𝑜 }, and p𝑖 = { ĉ𝑝 , b𝑝 }. ĉ𝑥 ; x ∈ {s, o, p} denotes the predicted class label and b𝑥 ; 𝑥 ∈ {𝑠, 𝑜, 𝑝} denotes the predicted bounding box. These predictions are to be matched with the groundtruth scene graph G = {(𝑠 𝑖 , 𝑜 𝑖 , 𝑝 𝑖 )} 𝑖 ⩽𝑚 , where 𝑚 represents the number of ground-truth triplets. 𝑁 is always larger than 𝑚.The triplet matching cost between a triplet prediction ( ŝ𝑖 , ô𝑖 , p𝑖 ) and a ground-truth triplet (𝑠 𝑖 , 𝑜 𝑖 , 𝑝 𝑖 ) consists of the cost of its subject, object, and predicate predictions as Equation (8) .EQUATIONWhere 𝑖 denotes the 𝑖-th triplet, 𝑥 ∈ {𝑠, 𝑜, 𝑝}. And the cost function C is determined by the predicted class probability p and 𝐿 1 loss L 1 , generalized IOU loss [40] L 𝐺𝐼𝑂𝑈 of bounding box.Considering the number of ground truths 𝑚 is less than 𝑁 , we pad the scene graph G with ∅ (no relation). The Hungarian algorithm is used to find a bipartite matching between two sets that has the lowest cost as Equation (9) . 𝜎 ∈ 𝔖 𝑁 is a permutation of 𝑁 elements. Instead of performing bipartite matching after 𝑙-th decoder layer akin to the original DETR [1] , we aggregate costs across all auxiliary decoder layers for a single, stable matching pass as [23] . This approach consolidates the set of predictions from each decoder layer into a unified decision-making process. σ = arg minEQUATIONThe triplet losses L 𝑡 can be represented as L 𝑡 = L 𝑠 + L 𝑜 + L 𝑝 . Given the bipartite matching σ, L 𝑠 , L 𝑜 , and L 𝑝 are computed as Equation (10) , Where 𝑥 ∈ {𝑠, 𝑜, 𝑝},L 𝑏𝑜𝑥 is a linear combination of L 1 loss and generalized IOU loss L 𝐺𝐼𝑂𝑈 . Due to the long-tail distribution of predicates, we conduct both biased and unbiased training for our method. For unbiased training, we adopt a reweighted loss as [23] , where the class weight of each predicate 𝑐 is 𝑤 𝑐 = max{( 𝛼 𝑓 𝑐 ) 𝛽 , 1.0}, 𝑓 𝑐 denotes the frequency of 𝑐 in training set, 𝛼 and 𝛽 are hyper-parameters.L 𝑥 = 𝑁 ∑︁ 𝑖 [-log p σ (𝑖 ) (𝑐 𝑥,𝑖 ) + 𝟙 {𝑐 𝑥,𝑖 ≠∅} L 𝑏𝑜𝑥 ( b𝑥, σ (𝑖 ) , 𝑏 𝑥,𝑖 )] (10)Given that bipartite matching based one-to-one assignments produce a limited number of positive samples, thereby restricting model learning of generalized and discriminative representations, we adopt the one-to-many paradigm similar to Group DETR [4] for triplet assignments. We employ 𝐾 groups of 𝑁 learned task-specific queries, denoted as {Q 𝑥,1 , Q 𝑥,2 , . . . , Q 𝑥,𝐾 } for 𝑥 ∈ {𝑠, 𝑜, 𝑝}, which correspondingly yield 𝐾 groups of triplet predictions. We conduct one-to-one assignments on each based on the cost matrix, resulting in 𝐾 times matching { σ1 , σ2 , . . . , σ𝐾 } for each ground-truth triplet. 
experiments 5.1 settings: 
 5.1.1 Dataset. Our methods are trained and evaluated on Visual Genome (VG) [25] , a large dataset featuring 108,077 structured images. Considering the extreme long-tail predicate distribution of VG, we adopt a frequently employed subset VG150 that contains the most frequent 150 object classes and 50 predicate classes. Following standard practice in previous works [55] , we allocate 70% of images for training and reserve the remaining 30% for testing.To evaluate our methods, we employ widely used class-agnostic metrics: recall@K (R@K) to assess the performance of predominant classes and mean recall@K (mR@K) to present the performance of tail classes. We also introduce harmonic racall@K (hR@K), introduced by [23] to show the overall improvement of recall and mean recall. Additionally, we utilize zero-shot recall@K (zs-R@K) to exhibit the model's generalization ability to unseen categories, and no-graph constraint recall@K (ng-R@K) to evaluate all possible predicates related to subject-object pairs. Note that, K= {20, 50, 100}. We also present the average precision AP@50 (IoU=0.5) to evaluate the quality of entities involved in relationships.Table 1 : Scene Graph Generation Compared to Existing Methods. B denotes the type of backbone, and D denotes the type of detector. † means evaluate methods with top-3 links followed by [23] [30] for fair comparison. ♦ indicates utilizing reweighted loss for unbiased training, and the scaling parameter 𝛼 and 𝛽 is set to be 0.07 and 0.75 respectively. Red denotes the best performance of one-stage models, blue denotes the best performance of two-stage models, and underline denotes UniQ (ours) achieves the best performance on both one-stage and two-stage models.AP 50 Mean Recall (↑) Recall (↑) Harmonic Recall (↑) #Params (↓) @20 @50 @100 @20 @50 @100 @20 @50 @100 (M) X101 We utilize ResNet-101 [12] as the backbone network and a 6-layer transformer in the image feature extractor component.(Section 4.1). A 6-layer decoder with 8 attention heads and feature size of 256 is used for relational triplet prediction. The number of queries 𝑁 is 300 and the number of query group 𝐾 is 3. The FFNs for relationalaware queries have 3 linear layers with ReLU. (Section 4.2)We use pre-trained parameters of DETR [1] for object detection on VG as previous works [9, 23, 30] to speed up the convergence. For training, the initial learning rate of the backbone and the encoderdecoder architecture are set to be 10 -5 and 10 -4 respectively with the optimizer ADAMW. We train UniQ and baselines on 4 RTX 4090 GPUs with a batch size of 12.As shown in Table 1 , we compared our method with both two-stage methods based on Faster R-CNN [39] ( [10, 18, 31, 36, 45, 46, 55, 57, 61, 64] ) and one-stage methods based on DETR [1] ( [23, 24, 30, 41] ). As for the one-stage methods comparison, our proposed UniQ outperforms the state-of-art method IterSG [23] with a margin of 0.5/0.8 and 0.8/1.1 on mean recall@50/100 and recall@50/100. SGTR [30] and IterSG [23] adopt a top-k strategy to select more possible candidate triplets. We reimplement the strategy by selecting the 3 most likely predicates for each entity pair to evaluate our model. The result shows that UniQ †♦ surpasses IterSG [23] and SGTR [30] on recall@100 (1.9 and 7.8). UniQ †♦ also achieves the best performance on harmonic recall@20/50/100. In conclusion, UniQ achieves Table 2 : Results of zs-R@K and ng-R@K on Visual Genome.zs-R ng-R @50 @100 @50 @100 MOTIFS [61] 0 [12] as the backbone. To be compared with two-stage models that have a heavy object detector and always use extra features, our method still outperforms most of the two-stage methods. [36, 46, 55, 57, 61, 64] are tow-stage methods without unbiased training strategy, while [10, 18, 31, 45] are tow-stage methods using unbiased strategies. Our method UniQ †♦ employs the reweighted loss strategy that can compete with all the unbiased two-stage methods. Our method UniQ without biased training achieves the highest recall@20 among all methods. We also report zero-shot recall (zs-R@50/100) and no-graph constraint recall (ng-R@50/100) results as shown in Table 2 . We select methods that demonstrate good performance on recall and mean Table 3 : Ablation on task-specific queries. STA: Single decoder with task-agnostic queries. STS: Single decoder with task-specific queries. TTS: Three decoders with task-specific queries.Baselines AP 50 R@20 R@50 R@100 #params (M) STA # RQ TSA R@20 R@50 R@100 #params (M) recall as baselines. MOTIFS [61] and BGNN [31] represent the performance of the biased and unbiased two-stage method respectively. Relationformer [41] and IterSG [41] represent the performance of the biased and unbiased one-stage method respectively. The experimental results exhibit that our approach achieves the best performance both on zs-R@K and ng-R@K, proving that UniQ has better generalization ability compared to existing methods. Our biased training method surpasses MOTIFS [61] and Relationformer [41] 2.8 and 1.8 on ng-R@100, 3.5 and 2.8 on ng-R@50 respectively. Our unbiased training method competes with IterSG [23] and sees performance improvement on all metrics.5.4.1 Ablation on task-specific queries to prove the effectiveness of decoupled features. We conduct experiments on three baselines to prove that task-specific queries model subjects, objects, and predicates in detail and achieve better performance with the tiny amount of parameters increasing. STA denotes the baseline only has a decoder and one type of query to predict the whole triplet. STA denotes the method we adopt, which inputs task-specific queries into a shared decoder. TTS denotes using three decoders with distinct sets of queries to decode subjects, objects, and predicates separately. The experiment results are reported in Table 3 , that STS and TTS baselines achieve better performance compared to STA baseline, which demonstrates the effectiveness of task-specific queries. Furthermore, STS baseline can compete with the TTS baseline with almost 25% reducing of parameters, which conveys that multiple decoders are almost redundant.Components to prove the effectiveness of coupled features. Besides incorporating task-specific queries to model decoupled features of subjects, objects, and predicates. Our proposed model further probes the modeling of coupled features shared within triplets. Relation-aware queries (RQ) are implemented before each decoder layer to fuse features within the Table 4 proves that capturing the semantic information within each triplet assists the model for relational reasoning.We employ a one-to-many assignment strategy by initializing multiple query groups to stabilize bipartite matching and augment positive samples. The ablation experiment on the number of query groups is conducted as Table 5 . The performance increases when the number of query groups grows. While the number of query groups reaches 5, the performance becomes stable. So we adopt 𝐾 = 3 in all experiments. Note that the mean Recall metrics do not see an obvious improvement when training with more groups. This result may be due to the data augmentation that is brought by multiple query groups is dominant by head categories in distribution.Ablation on the transferability of the STS paradigm. We apply the STS paradigm on IterSG [23] model to prove its transferability, i.e. reducing three decoders to a shared decoder with task-specific queries and adding an MLP layer to fuse task-specific queries to get coupled features. As Figure 4 shows, the STS paradigm reduces the number of parameters of IterSG [23] and improves its recall rate performance.We visualize the attention map of the last decoder layer for STA baselines (without task-specific queries) and our proposed UniQ as shown in Figure 5 . We compare the top 10 relationships predicted by them and find out our proposed UniQ has the ability to deal with the following problems that exist in STA baselines. Alleviate mismatch of subject-object pairs. According to the first row in Figure 5 , the STA baseline matched the giraffe in the yellow rectangle with another giraffe's head, while UniQ matched the giraffe with its head correctly. This may be due to UniQ facilitating the interaction within triplets that results in more consistent triplets.Improve the semantic accuracy of entities. In the second row of Figure 5 , the STA baseline misidentified the dog's tail for its ear, while UniQ predicted the subject as 'tail' accurately. The results prove that task-specific queries adopted by UniQ carry more detailed and accurate semantic representations compared to methods implemented with task-agnostic queries.Deal with the deficient bounding boxes problem. In the last row of Figure 5 , the STA baseline predicted the 'surfboard' with a tiny bounding box, which is not correct, while UniQ can locate the object 'surfboard' more precisely. Additionally, UniQ predicted the predicate as 'standing on' rather than 'on' (predicted by STA baseline), which contains more semantic information. The results also demonstrate that task-specific queries facilitate more robust representations. 
conclusions: 
 In this work, we propose a novel one-stage architecture constructed by a unified decoder with task-specific queries (UniQ) for efficient SGG. Our proposed method provides an available solution to the weak entanglement problem in relational triplet prediction. We leverage task-specific queries to locate entities separately and fuse semantic features within triplets to share coupled features. Extensive experiment results show that UniQ surpasses existing one-stage methods and two-stage methods with fewer parameters and prove the effectiveness of UniQ. 
: 
 Figure 1 : Comparison of Baselines. We compare the number of parameters and Recall@100 among three baselines in Section 3 and our method UniQ. It demonstrates the formulation of STS baseline that we adopt in UniQ achieves better performance with fewer parameters.Scene Graph Generation (SGG) intends to form the semantic graph for a given image, where detected objects serve as nodes and relationships between them serve as edges. A relationship can equally be represented as a <subject, predicate, objected> triplet, and each element of the triplet is detailed by its class label and spatial location. Generating such scene graphs for a given image supports structured reasoning over high-level semantics, facilitating downstream complex tasks like visual question answering(VQA) [15, 47] , image retrieval [19, 20] , and image captioning [14, 58] with powerful representation.Many works are proposed for the SGG problem, which can be roughly categorized into two classes, namely, two-stage based and one-stage based methods. The former usually employs a pre-trained object detector (eg. Faster R-CNN [39] ) to generate 𝐸 entity proposals. Then they pair each entity together as a complete graph for relation inference, by exhausting all possible 𝐸 2 subject-object pairs. However, relying on a two-stage architecture may introduce the risk of error propagation, i.e., if some objects are omitted in the object detection stage, all relationships related to them can not be considered. One-stage methods pave the way for SGG by formalizing SGG as an end-to-end set prediction task, similar to DETR [1] . This architecture bypasses the subject-object paring step by decoding a fixed set of 𝑁 queries, successfully reducing the resolution space from 𝑂 (𝐸 2 ) to 𝑂 (𝑁 ). However, due to SGG having three inter-dependent sub-tasks: subject, object, and predicate prediction, the challenge of weak entanglement still persists in SGG. On the one hand, determining which entity plays a role as the subject or the object depends on features shared among relationships. On the other hand, locating and predicting entities rely on the decoupled visual features specific to each entity or relationship. Some previous one-stage methods [7, 23] harness multiple decoders to emit decoupled visual features separately for each sub-task but overlook modeling the coupled feature to enhance interactions within triplets, while other methods [56] learn holistic triplet features by single decoder, which are not able to model precise positional locations and decoupled semantic features specific to subjects, objects, and predicates.In this work, we propose a new formulation through a Unified decoder with task-specific Queries, namely UniQ. UniQ treats each sub-task as decoding distinct sets of queries from a unified decoder rather than multiple decoders. The unified decoder takes three kinds of queries specific to subjects, objects, and predicates as input, and extracts decoupled visual features respectively for each sub-task in parallel. Different sets of task-specific queries undergo separate self-attention and cross-attention mechanisms with shared parameters. By sharing decoder parameters across sub-tasks, the decoder is enabled to concurrently generate features for subjects, objects, and predicates, facilitating mutual assistance among the sub-tasks. Besides utilizing task-specific queries to capture decoupled features for each sub-task, UniQ further enhances the interaction within triplets. In each decoder layer, UniQ fuses global and local context within triplets and embeds them into task-specific queries. This enables queries to dynamically capture detailed visual features according to the relational context. To acquire comprehensive global features, UniQ integrates subject, object, and predicate queries into a holistic query, thereby making every task-specific query perceive the entire triplet context. UniQ also employs a triplet self-attention module to capture the intricate interactions among subjects, objects, and predicates, thus extracting the local context in detail. The major contributions are summarized as follows:• We propose a novel one-stage SGG formulation, UniQ, using a unified decoder with task-specific queries for parallel triplet decoding. This method enables detailed representation modeling for each sub-task and reduces parameter compared to methods that utilize multiple decoders (Figure 1 ). • UniQ deals with the weak entanglement between entities and predicates by utilizing task-specific queries to model decoupled spatial locations and facilitating interaction within each triplet to model coupled semantic features. [13, 33, 52, 61] , Tree-LSTMs [32, 44, 46] , GRU [6, 51, 55] ), Graph Neural Network [5, 11, 17, 43, 53, 54, 57] , or attention mechanisms [21, 49, 66] to propagate information among entities and their relationships. Others leverage different kinds of prior knowledge, which are language prior [22] , statistical prior [8, 61] , and knowledge graph [3, 59] . Motifs [61] discovers that there are numerous motifs in SGG, indicating that predicate categories are largely determined by the labels of the subject-object pairs. [3] utilize a commonsense graph to predict visual relationships according to human instinct. The long-tail distribution of predicates existing in SGG datasets is a non-trivial problem. Loss re-weighting [2, 60] , data augumention [10, 18, 31, 62] , and unbiased representation learning [29, 37, 45, 50, 65] were used to mitigate this problem. However, two-stage methods are built upon an object detector with numerous parameters that result in substantial computations overhead and impede end-to-end training. Additionally, the twostage methods pair all objects to generate triplets proposals, leading to a quadratic growth 𝑂 (𝐸 2 ) in the solution searching space, where 𝐸 represents the number of object proposals.2.1.2 One-stage Scene Graph Generation. Inspired by DETR [1] , one-stage methods [7, 9, 23, 27, 30, 41, 48] predict all relationships at once by using an encoder-decoder architecture. These one-stage methods bypass pairing between all object proposals and directly decode <subject, predicate, object> triplets, which cost less parameters and computations. Integrating object detection and predicate prediction in the same step can present a quandary. While predicting the semantic label and spatial location of an entity may not necessitate any additional details from the triplet, identifying whether an entity corresponds to a specific relationship requires such information. PSGTR [56] utilizes a single decoder with holistic triplet queries, carrying features from different distributions (subject/object/predicate distributions), to predict the whole scene graph. These over-entangled features make it hard to model taskspecific representations. PSGFormer [56] , IterSGG [23] , TraCQ [9] devise multiple decoders to extract less entangled visual features by decoding subjects, objects, and predicates separately. In contrast, multiple decoders make coupled features shared in triplets hard to extract. We construct a simple but effective architecture to tackle this issue. We leverage a unified decoder taking task-specific queries as input, which not only extracts decoupled visual features separately but also facilitates intricate interactions within triplets. 
 
 Appendix:  


Decision: This paper presents a framework of unified decoder with task-specific queries for efficient scene graph generation. It receives one borderline accept and two borderline reject. The response well addresses the reviewers' concerns. All reviewers give borderline accept ratings after the rebuttal. The merits, including the new paradigm, less parameters, and good results, are well recognized by the reviewers. Please also incorporate the analysis and results in response into the camera-ready version. Overall, I think the current manuscript meets the requirements of this top conference.

Reviews: review: paper_summary: paper_summary: This work introduces a one-stage SGG formulation. The authors design a unified decoder with task-specific queries architecture, where task-specific queries generate decoupled visual features for subjects, objects, and predicates respectively. Experiments conducted on VG dataset demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The authors analyze the challenge of one-stage SGG methods and introduce a unified decoder with a task-specific queries architecture into the SGG task.
- The proposed method shows advantages on zero-shot and no-graph constraint SGG evaluation, indicating its ability to handle unseen relationships.
- The proposed method uses less parameters and achieves better results compared to one-stage SGG methods.
limitations: limitations: - Although this work has achieved promising results, it is difficult to say that “the proposed method has superior performance to both one-stage and two-stage methods”, since the authors have not comprehensively compared it with all two-stage methods, such as [10], which is cited but not compared in this work. The latest two-stage SGG method compared in this work was published in 2021. I strongly suggest that the authors use a more nuanced way to describe the performance.
- The representation of Algorithm 1 is unclear, making it difficult to understand its function. More detailed explanations of Algorithm 1 are needed.
- The overall presentation of the work requires significant enhancement. Especially in the method part, the motivations for some components are not clearly explained.
paper_summary: paper_summary: This paper proposes a new one-stage method, UniQ, for scene graph generation tasks. By adopting a unified decoder to extract features from subject-verb-object, UniQ efficiently addresses the weak correlation. It achieves this by using task-specific queries to model spatial locations independently and facilitating interaction within each triplet to model correlated semantic features. It not only significantly reduces computational costs but also improves prediction results
strength_weakness: strength_weakness: 1. Introduces a new paradigm for scene graph generation called Single Decoder with Task-Specific Queries (STS). By using a unified decoder for the subject-verb-object triplets, it not only significantly reduces computational costs but also improves prediction results.
2. Partially solves the challenge in one-stage methods arising from weak entanglement.
3. The STS paradigm appears to be simple and easy to transfer, providing reference for other methods in scene graph generation tasks.
limitations: limitations: 1. **Insufficient experiments**: The experimental section only includes simple comparisons and a small number of ablation experiments. The paper claims that the STS paradigm can simultaneously capture both coupled features shared within triplets and decoupled visual features, but there is no supporting evidence in the experimental section. Rational experiments can further substantiate the viewpoints of a thesis and make it more comprehensive.
2. **Lack of explanation for the STA and TTS paradigm methods:** Although there is a brief explanation of the STA and TTS paradigms in the third chapter, there are no specific references to the corresponding methods. Additionally, Table 1 only provides classification of encoders, lacking categorization and comparison of relevant paradigms.
3. **The transferability of the STS paradigm has not been discussed**. It is suggested to supplement experiments that demonstrate the transferability of the STS paradigm, which would also serve as evidence for its ability to capture both coupled features shared within triplets and decoupled visual features to some extent.
paper_summary: paper_summary: This paper introduces a novel approach to Scene Graph Generation (SGG), called UniQ, which aims to identify object entities and reason their relationships within images. Unlike prevailing two-stage methods, UniQ is a one-stage method that integrates learnable queries to jointly reason relational triplets. The paper addresses the challenge of weak entanglement in one-stage methods by introducing task-specific queries to model decoupled spatial locations and facilitate interaction within each triplet to model coupled semantic features. UniQ employs a unified decoder with task-specific queries for parallel triplet decoding, enabling detailed representation modeling for each sub-task while reducing parameter overhead compared to methods using multiple decoders. Experimental results on the Visual Genome dataset demonstrate that UniQ outperforms both one-stage and two-stage methods in terms of performance while utilizing fewer parameters.
strength_weakness: strength_weakness: Unified Decoder with Task-Specific Queries: UniQ introduces a unified decoder architecture with task-specific queries, allowing for detailed representation modeling for each sub-task. This approach enhances the flexibility and efficiency of scene graph generation.
Reduction in Parameter Overhead: Compared to methods utilizing multiple decoders, UniQ reduces parameter overhead. This reduction in parameters contributes to improved computational efficiency and model scalability.
Effective Handling of Weak Entanglement: UniQ effectively addresses the challenge of weak entanglement between entities and predicates by employing task-specific queries to model decoupled spatial locations and facilitate interaction within each triplet. This enables more accurate modeling of coupled semantic features.
limitations: limitations: The paper lacks attention to detail, for example, there are issues with the bold annotations in Table 1.
The improvement in results is not significant.
There is insufficient innovation in the method.