# Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation

Runfeng Qu, Ole Hall, Pia K Bideau, Julie Ouerfelli-Ethier, Martin Rolfs,

Klaus Obermayer, Olaf Hellwich

► To cite this version:

Runfeng Qu, Ole Hall, Pia K Bideau, Julie Ouerfelli-Ethier, Martin Rolfs, et al. Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation. WACV 2026 - IEEE/CVF Winter Conference on Applications of Computer Vision, Mar 2026, Tucson, Arizona, United States. hal-05467251

HAL Id: hal-05467251

https://hal.science/hal-05467251v1

Submitted on 20 Jan 2026

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2601.08728v1 [cs.CV] 13 Jan 2026

# Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation

Runfeng Qu¹,⁵ Ole Hall¹,⁵ Pia K Bideau³ Julie Ouerfelli-Ethier²,⁵

Martin Rolfs²,⁵ Klaus Obermayer¹,⁴,⁵ Olaf Hellwich¹,⁵

¹Technische Universität Berlin, Germany ²Humboldt Universität zu Berlin, Germany

³Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France

⁴Bernstein Center for Computational Neuroscience, Germany

⁵Science of Intelligence Research Cluster of Excellence, Germany

runfeng.qu@campus.tu-berlin.de

## Abstract

Scene Graph Generation (SGG) suffers from a long-tailed distribution, where a few predicate classes dominate while many others are underrepresented, leading to biased models that underperform on rare relations. Unbiased-SGG methods address this by implementing debiasing strategies, but often at the cost of spatial understanding—resulting in over-reliance on semantic priors. We introduce Salience-SGG, a novel framework featuring an Iterative Salience Decoder (ISD) that emphasizes triplets with salient spatial structures. To support this, we propose semantic-agnostic salience labels guiding ISD. Evaluations on Visual Genome, Open Images V6, and GQA-200 show that Salience-SGG achieves state-of-the-art performance and improves existing Unbiased-SGG methods in their spatial understanding as demonstrated by the Pair-wise Localization Average Precision. Code is available at: https://github.com/runfeng-q/Salience-SGG.

## 1. Introduction

For any given image, scene graph generation (SGG) models aim to detect a set of triplets in the form of {subject, predicate, object}. These triplets can be transformed into graph structures called scene graphs, where nodes indicate subjects and objects, while edges represent predicates. The primary challenge in SGG tasks is the long-tailed distribution problem. Unbalanced predicate samples result in predictions of SGG models dominated by a few frequent predicates (e.g. wearing, has), while undermining the rare predicates (e.g. lying on, flying in). Recent SGG studies focus on Unbiased Scene Graph Generation (Unbiased-SGG), which aims to create fine-grained scene graphs with more informative predicates by incorporating various debiasing methods such as re-weighting [2, 28], re-sampling [8, 26], and pseudo-labeling [9, 21, 51].

<figure><img src="image_2.png" /><figcaption>Figure 1. Salience-SGG. (a): IETrans [51] with standard debiasing, showing over-reliance on the semantic information, i.e. *coat* and *hanging from* (dashed lines). (b): Our salience-enhanced model favors spatially coherent triplets (bold).</figcaption></figure>

Despite the progress made in Unbiased-SGG studies, prevalent models do not demonstrate strong robustness to debiasing strategies. Improving performance on rare predicates often substantially reduces performance on frequent ones [26]. This phenomenon can be attributed to the model's preference for rare predicates, encouraged by debiasing strategies. For triplets such as {person, on, beach}, Unbiased-SGG models may incorrectly favor rare predicates (e.g., standing on). This example illustrates the trade-off that undermines performance on common predicates. However, we identify that debiasing strategies disrupt spatial understanding of entity pairs. Specifically, spatially
loosely associated entity pairs with rare predicates are detected with high confidences, as illustrated in Figure 1 (a). These spatially incoherent pairs, in turn, suppress other entity pairs with more salient spatial structures to be recalled, thereby further decreasing overall performance. The reason for this issue is that the prevailing debiasing methods rely solely on the frequency statistics of predicate and entity semantics. Applying these methods to detection tasks may result in a model that relies significantly on high-level semantic information of entity pairs alone, while disregarding low-level spatial information for global detection. As a consequence, Unbiased-SGG models may not capture the triplets in salient spatial structures. We refer to this issue as **salience insensitivity**, given the lesser sensitivity to the spatial information. To analyze salience insensitivity systematically, we introduce a measurement that evaluates the ability of SGG models to capture salient triplet structures (see Section 4.4).

To address the problem of salience insensitivity, we propose a novel framework that integrates a triplet salience estimation process into the Unbiased-SGG process. The salience estimation is achieved with the introduction of triplet salience labels and an Iterative Salience Decoder (ISD). The triplet salience labels are expressed by a mask in the form of {0, 1}, which indicates whether any two detected entities form a salient triplet that spatially matches the ground-truth triplets. Our triplet salience labels and ISD stand in contrast to existing approaches [17, 18, 44], which directly address the corresponding annotated triplets involving semantic content to create top-down binary labels. We generate bottom-up labels that consider only the low-level spatial information of the ground-truth triplets. The rich spatial features contained within these salience labels provide ISD with significant guidance for emphasizing the spatial configuration and globally exploring the triplet candidates in most important spatial structures within scenes, as shown in Figure 1 (b). Furthermore, such semantic-agnostic labels prevent the salience estimation from becoming a prediction of whether each triplet is positive. ISD formulates the triplet salience estimation as a salience message-passing process. To effectively capture the important spatial structures in scenes, we propose two types of enhanced attention layers to guide the message-passing. In summary, our contributions are as follows:

* We propose a novel framework for Unbiased-SGG that incorporates salience estimation to directly address the **salience insensitivity** caused by debiasing strategies.
* We introduce **bottom-up salience labels**, and **enhanced attention layers** to facilitate the exploration of the salient spatial structures of the visual scene by an iterative salience decoder.
* We provide extensive experimental results on three challenging dataset demonstrating that our proposed framework improves the performance on the Unbiased-SGG task by enhancing salience sensitivity.

## 2. Related Work

**Scene Graph Generation.** Early work on SGG follows the two-stage framework which contains an object detector such as Faster R-CNN [36] to propose entities. A predicate predictor is performed on the entity pairs to predict the predicates between them [33, 52]. Subsequently, hierarchical structures are proposed [5, 46, 47] to jointly update entity pairs and predicates. [45, 50] employ Long Short-Term Memory (LSTM) [13] and Attention Mechanism [41] to aggregate global context for entity updating and predicate prediction. Inspired by the Detection Transformer (DETR) [1], several one-stage SGG models [7, 19, 40] are developed achieving SGG by maintaining a set of learnable queries.

Recall (R@K) is the standard metric for evaluating SGG models, but it fails to reflect performance on rare predicates due to the long-tailed distribution. To address this, mean recall (mR@K) [38] equally weigh all predicate classes to better assess generalization. To achieve high mR@K, [39] propose an Unbiased-SGG model with causal effects. BGNN [26] re-balances the dataset using a bi-level re-sampling strategy. Various studies [2, 19, 37] utilize the re-weighting losses to adjust the decision boundary for each individual predicate. Finally, inspired by knowledge distillation, many recent studies [48, 49] focus on creating pseudo-labels based on predictions of a pre-trained SGG model to enrich annotations.

Current Unbiased-SGG models focus on optimizing mR@K, often overlooking the negative impact of debiasing strategies on R@K. To balance both, IETRans [51] propose F@K as an overall performance, which is the harmonic mean of R@K and mR@K. Mg-RMPN [44] discover that using a binary classification module to select likely triplets may greatly benefit overall performance. However, the binary classification module is trained using binary labels that are consistent with the annotated triplets. The high correlation between these labels and triplet semantic information may not sufficiently support the model exploring salient spatial structure. Our salience estimation module is trained using labels derived solely from the spatial information of ground-truth graphs. These semantic-agnostic labels guide the module to focus purely on spatial patterns.

**Salience-Guided Object Detector.** Salience estimation is commonly used to improve object detection tasks. Salience-DETR[14] and Focus-DETR [54] integrate feature-level salience modules into DETR to reduce the computational burden. Yolo-sg [10] creates pixel-level salience labels to help the detector focus on important regions of images. Inspired by these studies, we propose Salience-SGG, which encourages exploration of important spatial structures by leveraging triplet-level salience labels.
### 3. Method

SGG tasks aim to detect a scene graph represented as $G = \{\hat{\mathcal{V}}, \hat{\mathcal{E}}\}$, where $\hat{\mathcal{V}}$ denotes the set of entities and $\hat{\mathcal{E}}$ is the set of triplets. Each entity $\hat{v}_x \in \hat{\mathcal{V}}$ contains information about its category and location, i.e. $\hat{v}_x = \{\hat{b}_x, \hat{c}_x\}$. In general, each triplet $\hat{e}_k \in \hat{\mathcal{E}}$ is represented in the form of $\{\hat{s}_k, \hat{p}_k, \hat{o}_k\}$, where $\hat{s}_k, \hat{o}_k$ refer to the entity indices in $\hat{\mathcal{V}}$ of the corresponding subject and object, while $\hat{p}_k$ indicates the predicate category connecting the entity pair. To address salience insensitivity, we reformulate the triplet as $\{\hat{s}_k, \hat{p}_k, \hat{o}_k, \hat{m}_k\}$, where $\hat{m}_k$ is a binary label indicating triplet salience.

The overall framework of our approach consists of an Unbiased-SGG module and a salience decoder module, i.e. Iterative Salience Decoder (ISD), as depicted in Figure 2 (a). To investigate the impact of ISD on various debiasing strategies, we provide both one-stage and two-stage methods. The implementation of the two-stage method is achieved through the integration of ISD into the extant two-stage Unbiased-SGG models (see Section 4.3). Unless specified otherwise, the following sections refer to our one-stage method Salience-SGG. In the Unbiased-SGG part, a deformable DETR [55] is employed as an object detector to propose a set of entities $V = \{v_i\}_{i=1}^{N_e}$, where $N_e$ is the number of detected entities. Each detected entity $v_i$ contains its bounding box location and category distribution, denoted as $B = \{b_i\}_{i=1}^{N_e} \in \mathbb{R}^{N_e \times 4}$, $C = \{c_i\}_{i=1}^{N_e} \in \mathbb{R}^{N_e \times N_e}$, respectively, where $N_e$ is the total number of entity categories. In addition, we extract the queries from the last decoder layer of the deformable DETR as entity features $Q = \{q_i\}_{i=1}^{N_e} \in \mathbb{R}^{N_e \times d}$, where $d$ is the dimension of these features. The object detector is followed by a predicate decoder to predict the predicate connecting each detected entity pair, denoted as $G \in \mathbb{R}^{N_e \times N_e \times N_p}$ (see Section 3.1). Following previous work [17, 44], the entity labels and predicates labels for the detected entities are obtained by applying Hungarian Matching [23]. Entity labels are denoted as $V' = \{v'_i\}_{i=1}^{N_e}$, where $v'_i = \{b'_i, c'_i\}$, while the predicate labels are indicated as $G' \in \mathbb{R}^{N_e \times N_e \times N_p}$, where $N_p$ is the category number of predicates. Leveraging the ground-truth graph $G$, we construct the triplet salience labels $M' \in \mathbb{R}^{N_e \times N_e}$ as described in Section 3.2. Ultimately, $G, C, B, Q$ are sent to the ISD to estimate the triplet salience intensity $M \in \mathbb{R}^{N_e \times N_e}$ and approximate $M'$ (for details see Section 3.3).

### 3.1. Predicate Decoder

Our Salience-SGG uses a lightweight predicate decoder to reason about predicates directly. Given the entity information, the predicate features $R = \{r_{ij}\} \in \mathbb{R}^{N_e \times N_e \times D}$ are created as written in Equation (1). Subsequently, $R$ is transmitted to a MLP classifier to obtain $G$ without any further

refinement process.

$$r_{ij} = [b_i, \text{GloVe}_i, q_i, b_j, \text{GloVe}_j, q_j] \quad (1)$$

where $[\cdot, \cdot]$ denotes the operation of concatenation, and Glove indicates the glove embedding [35] of the predicted entity category from the object detector.

### 3.2. Triplet Salience Label

The triplet salience label is created to guide our ISD to explore the significant spatial structures related to the annotated graphs, thereby maintaining salience sensitivity during the Unbiased-SGG process. In order to preserve the geometric relationships between subjects and objects, we use the entity-level locations of the annotated triplets as a criterion to assign salience labels. A given triplet candidate is considered salient when both the subject and object boxes sufficiently overlap with the subject and object of any one of the ground-truth triplets at the same time. Particularly, the locations of subjects and objects in the ground-truth triplets set are denoted by $\{b'_{s_k}, b'_{o_k}\}_{k=1}^K$, where $K$ is the number of annotated triplets. $\{b_i\}_{i=1}^{N_e}$ refers to the bounding boxes of the detected entities. A subject salience matrix $\hat{M}^{sub} \in \mathbb{R}^{N_e \times K}$ is constructed by computing the pairwise Intersection over Unions (IoUs) between $\{b_i\}$ and $\{b'_{s_k}\}$. Similarly, an object salience matrix $\hat{M}^{obj} \in \mathbb{R}^{N_e \times K}$ reflects the pairwise IoUs between $\{b_i\}$ and $\{b'_{o_k}\}$. The salience labels of detected entity pairs are determined as follows:

$$M'_{ij} = \begin{cases} 1, & \exists a, \hat{M}^{sub}_{ia} \ge T \ \& \ \hat{M}^{obj}_{ja} \ge T \\ 0, & \text{otherwise} \end{cases} \quad (2)$$

Due to the omission of semantic information regarding entity pairs during salience label assignment, a significant number of triplet candidates with low semantic information or incoherent semantics are considered salient. This kind of label forces ISD to rely heavily on spatial information when making estimations.

### 3.3. Iterative Salience Decoder

The structure of ISD is illustrated in Figure 2 (b). Using the detected entities $Q, B$, and $C$ as well as the predicate matrix $G$, ISD estimates the salience of each entity pair $M \in \mathbb{R}^{N_e \times N_e}$. Because each image contains $N_e \times N_e$ entity pairs, globally estimating the salience by maintaining $N_e^2$ triplet salience features brings significant computational cost. In this study, we introduce subject salience queries $Q^{sub} \in \mathbb{R}^{N_e \times d}$ and object salience queries $Q^{obj} \in \mathbb{R}^{N_e \times d}$ to preserve subject and object salience information. $Q^{sub}$ and $Q^{obj}$ are updated through a message-passing process which is achieved by multiple salience decoder layers. Each layer of our iterative salience decoder contains two branches for updating $Q^{sub}$ and $Q^{obj}$ respectively (see Figure 2
<figure><img src="image_3.png" /><figcaption>Figure 2. (a) Overview of our full network architecture. Our framework consists of an Unbiased-SGG module and a model-agnostic iterative salience decoder (ISD) module. (b) Subject salience features and object salience features are updated through multiple salience decoder layers in ISD. (c) Details of the message-passing process of the salience decoder layer. G-ESA and P-ECA correspond to Geometry Enhanced Self-Attention and Predicate Enhanced Cross-Attention, respectively.</figcaption></figure>

(c)). Each branch consists of a Geometry Enhanced Self-Attention (G-ESA), a Predicate Enhanced Cross-Attention (P-ECA) and a Feed Forward Network (FFN). The final triplet salience M is estimated by fusing $Q^{sub}$ and $Q^{obj}$.

**Subject and Object Salience Queries Initialization.**

We denote the updated subject and object salience queries after l-th salience decoder layer as $Q_l^{sub}$ and $Q_l^{obj}$ respectively. Since ISD is a module to model the salience message-passing process, the initial salience queries $Q_0^{sub}$ and $Q_0^{obj}$ reflect the salience information of individual entities as determined by the object detector. Instead of creating additional learnable vectors, we design a set of entity salience features $Q^{ent} \in \mathbb{R}^{N_e \times d}$ with the entity category distributions C and features Q obtained from object detection as follows:

$$Q^{ent} = Q + \text{proj}(C) \quad (3)$$

where proj indicates a linear projector to map the category distribution to the same dimension as the entity features. Subsequently, two linear projectors are applied to $Q^{ent}$ to return $Q_0^{sub}$ and $Q_0^{obj}$.

**Geometry Enhanced Self-Attention.** G-ESAs are adopted to conduct intra-subject and intra-object salience message-passing. Before introducing our G-ESA, let us revisit the process of the regular self-attention process. Given the subject salience queries $Q_l^{sub}$, the regular self-attention first computes an attention matrix $A_{self} \in \mathbb{R}^{N_e \times N_e \times H}$, where H is the head number of self-attention:

$$A_{self} = \frac{\text{Que}(Q_l^{sub}) \cdot \text{Key}(Q_l^{sub})^T}{\sqrt{d}} \quad (4)$$

The queries are updated according to $A_{self}$ as follows:

$$Q_l^{sub} = Q_l^{sub} + \text{SoftMax}(A_{self}) \cdot \text{Val}(Q_l^{sub}) \quad (5)$$

The regular self-attention layers control the information flow solely based on the subspace similarity among queries as shown in Equation (4). According to the definition of our triplet salience described in Section 3.2, subjects or objects exhibiting high overlap areas should be characterized by similar salience. Inspired by [11, 15], we introduce G-ESA to reinforce the information interaction between the queries determined by their spatial overlapping. Specifically, we calculate the IoUs between every entity pairs forming $\mathcal{U} \in \mathbb{R}^{N_e \times N_e}$. Subsequently, $\mathcal{U}$ is projected to equal the channel to the head number H of the self-attention layer. Ultimately, the projected $\mathcal{U}$ is incorporated into the attention matrix $A_{self}$ in order to manipulate the information flow as below:

$$A_{self} = \text{Relu}(\text{MLP}(\mathcal{U})) + \frac{\text{Que}(Q_l^{sub}) \cdot \text{Key}(Q_l^{sub})^T}{\sqrt{d}} \quad (6)$$

The same operation is used for intra-object salience message-passing to obtain $Q_{l'}^{obj}$. Note that, to ensure training stability, a ReLU activation function is applied after the MLP output layer to suppress negative values.

**Predicate Enhanced Cross-Attention.** we introduce P-ECA to conduct inter-subject-object salience message-passing. Similarly to G-ESA, the interactions between subjects and objects are enhanced in accordance with the predicted predicates G. For instance, in the subject branch, subject salience queries are updated as follows:

$$A_{cross} = \text{Relu}(\text{MLP}(G)) + \frac{\text{Que}(Q_{l'}^{sub}) \cdot \text{Key}(Q_{l'}^{obj})^T}{\sqrt{d}} \quad (7)$$
$$Q_{l''}^{sub} = Q_{l}^{sub} + \text{SoftMax}(A_{cross}) \cdot \text{Val}(Q_{l}^{obj}) \quad (8)$$

The operation is identical for updating object salience queries, with the exception that G is first transposed to G^T. After P-ECAs, FFNs performed on $Q_{l''}^{sub}$ and $Q_{l''}^{obj}$ lead to the two following outputs: $Q_{l+1}^{sub}$ and $Q_{l+1}^{obj}$.

**Iterative Triplet Salience Refinement.** As demonstrated by our empirical results, an iterative refinement process yields superior results in comparison to directly considering the predictions after each salience decoder layer as the triplet salience. To reduce computational costs and in line with the salience message-passing process in the salience decoder layer, linear attention is utilized to fuse the subject and object salience queries. These fused queries are added to the triplet salience predicted from the previous layer to form the refined triplet salience as follows:

$$M_{l+1} = \text{sigmoid}(\text{inverse\_sigmoid}(M_l) + \frac{Q_{l+1}^{sub} \cdot (Q_{l+1}^{obj})^T}{\sqrt{d}}) \quad (9)$$

where sigmoid refers to the sigmoid activation function. For $l = 0$, the initial salience matrix $M_0$ is a zero matrix.

## 3.4. Training and Inference

For the entire training process of our Salience-SGG, the overall loss function $\mathcal{L}$ is expressed as follows:

$$\begin{aligned} \mathcal{L} &= \mathcal{L}_{ent} + \mathcal{L}_{salience} + \mathcal{L}_{pre} \\ \mathcal{L}_{ent} &= \mathcal{L}_{ent}^{cls} + \mathcal{L}_{ent}^{box} + \mathcal{L}_{ent}^{GIoU} \end{aligned} \quad (10)$$

where $\mathcal{L}_{ent}$ is the entity loss, which is identical to the loss function of the deformable DETR [55]. We compute focal loss [29] between $M$ and $M'$ as the salience loss $\mathcal{L}_{salience}$. Following [44], we employ re-weighting loss called seesaw loss [42] as the predicate loss $\mathcal{L}_{pre}$ in our Salience-SGG to solve the long-tailed distribution problem.

For inference, we follow [17], except that triplet candidates are sorted by involving our salience scores $M$.

# 4. Experiments

## 4.1. Experimental Setup

This section provides descriptions of datasets, evaluation metrics, and implementation details.

**Datasets.** We conduct experiments on three datasets: Visual Genome (VG) [22] contains 57,723 training, 5,000 validation, and 26,446 test images. It comprises of 150 entity and 50 predicate categories.

Open Image V6 (OIV6) [24] contains 126,368 training, 1,813 validation, and 5,322 test images, totaling 133,503 images across 601 entity and 30 predicate categories.

GQA-200 [16] consists of 57,623 training, 5,000 validation, and 8,208 test images. After processing, GQA-200 is composed of 200 entity and 100 predicate categories.

For all three datasets, we follow the pre-processing in [44].

**Evaluation Metrics.** For each dataset, we present the performance on the scene graph detection (SGDet) task. In the case of the VG and GQA-200 datasets, we follow previous work [51] reporting Recall@K (R@K), mean Recall@K (mR@K), and F@K to measure the overall performance. For the OIV6 dataset, we include micro-Recall@50 (micro-R@50), weighted mean AP of triplets (wmAPrel), weighted mean AP of phrase (wmAPphr), and the final score. The final score is calculated by score = 0.2 × Recall@50 + 0.4 ×wmAPrel + 0.4 ×wmAPphr. Furthermore, mR@K is reported to exhibit the capability of methods to solve the biased problem of methods.

**Implementation Details.** For all three datasets, we follow the same setup. A deformable DETR with backbone ResNet-50 [12] is pre-trained for 25 epochs. The number of entity queries, $N_e$, is set to 200. Subsequently, the pre-trained object detector is frozen during the joint training of the predicate decoder and ISD. In the Salience-SGG, the loss ratio between salience loss $\mathcal{L}_{salience}$ and predicate loss $\mathcal{L}_{pre}$ is 1 : 1. The re-weighting loss proposed in [42] is employed as a debiasing strategy. By default, the hyperparameters $\alpha$ and $\beta$ in the loss function are set to 1.0 and 0.2, respectively. The hyper-parameters $\mathcal{T}$ and $L$ introduced by ISD are 0.6 and 4. The training epochs of predicate decoder and ISD are 13, 8, 12 for VG, OIV6, GQA-200, respectively. We use AdamW [32] as optimizer, with an initial learning rate of $10^{-4}$ and a weight decay of $10^{-4}$. All training is performed on two NVIDIA RTX A6000 GPUs with a batch size of 16, using a fixed random seed of 42. Additional training details are provided in the **suppl. material**.

## 4.2. Comparison to SOTA Methods

We compare our approach with both two-stage methods (e.g. Motifs [50], BGNN [26], DRM [25], SHA [8]) and one-stage methods (e.g. Mg-RMPN [44], SSR-CNN [40], SpeaQ [20], Hydra-SGG [3]). Since our works are based on Unbiased-SGG methods, we focus on comparing to results in which at least one debiasing strategy is applied.

**VG.** In Table 1, we report the comparison results on the VG test dataset. The Salience-SGG model demonstrates superior holistic performance F@K compared to all existing SGG models which incorporate debiasing strategies. In the context of one-stage models, the Salience-SGG model achieves second best mR@20 with 12.8 and the SOTA mR@50, and mR@100 of 18.0 and 21.6, respectively. This is notable given that our model utilizes a significantly reduced number of parameters (77.7M) compared to the SSR-CNN model [40] (274.3M). In contrast to the Mg-RMPN approach [44], which also employs seesaw loss as the debiasing strategy, our approach significantly improves mR@K. This demonstrates that our salience estimation process enhances addressing the long-tailed distribution problem. Among two-stage models, DRM [25] exhibits notable
<table><caption>Table 1. Comparison with state-of-the-art methods evaluated on the VG test dataset. The methods are divided into two groups. The best and second-best results in each group are indicated with **bold** and **underlined** text, respectively. ‘*’ denotes the performance without any debiasing strategy. ‘†’ indicates the methods are combined with MOTIFS [50].</caption><thead><tr><th>Method</th><th>Backbone</th><th># params (M)</th><th>R@20</th><th>R@50</th><th>R@100</th><th>mR@20</th><th>mR@50</th><th>mR@100</th><th>F@20</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td colspan="11" style="text-align: center; font-weight: bold;">two-stage models</td></tr><tr><td>Motifs* [50] [CVPR2018]</td><td>ResNeXt101-FPN</td><td>369.9</td><td>25.5</td><td>32.8</td><td>37.2</td><td>5.0</td><td>6.8</td><td>7.9</td><td>8.4</td><td>11.3</td><td>13.0</td></tr><tr><td>TDE† [39] [CVPR2020]</td><td>ResNeXt101-FPN</td><td>369.9</td><td>11.9</td><td>16.6</td><td>20.2</td><td>6.6</td><td>8.9</td><td>11.0</td><td>8.5</td><td>11.7</td><td>14.3</td></tr><tr><td>BGNN [26] [CVPR2021]</td><td>ResNeXt101-FPN</td><td>341.9</td><td>23.3</td><td>31.0</td><td>35.8</td><td>7.5</td><td>10.7</td><td>12.6</td><td>11.3</td><td>15.9</td><td>18.6</td></tr><tr><td>SHA [8] [CVPR2022]</td><td>ResNeXt101-FPN</td><td>—</td><td>—</td><td>14.9</td><td>18.2</td><td><strong>14.2</strong></td><td><strong>17.9</strong></td><td><strong>20.9</strong></td><td>—</td><td>16.3</td><td>19.5</td></tr><tr><td>IETrans† [51] [ECCV2022]</td><td>ResNeXt101-FPN</td><td>369.9</td><td>17.5</td><td>23.5</td><td>27.3</td><td>11.0</td><td>15.7</td><td>18.2</td><td><strong>13.5</strong></td><td><strong>18.8</strong></td><td><strong>21.8</strong></td></tr><tr><td>ST-SGG† [21] [ICLR2024]</td><td>ResNeXt101-FPN</td><td>—</td><td>—</td><td>26.7</td><td>30.7</td><td>—</td><td>11.6</td><td>14.2</td><td>—</td><td>16.2</td><td>19.4</td></tr><tr><td>DRM [25] [CVPR2024]</td><td>ResNeXt101-FPN</td><td>—</td><td>—</td><td>19.0</td><td>22.9</td><td>—</td><td><strong>20.4</strong></td><td><strong>24.1</strong></td><td>—</td><td><strong>20.8</strong></td><td><strong>23.5</strong></td></tr><tr><td>SRD† [34] [WACV2025]</td><td>ResNeXt101-FPN</td><td>—</td><td>—</td><td>—</td><td>—</td><td><strong>13.5</strong></td><td>17.9</td><td>20.6</td><td>—</td><td>—</td><td>—</td></tr><tr><td colspan="11" style="text-align: center; font-weight: bold;">one-stage models</td></tr><tr><td>SSR-CNN [40] [CVPR2022]</td><td>ResNeXt101-FPN</td><td>274.3</td><td>18.4</td><td>23.3</td><td>26.5</td><td><strong>13.5</strong></td><td><strong>17.9</strong></td><td><strong>21.4</strong></td><td><strong>15.6</strong></td><td>20.2</td><td>23.7</td></tr><tr><td>EGTR [17] [CVPR2024]</td><td>ResNet50</td><td>42.5</td><td>22.4</td><td>28.2</td><td>31.7</td><td>8.8</td><td>14.0</td><td>18.3</td><td>12.6</td><td>18.7</td><td>23.2</td></tr><tr><td>Mg-RMPN [44] [ECCV2024]</td><td>ResNet50</td><td>—</td><td>22.5</td><td>29.1</td><td>33.5</td><td>10.3</td><td>14.4</td><td>17.3</td><td>14.1</td><td>19.3</td><td>22.8</td></tr><tr><td>SpeaQ [20] [CVPR2024]</td><td>ResNet101</td><td>93.4</td><td>25.1</td><td>32.1</td><td>35.5</td><td>10.1</td><td>15.1</td><td>17.6</td><td>14.4</td><td>20.5</td><td>23.5</td></tr><tr><td>Hydra-SGG [3] [ICLR2025]</td><td>ResNet50</td><td>67.6</td><td>21.9</td><td>28.6</td><td>33.4</td><td>10.3</td><td>15.9</td><td>19.4</td><td>14.0</td><td><strong>20.5</strong></td><td><strong>24.7</strong></td></tr><tr><td><strong>Salience-SGG (Ours)</strong></td><td>ResNet50</td><td>77.7</td><td>21.9</td><td>28.8</td><td>33.4</td><td><strong>12.8</strong></td><td><strong>18.0</strong></td><td><strong>21.6</strong></td><td><strong>16.2</strong></td><td><strong>22.1</strong></td><td><strong>26.2</strong></td></tr></tbody></table>

<table><caption>Table 2. Comparison with SOTA methods evaluated on OIV6.</caption><thead><tr><th>Method</th><th>mR@50</th><th>micro-R@50</th><th>wmAPrel</th><th>wmAPphr</th><th>score</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">two-stage models</td></tr><tr><td>GPS-Net [30] [CVPR2020]</td><td>35.2</td><td>74.8</td><td>32.9</td><td>34.0</td><td>41.7</td></tr><tr><td>BGNN [26] [CVPR2021]</td><td>40.5</td><td>75.0</td><td>33.5</td><td>34.2</td><td>42.1</td></tr><tr><td>RU-Net [31] [CVPR2022]</td><td>—</td><td>76.9</td><td>35.4</td><td>34.9</td><td>43.5</td></tr><tr><td>PE-Net [53] [CVPR2023]</td><td>—</td><td>76.5</td><td><strong>36.6</strong></td><td><strong>37.4</strong></td><td>44.9</td></tr><tr><td>DRM [25] [CVPR2024]</td><td>—</td><td>75.9</td><td><strong>40.5</strong></td><td><strong>41.4</strong></td><td><strong>47.9</strong></td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">one-stage models</td></tr><tr><td>SGTR [27] [CVPR2022]</td><td>42.6</td><td>59.9</td><td>37.0</td><td>38.7</td><td>42.3</td></tr><tr><td>SSR-CNN [40] [CVPR2022]</td><td>42.8</td><td>76.7</td><td>41.5</td><td>43.6</td><td>49.4</td></tr><tr><td>RelTR [4] [TPAMI2023]</td><td>—</td><td>71.7</td><td>37.2</td><td>37.5</td><td>43.0</td></tr><tr><td>EGTR [17] [CVPR2024]</td><td>—</td><td>75.0</td><td>42.0</td><td>41.9</td><td>48.6</td></tr><tr><td>Mg-RMPN [44] [ECCV2024]</td><td><strong>45.5</strong></td><td>74.2</td><td>35.5</td><td>36.4</td><td>43.6</td></tr><tr><td>Hydra-SGG [3] [ICLR2025]</td><td>—</td><td>76.1</td><td>42.8</td><td>44.3</td><td>50.1</td></tr><tr><td><strong>Salience-SGG (Ours)</strong></td><td><strong>48.0</strong></td><td>78.1</td><td><strong>45.6</strong></td><td><strong>44.9</strong></td><td><strong>51.8</strong></td></tr></tbody></table>

<table><caption>Table 3. Comparison with SOTA methods evaluated on GQA-200. ‘†’ denotes models combined with GCL as proposed in [8].</caption><thead><tr><th>Method</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td colspan="7" style="text-align: center; font-weight: bold;">two-stage models</td></tr><tr><td>Motifs† [50] [CVPR2018]</td><td>18.5</td><td>21.8</td><td>16.8</td><td>18.8</td><td><strong>17.6</strong></td><td><strong>20.2</strong></td></tr><tr><td>VCTREE† [38] [CVPR2019]</td><td>17.6</td><td>20.7</td><td>15.6</td><td>17.8</td><td>16.5</td><td>19.1</td></tr><tr><td>SHA† [8] [CVPR2022]</td><td>14.8</td><td>17.9</td><td><strong>17.8</strong></td><td><strong>20.1</strong></td><td>16.2</td><td>18.9</td></tr><tr><td>VETO [37] [ICCV2023]</td><td>26.1</td><td>29.0</td><td>7.0</td><td>8.1</td><td>11.0</td><td>12.7</td></tr><tr><td>DRM [25] [CVPR2024]</td><td>18.6</td><td>21.7</td><td><strong>18.9</strong></td><td><strong>21.0</strong></td><td><strong>18.7</strong></td><td><strong>21.3</strong></td></tr><tr><td>RA-SGG [48] [AAAI2025]</td><td>16.3</td><td>19.0</td><td>12.9</td><td>15.0</td><td>14.4</td><td>16.8</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">one-stage models</td></tr><tr><td>Pair-Net [43] [PAMI2024]</td><td>20.2</td><td>23.4</td><td>10.6</td><td>12.6</td><td>13.9</td><td>16.4</td></tr><tr><td>Mg-RMPN [44] [ECCV2024]</td><td>23.2</td><td>25.7</td><td><strong>12.8</strong></td><td>14.5</td><td><strong>16.5</strong></td><td>18.5</td></tr><tr><td>Hydra-SGG [3] [ICLR2025]</td><td>22.8</td><td>26.5</td><td>12.7</td><td><strong>15.9</strong></td><td>16.3</td><td>19.9</td></tr><tr><td><strong>Salience-SGG (Ours)</strong></td><td>23.6</td><td>26.6</td><td><strong>16.2</strong></td><td><strong>18.4</strong></td><td><strong>19.2</strong></td><td><strong>21.7</strong></td></tr></tbody></table>

mR@50 and mR@100 scores of 20.4 and 24.1, respectively. However, weak robustness to the debiasing strategy introduces substantial negative impacts on R@K by preventing it from achieving a superior overall performance F@K. We provide specific comparisons to evaluate the robustness (see Section 4.3). In addition, a more extensive comparison with additional methods is provided in **suppl. material**.

**OIV6.** For the OIV6 dataset, a different set of evaluation metrics is applied. Since micro-R@50, wmArel, wmAphr are more sensitive to the performance on frequent predicates, most existing works, except Mg-RMPN [44], omits debiasing strategies to evaluate their models, resulting in better results. In Salience-SGG, the debiasing strategy is maintained to illustrate the robustness of our framework to the debiasing strategies. In addition, we follow [27] and report mR@K to reflect the performance on rare predicates. As demonstrated in Table 2, our approach sets new SOTA standards on all evaluation metrics. It is imperative to emphasize the result on wmArel, which is employed to assess the AP of the predicted triplets, wherein both the subject and object boxes exhibit an IoU of at least 0.5 with the ground-truth. Our approach outperforms the second best result from Hydra-SGG [3] by a margin of 2.8 on wmArel. This may stem from the relative geometric relationships encoded in the triplet salience labels.

**GQA-200.** Table 3 illustrates the comparison results on the GQA-200 test dataset. As with the VG dataset, the Salience-SGG sets new SOTA F@k scores of 19.2 and 21.7, respectively. This outcome surpasses the second best results from SHA [8] and DRM [25]. Furthermore, Salience-SGG outperforms one-stage models Mg-RMPN [44] and Hydra-SGG [3] on all evaluation metrics.

### 4.3. Complementary Comparisons

This section presents additional evaluations on the VG test set to assess (1) whether Salience-SGG exhibits superior robustness to debiasing compared to Unbiased-SGG models, (2) the compatibility of our salience estimation framework with alternative debiasing strategies, and (3) the performances of our ISD on predicates at varying frequencies.

**Robustness Evaluation.** Debiasing strategies can shift
<figure><img src="image_4.png" /><figcaption>Figure 3. Comparison of the top-5 detected triplets. For each image, the colors indicate triplet identifications; identical colors before and after re-ranking refer to the same triplets.</figcaption></figure>

the trade-off between mR@K and R@K. When two approaches differ greatly in mR@K, F@K alone may not reflect their robustness. To assess robustness, we train Salience-SGG on VG with varying seesaw loss hyperparameters, yielding results across different mR@K values. Since mR@K is more sensitive to $\beta$, we vary $\beta$ while keeping $\alpha$ fixed. For a fair comparison, we select the Salience-SGG result with the closest mR@K, so differences in F@K more clearly reflect robustness to debiasing. Table 4 reports Salience-SGG with different $\beta$ values and three strong baselines. Comparing Salience-SGG $\beta$ = 0.35, 0.25, and 0.125 with DRM [25], SHA [8], and SpeaQ [20], respectively, highlights Salience-SGG's robustness to debiasing. Benefiting from our salience estimation process, our Salience-SGG set new SOTA mR@50 and mR@100 scores with 21.4 and 25.3 in the case where $\beta$ = 0.5.

### ISD Compatibility with Other Debiasing Strategies.

To examine the compatibility of triplet salience estimation with diverse debiasing strategies, the ISD is integrated into the pre-trained TDE [39] and IETTrans [51]. The original detection set is re-ranked by involving ISD salience scores. Table 5 shows that our salience estimation improves both TDE and IETTrans, demonstrating the ISD module's broad compatibility with debiasing strategies. We also provide the qualitative comparisons for TDE versus TDE+ISD, and IETTrans versus IETTrans+ISD. In order to provide a comprehensive understanding, we visualize the top-5 detections from each models as illustrated in Figure 3. For both models, the spatial incoherent triplets such as {man, sitting-on, chair}, {man, behind, table} are removed after re-ranking. The triplet {man, with, hair} detected by IETTrans and IETTrans+ISD shows that re-ranking is not simply the result of lowering confidences for rare predicates. For more qualitative comparisons please see **suppl. material**.

### Group-wise Performance Comparison.
Following [6], predicates are grouped into head, body, and tail based on training frequency, and the performance of each group is measured by the average mR@100. Table 6 shows that integrating ISD significantly improves TDE and IETTrans on the head and body groups. Although IETTrans+ISD scores are lower on the tail group, overall performance is still improved. One-stage Salience-SGG achieves SOTA on head

<table><caption>Table 4. Robustness to debiasing strategies evaluation.</caption><thead><tr><th>Method</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td>DRM [25]</td><td>19.0</td><td>22.9</td><td>20.4</td><td>24.1</td><td>20.8</td><td>23.5</td></tr><tr><td>SHA [8]</td><td>14.9</td><td>18.2</td><td>17.9</td><td>20.9</td><td>16.3</td><td>19.5</td></tr><tr><td>SpeaQ [20]</td><td>32.1</td><td>35.5</td><td>15.1</td><td>17.6</td><td>20.5</td><td>23.5</td></tr><tr><td>Salience-SGG (β = 0.5)</td><td>20.3</td><td>24.3</td><td>21.4</td><td>25.3</td><td>20.8</td><td>24.8</td></tr><tr><td>Salience-SGG (β = 0.35)</td><td>24.0</td><td>28.4</td><td>20.4</td><td>24.4</td><td>22.0</td><td>26.2</td></tr><tr><td>Salience-SGG (β = 0.25)</td><td>26.8</td><td>31.4</td><td>19.2</td><td>22.9</td><td>22.4</td><td>26.5</td></tr><tr><td>Salience-SGG (β = 0.125)</td><td>32.2</td><td>36.2</td><td>15.7</td><td>19.0</td><td>21.1</td><td>24.9</td></tr></tbody></table>

<table><caption>Table 5. Results of combing ISD with existing two-stage Unbiased-SGG models.</caption><thead><tr><th>Method</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td>TDE [39]</td><td>16.6</td><td>20.2</td><td>8.9</td><td>11.0</td><td>11.7</td><td>14.3</td></tr><tr><td>TDE+ISD</td><td>19.9</td><td>24.4</td><td>10.5</td><td>12.4</td><td>13.6</td><td>16.4</td></tr><tr><td>IETTrans [51]</td><td>23.5</td><td>27.3</td><td>15.7</td><td>18.2</td><td>18.8</td><td>21.8</td></tr><tr><td>IETTrans+ISD</td><td>27.6</td><td>32.5</td><td>15.9</td><td>18.6</td><td>20.2</td><td>23.7</td></tr></tbody></table>

<table><caption>Table 6. Group-wise performance comparison</caption><thead><tr><th>Model</th><th>Head (16)</th><th>Body (17)</th><th>Tail (17)</th><th>mR@100</th></tr></thead><tbody><tr><td>DT2-ACBS [6]</td><td>22.3</td><td>26.7</td><td><strong>24.0</strong></td><td>24.4</td></tr><tr><td>EGTR [17]</td><td>24.3</td><td>19.4</td><td>13.3</td><td>18.9</td></tr><tr><td>TDE [39]</td><td>19.8</td><td>13.1</td><td>0.1</td><td>11.0</td></tr><tr><td>IETTrans [51]</td><td>23.8</td><td>18.0</td><td>13.2</td><td>18.2</td></tr><tr><td>ST-SGG [21]</td><td>21.5</td><td>15.0</td><td>0.1</td><td>14.2</td></tr><tr><td>BGNN [26]</td><td>22.3</td><td>11.9</td><td>0.1</td><td>12.5</td></tr><tr><td>TDE+ISD</td><td>21.6</td><td>14.3</td><td>0.1</td><td>12.4</td></tr><tr><td>IETTrans+ISD</td><td>26.1</td><td>19.2</td><td>11.0</td><td>18.6</td></tr><tr><td>Salience-SGG (β = 0.5)</td><td>25.9</td><td><strong>28.0</strong></td><td>22.1</td><td><strong>25.3</strong></td></tr><tr><td>Salience-SGG(β = 0.2)</td><td><strong>27.4</strong></td><td>22.8</td><td>15.0</td><td>21.6</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4. Pairwise localization avg. precision (pl-AP) versus SGG performance (F@100). Color indicates the model family. The green text marks our contributions</figcaption></figure>

and body groups with superior overall performance.

### 4.4. Salience Sensitivity Analysis

Pairwise Localization Average Precision (pl-AP) measures the model's ability to localize salient subject-object pairs in predicted triplets. For the top-100 detected triplets, we compute category-agnostic precision and recall, where precision reflects accuracy in capturing salient spatial structure and
<table><caption>Table 7. Ablation study on the individual model components.</caption><thead><tr><th>ISD</th><th>G-ESA</th><th>P-ECA</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td>×</td><td>×</td><td>×</td><td>26.4</td><td>29.7</td><td>13.5</td><td>16.4</td><td>17.8</td><td>21.1</td></tr><tr><td>✓</td><td>✓</td><td>×</td><td>27.0</td><td>31.2</td><td>17.8</td><td>20.6</td><td>21.5</td><td>24.9</td></tr><tr><td>✓</td><td>×</td><td>✓</td><td>27.8</td><td>32.3</td><td>17.1</td><td>20.5</td><td>21.2</td><td>25.1</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>28.8</td><td>33.4</td><td>18.0</td><td>21.6</td><td>22.1</td><td>26.2</td></tr></tbody></table>

recall reflects its coverage. Similar to the saliency triplet label assignment, a true positive is assigned if the IoU between subject and object pairs is at least 0.5. The pl-AP is then calculated as the area under the precision-recall curve. We report pl-AP with the SGG performance (F@100) of multiple SGG and Unbiased-SGG models, as illustrated in Figure 4. From this figure, the following observations can be made: 1) The comparisons of pl-AP and F@100 among the Unbiased-SGG models clearly demonstrate that a higher pl-AP leads to a better F@100. 2) The debiasing strategies tend to impede the capture of salient spatial structures. In particular, Motifs [50] achieves a higher pl-AP than TDE [39] and ST-SGG [21]. Of note, IETrans [51] achieves a similar pl-AP to Motifs despite IETrans introducing spatial incoherent triplets (i.e., lower precision). This is likely the result of increased diversity of captured salient structures (i.e., higher recall). 3) Comparing IETrans and IETrans+ISD highlights that ISD not only improves F@100 but also leads to an increased pl-AP.

## 4.5. Ablation Studies

Finally, in this section, we present ablation studies to provide detailed analyses of our approach. All experiments are conducted on the VG dataset. The analysis of hyperparameters (i.e. *T* and *L*) is provided in **suppl. material**.

**Impact of Individual Components.** We propose an ISD module containing G-ESA and P-ECA. Table 7 illustrates the performance when each component is removed. When G-ESA and P-ECA are partially removed, we use the regular self-attention and cross-attention layers instead to maintain the module at the same parameter level. If the ISD is completely removed, performance drops significantly. This demonstrates that our ISD can greatly improve the performance of Unbiased-SGG. Partial removal of the G-ESA and P-ECA modules causes a decline in both R@K and mR@K. These results highlight the importance of our G-ESA and P-ECA modules. The G-ESA may facilitate spatial structure learning, while the P-ECA enhances predicate understanding (i.e. the salience loss serves as a regularization for the predicate prediction through the P-ECA). A detailed analysis of predicate understanding and spatial structure learning is provided in the **suppl. material**.

**Iterative Refinement Analysis.** Table 8 demonstrates that an iterative refinement process in ISD leads to a better performance than the non-iterative process.

**Triplet Salience Label Analysis.** To investigate the effect

<table><caption>Table 8. Ablation study on the iterative refinement process in ISD.</caption><thead><tr><th>Iterative Refinement</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td>×</td><td>28.6</td><td>33.0</td><td>17.1</td><td>20.7</td><td>21.5</td><td>25.4</td></tr><tr><td>✓</td><td>28.8</td><td>33.4</td><td>18.0</td><td>21.6</td><td>22.1</td><td>26.2</td></tr></tbody></table>

<table><caption>Table 9. Ablation study on the type of salience labels.</caption><thead><tr><th>Type</th><th>R@50</th><th>R@100</th><th>mR@50</th><th>mR@100</th><th>F@50</th><th>F@100</th></tr></thead><tbody><tr><td>top_down_gt</td><td>27.5</td><td>32.0</td><td>15.1</td><td>18.4</td><td>19.5</td><td>23.3</td></tr><tr><td>top_down_entity</td><td>28.5</td><td>33.0</td><td>16.3</td><td>19.6</td><td>20.7</td><td>24.6</td></tr><tr><td>top_down_triplet</td><td>27.9</td><td>32.5</td><td>15.7</td><td>19.1</td><td>20.1</td><td>24.1</td></tr><tr><td>Ours</td><td>28.8</td><td>33.4</td><td>18.0</td><td>21.6</td><td>22.1</td><td>26.2</td></tr></tbody></table>

of our bottom-up triplet salience labels, we create three types of top-down labels which consider the semantic information concurrently, namely *top_down_gt*, *top_down_entity* and *top_down_triplet*. *top_down_gt* is created by mapping the predicate label *G* to a binary mask as in previous work [17, 18, 44]. Following [3], *top_down_entity* is created by performing an additional one-to-many matching on the object detector results. *top_down_triplet* is constructed by performing one-to-many matching at the triplet level as in [20]. For all triplet salience labels, the predicate label *G'* is consistently obtained by the one-to-one matching results from the object detector. The details of the construction process of each type of triplet salience label are specified in the **suppl. material**. Table 9 shows that Salience-SGG, which is supervised by bottom-up triplet salience labels offering maximal spatial supervision, achieves the best results.

## 5. Conclusion

The present study provides a detailed examination of the underlying causes of the substantial performance deterioration of Unbiased-SGG models on frequent predicates. Based on our observations, we propose a framework incorporating an iterative triplet salience estimation process to enhance the Unbiased-SGG process. The salience estimation process, supervised by semantic-agnostic salience labels, enables Unbiased-SGG models to globally highlight the triplets characterized by salient spatial structures. This improvement is demonstrated quantitatively by a specific evaluation metric that measures a model's ability to capture salient triplet structures. To model the process of salience estimation, two enhanced attention layers are devised to direct the salience message-passing. Extensive experimentation and ablation studies are conducted, demonstrating that the introduced framework achieves SOTA performance and exhibits robustness to diverse debiasing strategies.
## Acknowledgement

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2002/1 "Science of Intelligence" - project number 390523135. In addition, this work was partially supported by MIAI@Grenoble Alpes, (ANR-19-P3IA-0003).

## References

[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In *European conference on computer vision*, pages 213–229. Springer, 2020. 2

[2] Chao Chen, Yibing Zhan, Baosheng Yu, Liu Liu, Yong Luo, and Bo Du. Resistance training using prior bias: toward unbiased scene graph generation. In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 212–220, 2022. 1, 2

[3] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. *arXiv preprint arXiv:2409.10262*, 2024. 5, 6, 8

[4] Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(9):11169–11183, 2023. 6

[5] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In *Proceedings of the IEEE conference on computer vision and Pattern recognition*, pages 3076–3086, 2017. 2

[6] Alakh Desai, Tz-Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 15404–15413, 2021. 7

[7] Qi Dong, Zhuowen Tu, Haofu Liao, Yuting Zhang, Vijay Mahadevan, and Stefano Soatto. Visual relationship detection using part-and-sum transformers with composite queries. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 3550–3559, 2021. 2

[8] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19427–19436, 2022. 1, 5, 6, 7

[9] Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. Not all relations are equal: Mining informative labels for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 15596–15606, 2022. 1

[10] Rong Han, Xiaohong Liu, and Ting Chen. Yolo-sg: Salience-guided detection of small objects in medical images. In *2022 IEEE International conference on image processing (ICIP)*, pages 4218–4222. IEEE, 2022. 2

[11] Xixuan Hao, Danqing Huang, Jieru Lin, and Chin-Yew Lin. Relation-enhanced detr for component detection in graphic design reverse engineering. In *Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence*, pages 4785–4793, 2023. 4

[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016. 5

[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. *Neural computation*, 9(8):1735–1780, 1997. 2

[14] Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, and Badong Chen. Salience detr: Enhancing detection transformer with hierarchical salience filtering refinement. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 17574–17583, 2024. 2

[15] Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, Badong Chen, and Xuguang Lan. Relation detr: Exploring explicit position relation prior for object detection. In *European Conference on Computer Vision*, pages 89–105. Springer, 2024. 4

[16] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6700–6709, 2019. 5

[17] Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 24229–24238, 2024. 2, 3, 5, 6, 7, 8

[18] Deunsol Jung, Sanghyun Kim, Won Hwa Kim, and Minsu Cho. Devil's on the edges: Selective quad attention for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 18664–18674, 2023. 2, 8

[19] Siddhesh Khandelwal and Leonid Sigal. Iterative scene graph generation. *Advances in Neural Information Processing Systems*, 35:24295–24308, 2022. 2
[20] Jongha Kim, Jihwan Park, Jinyoung Park, Jinyoung Kim, Sehyung Kim, and Hyunwoo J Kim. Group-wise query specialization and quality-aware multi-assignment for transformer-based visual relationship detection. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 28160–28169, 2024. 5, 6, 7, 8

[21] Kibum Kim, Kanghoon Yoon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Adaptive self-training framework for fine-grained scene graph generation. In *ICLR*, 2024. 1, 6, 7, 8

[22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. *International journal of computer vision*, 123:32–73, 2017. 5

[23] Harold W Kuhn. The hungarian method for the assignment problem. *Naval research logistics quarterly*, 2(1-2):83–97, 1955. 3

[24] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. *International journal of computer vision*, 128(7):1956–1981, 2020. 5

[25] Jiankai Li, Yunhong Wang, Xiefan Guo, Ruijie Yang, and Weixin Li. Leveraging predicate and triplet learning for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 28369–28379, 2024. 5, 6, 7

[26] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 11109–11119, 2021. 1, 2, 5, 6, 7

[27] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 19486–19496, 2022. 6

[28] Wei Li, Haiwei Zhang, Qijie Bai, Guoqing Zhao, Ning Jiang, and Xiaojie Yuan. Ppdl: Predicate probability distribution based loss for unbiased scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19447–19456, 2022. 1

[29] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In *Proceedings of the IEEE international con-

ference on computer vision, pages 2980–2988, 2017. 5

[30] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 3746–3753, 2020. 6

[31] Xin Lin, Changxing Ding, Jing Zhang, Yibing Zhan, and Dacheng Tao. Ru-net: Regularized unrolling network for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19457–19466, 2022. 6

[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. *arXiv preprint arXiv:1711.05101*, 2017. 5

[33] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In *Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14*, pages 852–869. Springer, 2016. 2

[34] Thanh-Son Nguyen, Hong Yang, and Basura Fernando. Effective scene graph generation by statistical relation distillation. In *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 8420–8430. IEEE, 2025. 6

[35] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*, pages 1532–1543, 2014. 3

[36] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. *IEEE transactions on pattern analysis and machine intelligence*, 39 (6):1137–1149, 2016. 2

[37] Gopika Sudhakaran, Devendra Singh Dhami, Kristian Kersting, and Stefan Roth. Vision relation transformer for unbiased scene graph generation. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 21882–21893, 2023. 2, 6

[38] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6619–6628, 2019. 2, 6

[39] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 3716–3725, 2020. 2, 6, 7, 8

[40] Yao Teng and Limin Wang. Structured sparse r-cnn for direct scene graph generation. In *Proceedings of the*
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19437–19446, 2022. 2, 5, 6

[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *Advances in neural information processing systems*, 30, 2017. 2

[42] Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9695–9704, 2021. 5

[43] Jinghao Wang, Zhengyu Wen, Xiangtai Li, Zujin Guo, Jingkang Yang, and Ziwei Liu. Pair then relation: Pair-net for panoptic scene graph generation. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024. 6

[44] Lei Wang, Zejian Yuan, and Badong Chen. Multi-granularity sparse relationship matrix prediction network for end-to-end scene graph generation. In *European Conference on Computer Vision*, pages 105–121. Springer, 2024. 2, 3, 5, 6, 8

[45] Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Linknet: Relational embedding for scene graph. *Advances in neural information processing systems*, 31, 2018. 2

[46] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 5410–5419, 2017. 2

[47] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao, and Chen Change Loy. Zoomnet: Mining deep feature interactions for visual relationship recognition. In *Proceedings of the European conference on computer vision (ECCV)*, pages 322–338, 2018. 2

[48] Kanghoon Yoon, Kibum Kim, Jaehyeong Jeon, Yeonjun In, Donghyun Kim, and Chanyoung Park. Ra-sgg: Retrieval-augmented scene graph generation framework via multi-prototype learning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 9562–9570, 2025. 2, 6

[49] Yujie Zang, Yaochen Li, Yuan Gao, Yimou Guo, Wen-neng Tang, Yanxue Li, and Meklit Atlaw. Refine and redistribute: Multi-domain fusion and dynamic label assignment for unbiased scene graph generation. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 1318–1327, 2024. 2

[50] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with

global context. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 5831–5840, 2018. 2, 5, 6, 7, 8

[51] Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Fine-grained scene graph generation with data transfer. In *European conference on computer vision*, pages 409–424. Springer, 2022. 1, 2, 5, 6, 7, 8

[52] Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embedding network for visual relation detection. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 5532–5540, 2017. 2

[53] Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 22783–22792, 2023. 6

[54] Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less is more: Focus attention for efficient detr. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 6674–6683, 2023. 2

[55] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. *arXiv preprint arXiv:2010.04159*, 2020. 3, 5