Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization for Scene Graph Generation
Authors: Guoqing Zhang
Shichao Kan
Fanghui Zhang
Wanru Xu
Yue Zhang
Yigang Cen
Venue: ICML 2025 poster (2025)
Keywords: Scene Graph Generation; Diffusion Model; Feature Reconstruction
Primary area: deep_learning
TL;DR: For the scene graph generation task, we propose, for the first time, a feature reconstruction enhancement method based on diffusion models and discretization mapping to mitigate biased predictions caused by the long-tail distribution problem.

Abstract: Scene Graph Generation (SGG) is a fundamental task in visual understanding, aimed at providing more precise local detail comprehension for downstream applications. Existing SGG methods often overlook the diversity of predicate representations and the consistency among similar predicates when dealing with long-tail distributions. As a result, the model's decision layer fails to effectively capture details from the tail end, leading to biased predictions. To address this, we propose a Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization (NoDIS) method. On the one hand, expanding the predicate representation space enhances the model's ability to learn both common and rare predicates, thus reducing prediction bias caused by data scarcity. We propose a conditional diffusion model to reconstructs features and increase the diversity of representations for same category predicates. On the other hand, independent predicate representations in the decision phase increase the learning complexity of the decision layer, making accurate predictions more challenging. To address this issue, we introduce a discretization mapper that learns consistent representations among similar predicates, reducing the learning difficulty and decision ambiguity in the decision layer. To validate the effectiveness of our method, we integrate NoDIS with various SGG baseline models and conduct experiments on multiple datasets. The results consistently demonstrate superior performance.

Noise-Guided Predicate Representation Extraction and Diffusion-Enhancel Discretization for Scene Graph Generation
Guoqing Zhang 123 Shichao Kan
Fanghui Zhang?
Wanru Xu²
Yue Zhang
Yigang Cen 123
Abstract
X
Scene Graph Generation (SGG) is fundamen- tal task in visual understanding, aimed at provid- ing more precise local detail comprehension for downstream applications. Existing SGG meth- ods often overlook the diversity of predicate rep- resentations and the consistency among similar predicates when dealing with long-tail distribu- tions. As result, the model's decision layer fails to effectively capture details from the tail end, leading to biased predictions. To address this, we propose a Noise-Guided Predicate Rep- resentation Extraction and eiffusion-Enhancedd Discretization (NoDIS) method. On the one hand, expanding the predicate representation space en- hances the model's ability learn both common and rare predicates, thus reducing prediction bias caused by data scarcity. We propose a condi- tional diffusion model to reconstructs features and increase the diversity of representations for same category predicates. On the other hand, in- dependent predicate representations in the deci- sion phase increase the learning complexity of the decision layer, making accurate predictions more challenging. To address this issue, we in- troduce discretization mapper that learns con- sistent representations among similar predicates, reducing the learning difficulty and decision am- biguity in the decision layer. To validate the ef- fectiveness of our method, we integrate NoDIS
Meld!
Miae Oodked
(a) NoDIS for Discretized Feature Enhancement.
04
60
64
enoplig
(If) Valiatims beriven caoi lace spreccatioon and wnights same caaggry.
(b) Intra-Class Variance Changes.
Figure I. (a) Our method enhances diversity and homogeneity of representations for predicates within the same category by intro- ducing diffusion and discretization mapping, effectively mitigating the scene graph bias problem. (b) Variance change between fea- ture distributions. A larger variance indicates greater difference between the feature distributions.
with various SGG baseline models and conduct experiments on multiple datasets. The results consistently demonstrate superior performance. We have uploaded the code GitHub: ht atsps: / / /ggt thub. com/ heer//avinnnn seavin-gqzane waggggzzagg//gggg /NDDn.. NoDIS.
1. Introduction
'State Key Laboratory Advanced Rail Autonomous Opera- tion, Bejing Jiaotong University, Beijing, China 'School of Com- puter Science and Technology, Bejing Jiaotong University, Beijing, China 3 Visual Intelligence +X International Cooperation Joint Laboratory of MOE, Bejing Jiaotong University, Beijing, China "School Computer Scienee and Technology, Central South Uni- versity, Hunan, China 'School of Artificial Intelligence, Henan University, Henan, China CCollege of Computer and Information Engineering, Henan Normal University, Henan, China. Corre- spondence to: Wanru <xwanru bjtu.edu.cn... Yigang Cen
Scene graph generation constructs relational representations for object pairs in an image, which provides essential in- sights for downstream tasks such as image captioning (Chen et al., 2020; Zhong et al., 2020), visual question answering (Hudson & Manning, 2019b; Teney et al., 2017), and image retrieval (Johnson al., 2015; Schroeder & Tripathi, 2020), helping understand local details within images.
However, the long-tail distribution inherent in datasets poses significant challenges. Traditional methods for predicate prediction tend to favor head predicates with shallow seman- ties while neglecting tail predicates with deeper meanings.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Noise-Guided Predicate Representation Extraction and
Discretization for Scene Graph Generation
This leads to meaningless or erroneous triplet generation, severely impairing the understanding of fine-grained im- age details in downstream tasks like image captioning and visual question answering. To address this issue, current approaches optimize the algorithms from three perspectives: loss functions (Yan et al., 2020; Kang & Yoo, 2023), sam- pling strategies (Desai et al., 2021; Li et al., 2021b), and feature enhancement (Yu et al., 2023). However, these ap- proaches concentrate exclusively on tail class enhancement while neglecting information from head classes. Recently, some methods (Li et al., 2024a; Wang et al., 2023a) have leveraged VAE (Kingma& & Welling, 2022) or GAN (Good- fellow et al., 2020) for feature augmentation to strengthen the representation of tail predicates. However, these meth- ods heavily rely on coarse-grained predicate representations as latent features and result in highly similar features before and after enhancement, which limits the effective expansion of the predicate representation space and fails to increase the diversity among similar predicates. Moreover, directly applying these methods for feature enhancement and using them to fine-tune the final decision layer weights treats each sample as an independent entity, neglecting the homogene- ity among similar predicate representations, which increases the learning difficulty at the decision layer.
aids the model in effectively learning diverse representation knowledge. Finally, to alleviate the learning difficulty of the decision layer caused by the independence of sample rep- resentations, the Feature Discretization Mapping module is introduced. This module employs a learnable discretization encoder to learn unified representations among the same category predicates and aggregates them into an indepen- dent representation space, which is then used for learning the independent representation distributions of different cat- egories in the decision layer. This approach effectively reduces the differences between similar predicate repre- sentations, leading to a significant decrease in intra-class feature distribution variance (as shown in Figure 1b (D)) Furthermore, by using the aggregated independent predicate representations to train the decision layer, the distribution discrepancy between similar predicate representations and their corresponding class decision weights is effectively minimized (as shown in Figure 1b (I)). This enables the decision layer to more effectively learn the aggregated in- dependent predicate representations, thus minimizing the impact of heterogencous predicate representations on the decision-makir process and improving decision accuracy.
We conducted extensive experiments on datasets such as VG (Krishna et al., 2017), GQA (Hudson & Manning, 2019a) and OpenImage V6(Kuznetsova et al., 2020), achieving excellent performance, which demonstrates that our method effectively performs feature reconstruction and mitigates the biased predictions caused by long-tail distribution.
To address the above issues, we propose a solution from two perspectives: 1) Explore new feature enhancement meth- ods to expand the representation space and improve the model's ability to learn generalizable representations. By increasing the diversity of intra-class predicate representa- tions, this approach alleviates the model's 's underfitting rare information, thus enabling better learning of tail-class information. 2) In the decision phase, we alleviate the learn- ing difficulty and decision burden caused by the diverse independent representations of samples by enhancing the homogeneity among representations of predicates within the same category. By learning the unified representations of similar predicates, the decision layer is trained to learn the differences between predicate representations of different categories, thereby improving both the learning speed and decision accuracy of the decision layer.
In summary, the contributions of this paper are as follows: 1) We propose the NoDIS method, which enhances the model's s generalization ability by increasing the diversity of predi- cate representations within the same category. At the same time, it strengthens the homogeneity of these representa- tions to aid the decision layer in learning the differences between representations of different categories, effectively mitigating bias prediction issues. 2) We introduce the Fea- ture Enhancement Based on Diffusion Module, which, for the first time, applies diffusion techniques to increase the diversity of predicate representations within the same cat- egory, expand the representational space, and enhance the model's ability to generalize knowledge extraction. 3) We design a Feature Discretization Mapping module that learns consistent representations among predicates of the same category and projects them into independent representation spaces for training the decision layer. This approach effec- tively alleviates convergence issues and semantic ambiguity in the decision layer, thereby improving decision accuracy.
Based on these ideas, we propose the Noise-Guided Pred- icate Representation Extraction and Diffusion-Enhancee Diseretization (NoDIS) method. As shown in Figure la, NoDIS introduces the Noise-Guided Predicate Extraction module, which extracts independent predicate features ing one-step noise addition and denoising algorithm. To expand the predicate representation space and enhance the diversity of predicate features within the same category, a conditional feature reconstruction diffusion learning method proposed. This method significantly increases the diver- sity of predicate representations within the same category, leading to notable rise in intra-class feature distribution variance (as shown in Figure lb (D). This enhancement
2

---

Noise-Guided Predicate Representation Extraction and Diflusion-Enhancee Discretization for Scene Graph Generation
2. Related Works
resentations and predicting predicates between entity pairs using the entity information obtained in the first stage. To achieve this, we design independent Predicate Refine- ment Module. Assuming that the predicate between the i-th entity and the j-th entity is rij. the complete set of predicate categories can be represented as R = {roI Tu2: ..., Tiy where j corresponds to the (i 1)-th entity. Additionally, we use pre-trained embeddings GloVe (Pennington et al.. 2014) to encode all predicate categories, initializing the predicate prototype representations P = {pi, LiP,,P,,... .P.)... .Po). where represents the total number of predicate categories.
2.1. Scene Graph Generation
The long-tailed distribution in datasets often leads to biased predictions in models. The early methods (Zellers et al., 2018; Xu al., 2017; Chiou et al., 2021) aimed mitigate this bias by taking advantage of distribution patterns within the dataset as additional prior knowledge. In recent years, one-stage Scene Graph Generation models (Cong et al., 2023) have become the hot issue of research. These models leverage Transformer-base based architectures (Vaswani et al., 2017) to iteratively refine predicate representations while simultancously determining predicates and entities. Addi- tionally, increasing attention has been paid to the design of training strategies and loss functions(Kang & Yoo, 2023; Yan et al., 2020). Techniques such as data resampling (Li & Vasconcelos, 2019), transfer learning (Liu et al., 2019), and causal learning (Tang et al., 2019; 2020) have been explored to alleviate the impact of long-tailed distributions.
3.2. Noise-Guided Predicate Representation Extraction
To enhance the independence of predicate representations and provide effective prior information for subsequent pred- icate reconstruction, we construct predicate representations from a node-edge-node structural perspective and decouple entity- peitycpredictte representations from an entity-noise per- spective. As illustrated in Figure 2(a). the overall process divided into two parts: Neighborhood Context Extraction and boise-Gudded Predicate Refinement, as follows.
2.2. Diffusion Model
Diffusion has become one of the most widely used meth- ods in generative tasks, such as image generation, super- resolution, etc. Initially (Ho et al., 2020) introduced the Denoising Diffusion Probabilistic Model (DDPM), which simulates the diffusion process and then reverses it gener- ate data, achieving promising results. Subsequently, Score- based Generative Models (Song et al., 2020b) were pro- posed, adopting a framework similar to the diffusion pro- cess but using score models to recover the data distribution in reverse. Latent Diffusion (Rombach et al., 2022) has established Diffusion general framework for generative tasks, significantly improving its performance and quality in image generation. Recently, researchers have increasingly focused on incorporating conditions into diffusion models (Zhang et al., 2023) to enable more precise control and higher-quality outputs.
Neighborhood Context Extraction: We treat entity pairs as nodes and construct learnable Query Tokens to represent edges between entity-pair nodes. By iteratively optimiz- ing the learning capability of Query Tokens, we achieve an effective predicate representation construction between entities.
First, we use entity representations preprocessed by the base model (Zellers et al., 2018; Vaswani et al., 2017; Zheng et al., 2023b) to construct entity pair representations E, and F.g, representing subject and object entity pairs, respectively. Then, through two distinct cross-attention modules, the two entity representations alternately serve as Query, enhancing the uniform expression between entities, as shown in Eq. 2.
Alln(Q, K, V) = Softmart QKT
(1)
Vd
3. Method
E. Attn(EE, E., Eo), E. Attn(E,. E,, E.)
(2)
3.1. Problem Formulation and Symbolic Representation
Next, leveraging the cross-attention modules, we employ a learnable Query Token Qp to extract consistency representa- tions between entity pairs. This consistency representation serves as a shared edge structure for entity pairs, represent- ing the predicate, as illustrated in Eq. 3. Additionally, to prevent information loss, we introduce a joint visual rep- resentation of the entity pairs to refine the constructed predicate representation. As shown in Eq. 4, we filter the node information of entity pairs from the joint visual rep- resentation, suppressing regions with weak correlations to the nodes of entities while strengthening the intensity of the representation of the edges between nodes.
We adopt two-stage architecture for the scene graph gen- eration task. In the first stage, Faster R-CNN (Ren et al., 2015) is used for object detection, identifying all entities in the image. Suppose that there are entities, their categories are denoted as O = (Oo. ...... O,}. their bounding boxes as B = (bg. bi.... bz). ). and their visual features, extracted using RolAlign (He et al., 2017), as E- = (eo, ei,..., ei). ex). Additionally, we compute the union visual features for all entity pairs, denoted as U = {uo, ui, Uz}. where j = i x 1).
In the second stage, we focus on extracting predicate rep-
Qp = Attn((pp [F.s, Foj, [Fs,, Fea))
(3)
3

---

Noise-Guides Predicate Representation Extraction and Discretization for Scene Graph Generation
Faster RCNN
Racosstaadii prodeas tele repissentiiss
Modile
Contect
Fuuleye
V
discrdle mapper
hoad
man
aEm
C
C
C
Pair representation
leg
hand
surfhourd
Fimheeding
MLP
Suttes
Prediction
A
A
Gaaacan hear
Generat
Naighborhaod
Nakue- laried
Castool Extrctioo
Dijeer
Erabedding
v/k
MLP dosk
OAjec
Clr
LTT.
Wagli
( Noise-Gusted Prodicate Represenation Extraction
Feature Eahancement Based Diffision
Letraale Feature Discrtizatoon
Figure 2. Diagram of the overall structure for the (NoDIS) method.
Predicate Representation Extraction and Diffusion-Enhanced
Enhanced Discretization
U' Altn(U, EE,, Eo),,
T, - (nn E, E,
(8)
U - - (( Eal)) U²,
(4)
0g Attn(Qp: U', U')
In the above equation, refers the ReLU function, which filters out irrelevant features or noise.
In the above equations, [ denotes the concatenation opera- tion between features. w represents learnable parameter matrix that is used for feature mapping and dimensional alignment. oo stands for the Sigmoid function, which calcu- lates feature weights and smoothly maps them to the range of 0 to 1.
By enhancing the entity representation within the noisy triplet, we subtract the antity-strengthened triplet representa- tion T, from the noisy triplet representation T,, as described in Eq. 9. Since the noise in the noisy triplet distributed across the entire representation space, this subtraction re- duces the overall noise intensity. By iterating this process multiple times, both entity-related and noise components are progressively removed, ultimately yielding independent and clean predicate representations.
Noise-Guided Predicate Refinement: Based on coarse- grained predicate representations extracted from the Neigh- borhood Context Extraction Module, we construct triplet representations T by incorporating entity pair information and enhancing them using joint visual features, as shown in Eq. 5. For the initialized triplet representations, ran- dom Gaussian noise G is added, with the noise intensity X randomly initialized and injected only once during training.
T, = Th (p(w(E)) T
Finally, predicate representations are extracted from the de- noised triplet representations and filtered, resulting in the preliminary refinement of the predicate context representa- tion Cp.
T E,, Op Eol, T = T' w((((((T)) UU
(5)
((=
(10)
G 1,. =
(6)
3.3. Feature Enhancement Based on Diffusion
In the denoising process, the goal is to use the entity pair representations within the triplet and the added noise as denoising targets obtain clean predicate representations. To achieve this, enhanced entity representations are con- catenated to construct clean entity pair representations E, and feature filtering is applied to remove noise effects, as deseribed in Eq. 7. Subsequently, a cross-attention module employed, using the noisy triplet T, as the Query and the clean entity pair E as the Key and Value strengthen the representation of entity-related regions within the noisy triplet, as shown in Eq. 8.
To expand the visual space of predicate representations and enhance feature diversity, we propose Diffusion- based Feature Reconstruction Enhancement Module. Similar to existing diffusion models, it adheres to the same training and testing rules. However, during testing, we construct conditional Gaussian distribution for feature reconstruc- tion instead of using random Gaussian distribution. As illustrated in Fig.2(b), the preprocessed predicate represen- tations are first aligned with the prototype representations. Using a shared weight matrix, the predicate representations and prototype representations are mapped into a unified visual space, followed by refinement through joint visual
F [Es FoJ, F. F. w(((w(E))) E

---

Noise-Guided Predicate Representation Extraction and Diflusion-Enhannee Discretization for Scene Graph Generation
representations, as expressed in Eq.
yielding the reconstructed predicate representations Gp. as shown in Algorithm 1. During testing, all time steps used instead of selecting
{ P MLP(P),C, Cp C,PP),,C=MMM. MML((C)) U = Attn(U, P', PP',,, = AttmlCp, U')
(11)
Algorithm 1 Feature Reconstruction Training Process Based on Diffusion
During the diffusion process, we use the predicate proto- types corresponding to the ground-truth categories of all samples as the target feature distribution T-- Noise of vary- ing intensities is then added based on the current timestep, where the noise schedule follows the same strategy as (Luo & Hu, 2021). Assuming the timestep is t, the noise intensity is denoted as "/e.
1: T, n {T: total number of iteration steps, n: randomly selected step sizes]
m w(Cp), v = (Cp)) {Init feature distribution}
G' + + Vo N0 1) (Init noise input)
Gp G
5: for in random (T, n) do
6: E, t- Embedding(t) {Initialize time embedding)
T PlGt), Gt = {Git Gtz, .., G T - Vn Tp + v(1 -)). G
7: N <- Attn(Gp: E,, F2) (Conditional diffusion]
(12)
8:
N & Alln(N Cps Cp) (Conditional diffusion)
9:
N + Attn(N .Tp, Tp) {Conditional diffusion]
In the above equations, Gl represents the ground truth pred- icate category for each sample, while Gt and Gt, denote the predicate categories for the 1-th and i-th samples, re- spectively. G refers to a Gaussian distribution initialized randomly, which is added as noise to the predicate proto- types corresponding to the ground truth categories of each
10:
'' + w(N) (((W(Cp)) + wCp)) (Noise predic-
tion}
11: Gp + n- (C--V~,' {Single-step denois-
ing)
12: end for
sample.
13: is training then
14: loss +- MSELoss (Gp: Tp)
In the noise prediction process, we introduce a conditional noise prediction module, as shown in Equation 13. Based on a Cross- Attention mechanism, the noise target dis- tribution T is used as the Query, while the current timestep embedding, prototype representation, and predicate repre- sentation serve as the Key and Value, respectively, to pro- gressively enhance effective feature representations. Finally, simple linear mapping and bias learning are employed to refine the enhanced representations and predict irrelevant regions as noise, as described in Equation 14.
15: return Gp, loss
16: end if
17: return Gp
3.4. Learnable Feature Discretization Mapping Module
We observe that using Diffusion for predicate representa- tion reconstruction can lead to a discrete distribution where each sample corresponds a unique feature, which hin- ders subsequent predicate classification. To address this, we propose the introduction of discrete encoder similar that in (Van Den Oord et al., 2017). First, we construct a learnable parameter matrix Lp and initialize it using prepro- cessed encoded predicate prototypes P Next, calculate the distance between the reconstructed predicate representa- tions and the features of each category in Lp- The closest parameter feature to the reconstructed feature is selected as the affine- transformed feature G1, as shown in Eq. 15.
E. = Embedding (t), T = Attn(T pr Ex),
T Alln(T, py Cp, Cp), T2 Alln(T pt Tp: Tp)
(13)
N' =(TH) p(w(C,)) + ((C))
(14)
Based on the above method, noise prediction for the tar- get feature distribution with added noise can be effectively achieved. However, during the reconstruction phase, Gaus- sian noise with random initialization, which contains no prior information, is used as input and gradually denoised. This process may result in reconstructed features deviating from the target feature distribution. To address this issue, we introduce a parallel branch during training for feature reconstruction, further constraining the feature generation direction. Specifically, Gaussian noise distributions are re- constructed using the mean and variance of the preprocessed predicate representations Cp. Reverse diffusion is then ap- plied to predict the noise and iteratively denoise the features. By randomly selecting n time steps, noise is predicted it- eratively and denoised according to the noise intensity 7/t
{ Lp ~ P', = VEGG, 1p) G = Lpppaargnnnn
(15)
In the above equation, G, represents the predicate repre- sentations reconstructed based on Algorithm I, and Dis represents the distance between the reconstructed features and the discretized prototype features. Assuming the num- ber of samples is m and the number of predicate categories is c, the scale of Dis is (m, c). argmin (Dis) denotes the index of the shortest distance between each reconstructed sample and the discretized representation.
5

---

Noise-Guided Predicate Representation Extraction and Diffusion-Enhancee Discretization for Scene Graph Generation
Finally, we perform feature mapping on the affine- transformed feature Gj and aligned predicate prototypes P' from Eq. 11, and compute the similarity between the discretized dffine-transformee features and prototype repre- sentations. This similarity serves as the final prediction of the predicate category, as shown in Eq. 16. During training, the pre-processed aligned predicate prototypes P are not affected by the gradients from this part, and the computa- tion graph is disconnected, allowing independent updates for optimization.
Finally, we incorporate a KL divergence-based feature con- straint. By calculating the feature distribution between the extracted predicate representations and the predicate proto- types, this constraint optimizes the distribution of the ex- tracted representations to align with the feature distribution of the prototypes.
me - (CCp), = =PP),
logue- = w/ (Cp),,ogu, (P').
LRI = E
gogue (m--mp)²
+ e logue - Logu,)
glogup elogup
(18)
G MLP(GI), P MLP(
(16)
Ra = G
In the above equations, me and my share the same weight matrix, logr, and logup share the same weight matrix.
where D represents the stopping of gradient propagation.
Feature Enhancement Based on Diffusion: First, to en- hance the noise prediction capability of the model, we use MSE loss to calculate the loss between the predicted noise N' (from Eq. 14) and the initial added noise G (from Eq, 12), thereby optimizing the noise extraction process. Second, to effectively control the denoising direction, we follow the computation steps in Algorithm 1, randomly selecting time steps for noise prediction and iterative denoising. Finally, the denoised predicate representations Gp is compared with the predicate prototype Tp corresponding to the sample's category (from Eq. 12) by the MSE loss to ensure feature consistency between the reconstructed representation and the prototype.
3.5. Loss Function
Noise-Guided Predicate Representation Extraction: In this module, our loss function consists of three main com- ponents. we adopt the predicate representation and proto- type distance metric loss proposed in PENet (Zheng et al., 2023b) to jointly constrain the predicate representations 2, and Cp extracted from two subcomponents. The Euclidean distance is used to compute the distances between Qp. Cp- and the predicate prototypes, resulting in the losses Lou and Lod- These losses enforce minimal intra-class predicate representation distances and maximal inter-class predicate representation distances.
Additionally, we introduce a re-weighted cross-entropy loss function, where the weights are dynamically adjusted based on the prediction status. Specifically, we construct positive and negative prediction score tables, Sp and Sy. respec- tively. The prediction score for the true category of each sample is used as the positive sample weight, while the prediction scores for other categories serve as negative sam- ples. We also track the occurrence frequency of each sample across all categories, denoted as Sr. The dimensions of Sp. SN and S, are all (c,) ), where e is the number of categories. Then, using a momentum update method, the loss weights for each category are updated iteratively. The initial weight for each category S; is set to I.
Lree = I| - G112 + I|G -Tpll
(19)
Learnable Feature Discretization Mapping Module: For discretizing the reconstructed predicate representations, we adopt the loss function design of (Van Den Oord et al., 2017) to calculate the quantization loss between features before and after mapping. Additionally, we supervise the distance between features to prevent the reconstructed rep- resentations from being associated with incorrect discrete encoded features. As shown in Eq. 20, the distance Dis (defined in Eq. 15) between the reconstructed representa- tion and the discrete encoded representation is normalized and constrained by the ground truth labels of the samples. This ensures that the reconstructed representations remain homogeneous with the discrete encoded representations of the same category.
R. = Cp x P',S, = S.+ Pgt:
Sp Sp + R. Pot. SN Sy + Re Pott
(17)
S S. 8 + (1 8) - 1g (1+
8,7e +
bp
Lais ||G, - [G) -G,1 |Dis P l
Innee = CE(Re, GI, weight = S)
(20)
4. Experiments
In the above equation, Pst represents the one-hot encoded distribution of the true predicate categories, where the index of the true category for each sample is set to 1. Pax denotes the inverse of the not encoded distribution of the true predicate categories.
4.1. Experiment Setting
Dataset: We use the Visual Genome (VG) (Krishna et al., 2017) and GQA (Hudson & Manning, 2019a) datasets for
6

---

Noise-Guided Predicate Representation Extraction and Diffusion- Ennaanced Discretization for Scene Graph Generation
PredCCs
SGCIs
SGDet
Method
mi 56/100
R 50/100
F 50/100
mR 50/1000
F@500000
mR 50/100
Fe
RelTR (Cong et 2023) mwMrrs
21,2/-
64.2//
31.9/-
11.4/-
36/6/-
17.4/
27.5/-
15.5/-
HedSGG (Yoon et al., 2023) AAAF2
32,3/34,5
57.1/ 59.4
41.3/43,6
15,8/17.7
37.6/ 38.5
22.3/24.3
11.5/13.5
30.2/ 34,5
16.7/ 19.4
DCNer (Han ol., 20222a) mrserrs
33.4/35.6
57.3/9.1
42.2/44.4
21.2/22.22
360/6688
26.7/27.7
14.3/7.3
28.6/32.9
19.1/22.7
ST-SGG (Kim et ol., 2024) X2L
32,7/35.6
52.5/54.3
40.3 /33.0
21.0/22.4
36,3/37.3
26.6/27.9
12.6/15.1
20.7/24.9
18.7/188
PCPL et al., 2020) 3o
35.2/37.8
50.8/52.6
41,6/44,0
18.6/19.6
27.6/884
22.2/23.2
9,5/11.7
14.6/18.6
11.5/144
HiKER (Zhung al., 2024) everse
39,3/41.2
20.3/1.4
+/-
+
of
EGTR (Im et al., (0224) EX9R3
---
-
14.0/8.3
28.2/31.7
187/222
VTrunSE (Zhang et 2017) cynee
17.4/18.7
66.0/67.8
27.5/29.3
11.2/12.3
40.3/414
17.5/19.0
7.3/8.5
314/35.6
1LS/13.77
GCL (Dong et al., 2022) coree
34,2/36.3
+-
20.5/21.2
J.
-/-
13.6/15.5
---
J
+NoDIS(ours)
38.68/8.04
57.83 59.74
44.13/46.48
20.83/2.14
3884/39.96
27.12/28.49
13.77/15.88
27.96/31.99 1.7
18,45/21188
Motifs (Zellers 2011)) cowrn
2018)
17,4/19.3
65.7/67.9
27, 07.500
10.9/12.0
41.44255
17.3/18.8
7.3/8.6
32.0/33.2
119/14.
DBiised (Han et al., 20226) new
34.7/36.6
5R 8/007
43.6/45.7
20.3/21.2
36.5/374
26/1/27.1
14.9/ 17.5
29 4/33.9
19.8/ 23.1
+PCL (Tao sl. 2022) Tir
33.6/35.8
55.0557.
41.7/44.1
18.2/19.1
34.2/35.2
23.8/24.8
14,2/16.6
2/30/3344
19.1/22.2
+NICE (Li et al., 2022a) o21222
29.9/32.3
$5.1/57.2
38133
16.6/17.9
33.1/34.0
22.1/23.5
12.2/14.4
27.8/31.8
17.0/19.8
+HML (Deng al., 2022) socvzz
36.3/38.7
47.1/49.1/
41.0/43.3
20.8/22.1
26.1/27
23.2/24.5
17.3
176/21.0
16.0/19.0
+Inf (Biswas & Ji, 2023) cvee
24.7/30.7
$1.5/55.1
33.4/39.4
14.5/17.4
32.2/33.8
20.0/23.0
9,4/11.7
23.9/27.1
13.5/16.3
+LS--
al.,, 2023b) resvrrr
34.2/37.9
47.6/50.9
8/43,
18.7/20.9
31.2/32.7
23.4/25.5
13.7/16.6
23/5/27.2
17.3/20.6
+QuaRE (Wang et al, 2023c) ToMess
15.8/17.1
66.2/ 68.0
25.5/27.3
9.2/9.8
40,3/41.2
15.0015.8
6,2/7.6
32.5/ 37.4
10.4/12.6
+CFA (l. et al. 2023g) 300V2
35.71 38.2
1/56.6
43.01/45.61
17,0/184
34.9/36.1
22.86/24.38 38
13,2/15.5
21.4 31.8
17,82/20884
+MEET (Sudhakaran et al., 2023) 1ocv
25,3/33.5
67.4/
36.79/ 45.87
23.7
40.2/43.2
25.87/ 30.61
8,5/118
27,9/33.3
13.03/17.43
+NoDDS(ours)
33,83/36.48
62.72/64.98
43.95/46.73
22.68/24.35
36.46/37,48
27.96/29.52
13.58/16.35
30,91/35.42
18.87/22.37
Transformer (Varwani al., 2017) 17
21.4/23.7
66.71688.8
32.4/35.2
12.2/13.3
41.7/ 42.7
18,9/20.3
7.699.
12.3/14,5
+CogTree (Yu et al, 2020) 0CAr2:
28.4/31.0
38.4/39.7
32
15.7/16.7
22.9/23.4
18.6/19,5
1111/1277
19.5/21.7
14.1/16.0
+HML. al., 2022) 20v
33.3//3..9
45.6/47.8
22.5/23.88
20.71 22.0
15.7
4/18/
+.L.F (Chen al., 2024) MwM
35.9/38.4
83.3/55.2
42.9/45.3
21.3/22.6
29.0/30.0
24.6/25.8
14.8/7.55
22.8/26.2
17:9/210
+BIC (Yang et al., 2024) T
34.6/ 37.2
53.4/56.0
42,0/44.7
21,7
33.0/34.0
24.7/26.0
19.7
23.3/27.3
19 5/22.5
+NoDIS(ours)
37,25/39.97
49. 99.96/53.21
42,68/45.65
21,53/23.35
34,42/35.63
26.49/28.21
16.91/19.25
25,26/28.77
20,26/23.07
PetNet iZhong et al., 20R3 oVe
31,3153.8
64.9/ 67.2
42.4/45.0
17,8/1899
39.4/ 40.7
24.3/25.8
12,4/14.5
30.7/35.2
17.7/20.5
+NoDISiours)
38.72/41.93
50.13/53.87
43.69/47.16
22.31/23.69
30.65/32.19
25.82/27.29
14.78/17.16
23.16/26.95
18.05/20.97
Table the VG dataset, we compare the performance of our method with existing methods in the PredCIs, SGCIs, and SGDet tasks based on R R@K, mR @ K and F@ K (%) metrics.
model training and evaluation. The VG dataset consists of 108,077 images, and following the decomposition method in (Xu et al., 2017), we created a dataset containing 150 object categories and 50 relationship categories. Compared to VG, the GQA dataset GQA dataset includes 200 object categories and 100 relationship categories. Both VG and GQA datasets are split using the same method: 70% of the samples are used for training, 30% for testing, with 5,000 samples selected from the training set for validation.
4.2. Comparison with state-of-the-art Methods
We trained and evaluated three tasks the VG dataset (Krishna et al., 2017). Considering that additional prior information or augmentation methods are not available in real-world scenarios, we did not use any data augmenta- tion or prior knowledge, relying solely on the existing data and model structure. As shown in Table 1, our method sig- nificantly improves the mR @K K metric, outperforming the current state-of of-the-art methods. However, our method may not be optimal in terms of R K, as mR @ K evaluates over- all accuracy across the head, body, and tail categories, while R @ K focuses more on the performance of local categories. Additionally, since the Motifs model uses prior knowledge during training, removing this prior information affected its representation ability, leading to suboptimal performance. As illustrated in Figure 3, our method effectively enhances the basic model's prediction performance for body and tail categories while maintaining its accuracy for the head cate- gory, ultimately improving overall prediction accuracy.
Additionally, we employed the Open Images (Kuznetsova et al., 2020) dataset to further evaluate the generalization capability of our method. Open Images contains approxi- mately 9 million images, 600 object categories, and 375k annotated relationships. Open Image V6 extends this to 391k visual relationships and introduces action annotations and localized narratives, enhancing the diversity of local- ized understanding. Following the data processing approach proposed in (Li et al., 2021a; Zheng et al., 2023b; Lin et al., 2020; Zhang et al., 2019), we used 126,368 images for training, 1,813 for validation, and 5,322 for testing.
Meanwhile, based on the SGDet experimental setting, we also conducted experiments on the Open Images V4 and V6 (Kuznetsova al., 2020) datasets, as shown in Table 2. We adopted a simple Transformer (Vaswani et al., 2017) as our baseline model. Compared to VCTree (Tang et al., 2019) and Motifs (Zellers et al., 2018), the Transformer has relatively weaker feature extraction capabilities and fewer parameters, making it less complex overall. However, when combined with our proposed method, its discriminative
We follow traditional task evaluation methods (Tang et al., 2019), training and evaluating on three tasks: PredCIs (Predicate Classificationn, SGCIs (Scene Graph Classification), and SGDet (Scene Graph Detection). For performance evaluation, we use R@K, mR@K, and F@K K (Zhang et al., 2022) metrics, with K set to 20, 50, and 100.
Evaluation Tasks and Metries:
7

---

Noise-Guided Predicate Representation Extraction and hiffusion-Enhaneee Discretization for Scene Graph Generation
ability on these datasets is significantly improved, achiev- ing superior performance across multiple metrics. Specif- ically, the wm.APret and wmn.Ppphr metrics intuitively re- flect the model's ability to detect predicate categories and complete triplet phrases, respectively. As shown in Table 2, our method consistently achieves the best performance in Aret. w.A.Pprr. phr. and score on both the Open Images V4 and V6 datasets. This demonstrates that our approach not only offers strong generalization capability but also accurately identifies triplet information.
4.3. Ablation Studies
We conducted ablation studies from four perspectives: predicate representation extraction, loss function design, diffusion- based feature enhancement, and discretized fea- ture aggregation. As shown in Table 3, using only noise- guided decoupled predicate representation or predicate pro- totype alignment refinement results in negligible perfor- mance improvement. However, when both modules are introduced simultaneously. the performance on mR @ 100 improves by at least 2.25%. For the loss function, as illus- trated in Table 4, incorporating only adaptive re-weighted cross-entropy loss or KL loss individually yields limited performance gains, When both adaptive re-weighted cross- entropy loss and KI. loss are jointly applied to constrain predicate representations, the performance significantly im- proves, with an increase of least 2.43% on mR @ 100.
Feed
Finally, regarding diffusion-based feature enhancement and discretized feature aggregation, as shown in Table 5, in- troducing either module alone does not lead to noticeable performance improvements and may even result in perfor- mance degradation. However, when both modules are ap- plied together for feature enhancement and aggregation, the performance improves significantly. achieving a 1.34% increase compared to the baseline without these modules. Detailed ablation analysis can be found in Appendix C.
Figure 3. Recall values for each category with VTransE (Zhang et al., 2017) and after incorporating NoDIS, with predicate fre- quencies decreasing from left to right in the dataset.
We further analyze and compare the model parameters, as shown in Fig. 4. Specifically, we integrate NoDIS with three representative baseline models: Motifs (Zellers et al., 2018), VCTree (Tang et al., 2019), and Transformer (Vaswani et al., 2017), and measure the change in parameter counts. Al- though incorporating our method leads to an increase of approximately 40-60M parameters, this growth is relatively modest compared to other widely adopted methods or struc- tures (Tang et al., 2020; Suhail et al., 2021; Li et al., a0) Moreover, despite the smaller increase in parameters, our method yields more significant improvements in model per- formance. Detailed evaluation and performance comparison analysis can be found in Appendix B.
4.4. Quantitative and Qualitative Analysis
Quantitative Analysis Based the distribution variance of predicate representations within the same category, as shown in the Figure 1b (I), after applying the Diffusion method for feature enhancement, the variance of features within the same category increases. This indicates that the introduction of the Diffusion method effectively expands the feature space, enhancing the diversity of predicate rep- resentations within the same category. After applying the discretization mapping module, the variance between predi- cate representations sharply decreases. This suggests that discretization mapping effectively learns consistent repre- sentations from the expanded feature space, strengthening the homogeneity among predicate representations within the same category.
Analyzing the distribution between predicate representations and decision head weights. shown in the Figure lb (II), after enhancing the homogeneity of similar predicate repre- sentations, the distributions of predicate representations and decision weights within the same category become more similar. Without NoDIS method, the feature distribution ex- hibits greater variance. This indicates that the introduction of the NoDIS module effectively enhances the homogeneity of similar predicate representations, which aids the decision layer in making accurate and efficient category predictions. For further quantitative analysis, please refer to Appendix
Vidti
VCTTmo
TEME
Tranfeeer
</5005
STredt
DONS
Figure 4. Comparison of Parameter Counts After Integrating NoDIS with Base Models (Zellers et al., 2018; Tang et al., 2019; Vaswani et al., 2017).
8

---

Noise-Guided Predicate Representation Extraction and Diflusion-Enhancee Discretization for Scene Graph Generation
Dataset
Method
mR @ 50
R@50
F@50
wmAA-rr
wmAP -phr
score_wid
RelDN (Zhang al., 2019)
70.4
75.7
73.0
36.1
39.9
45.2
GPS-Net (Lin et al., 2020)
69.5
74.7
72.0
35.0
39.4
44.7
OI V4
BGNN (Li et al., 202
72.1
75.5
73.8
37.8
41.7
46.9
Transformer-NoDISSCC
70.34
74.84
72.52
38.21
42.35
47.19
DBiased (Han et al., 2022b)
42.1
74.6
53.8
34.3
34.4
42.3
PGSG (Li al., 2024b)
40.7
62.0
49.1
19.7
27.8
28.7
HOTR (Kim et al., 2021)
40.1
52.7
45.5
19.4
21.5
26.9
OI V6
PE-Net (Zheng et al., 2023b)
39.3
76.5
51.9
36.6
37.4
44.9
SGTR (Li et al., 2022b)
42.6
59.9
49.8
38.7
37.0
42.3
CSL (Liu et al., 2023b)
41.7
75.4
53.7
34.3
35.4
42.9
BCTR (Hao et al., 2025)
48.8
68.6
57.0
36.0
39.0
43.7
SQUAT (Jung et al., 2023)
75.8
34.9
35.9
43.5
Transformer-NoDIiSOON
48.93
74.11
58.94
38.87
38.95
45.95
Table We compared with the state-of-te oate-of-thee-ae methods Open Image(Kuznetsonn al., 2020) dataset using the evaluation metries proposed in (Li et al., 2021a).
NPR
PPA
R@50/mR@5 550
R@ 1000m00 100
FED
Random
Condition
CGNG
FDM
R T0URiR 100
12.08/34.75
45.31/37.99
X
59.89/31.76
62.32/34.06
X
X
54.22/3.775
56.60/3.422
X
/
47.42/33.19
49.61/36.16
17.95/38.95
x
A
45.34/36.03
48.13/39.18
/
54.26/35.76
56.63/38.41
49 49.98/36.90
$2,44/39.66
Table 3. Ablation Study Submodules of the Noise-Guided Pred- ieate Representation Extraction Module. refers Noise- Guided Predicate Refinement, and PPA denotes the Predicate- Prototype Align module.
Table S. Ablation Study of Feature Enhancement Based on Dif- fusion (FED) Module and Feature Discretization Mapping (FDM) Module. "Random" indicates feature reconstruction with ran- dom time steps. "Condition" introduces the conditional diffusion method, denotes providing prior conditions for Diffusion, and indicates discretizing the features after Diffusion-based recon- struction.
Lace
Lkt
R @50
R@ 100/mR @ 100
X
59.46/31.88
61.89/34.22
59.18/33.35
61.60/35.98
X
/
59.84/32.15
62.24/34.40
PENet
PENer NoDIS
54.26/35.76
56.63/38.41
Table 4. Ablation Study the Loss Functions in the Noise-Guided Predicate Representation Extraction Module.
Inter
D.
Figure 5. Qualitative comparisons conducted on the test dataset based on the PredCIs task. Black arrows and descriptions indi- cate correct predicate relationships, red ones represent incorrect predicate relationships, and green ones signify more accurate and superior predicate relationships.
Qualitative Analysis We conducted qualitative evalua- tion of model outputs the test dataset based the Pred- Cls task. As shown in Figure 5, compared to the baseline model, our method generates more accurate relationships with richer semantic information, such as "letter painted on train" and "person looking at train." Additional qualitative comparisons and analyses are provided in Appendix D.
introducing a conditional diffusion model, NoDIS expands the predicate representation space, enabling the model to better learn underrepresented knowledge from tail classes. Second, discretization encoder is designed to seek consis- tent representations among predicates of the same class and aggregate them into an independent representation space for training at the decision layer, thereby alleviating issues of insufficient learning and decision confusion caused by the discrete nature of independent samples.
5. Conclusion
This paper proposes the NoDIS method to address the bias in prediction from two perspectives: enhancing the diversity of predicates within the same class and ensuring homo- geneity among similar predicate representations. First, by

---

Noise-Guided Predicate Representation Extraction and
Discretization for Scene Graph Generation
Acknowledgements
Desai, A., Wu, T.-Y., Tripathi, S., and Vasconcelos, N. Learning of visual relations: The devil is in the tails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15404-15413 2021.
This work was supported in part by the National Natural Sci- ence Foundation of China under Grant 62473033, 62463002, 62406105 and 62202499, in part by the Beijing Municipal Natural Science Foundation, China under Grant L231012, in part by the Science and Technology Research Project of Henan Province 242 102210055 and 252102211026.
Dong, X., Gan, T., Song, X., Wu, J., Cheng, Y., and Nie, I., Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19427-19436, 2022.
Impact Statement
The proposed work addresses the issue of biased predic- tions in scene graph generation tasks. Errors in determining relationships between objects may lead to misjudgments, potentially resulting in incorrect understanding of local im- age details. However, since the relationship categories are constrained during the determination process, the risk of potential ethical concerns is effectively avoided.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Farle- D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(1 (3(11):139-144 2020.
Han, X., Dong, X., Song, X., Gan, T., Zhan, Y., Yan, Y., and Nie, L. Dividc-and-conquet predictor for unbiased scene graph generation. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8611-822 2022a.
References
Biswas, B. A. and Ji, Q. Probabilistic debiasing scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10429- 10438, 2023.
Han, X., Song, X., Dong, X., Wei, Y., Liu, M., and Nie, L. Dbiased-p: Dual-biased predicate predictor for un- biased scene graph generation. IEEE Trunsactions on Multimedia, 2022b.
Chen, Q., Liu, J., Lyu, X., Gao, I.,, Shen, H. T., and Song, Alf: Adaptive label finetuning for scene graph genera- tion, 2024. URLhttps //tp:///aaivvv//eeeeaeeeeeeaiiva org org hbtp//bs///3/ 2312. 17425.
Hao, P., Wang, W., Wang, X., Jiang, Y., Jia, H., Cui, S., Wei, J., and Hao, X. Betr: bidirectional conditioning trans- former for scene graph generation. Information Fusion, pp. 103260, 2025.
Chen, S., Jin, Q., Wang, P., and Wu, Q. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pp. 1962299771 2020.
He, K., Gkioxari, cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
G., Dollár, P., and Girshick, R. Mask r-
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6857 6851, 2020.
Chen, Z., Qing. J., Xiang, T., Yue, W. L., and Zhou, J. H. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 227 17700227 2023.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019a.
Chiou, M.-.., Ding, H., Yan, H., Wang. C., Zimmermann, R., and Feng, J. Recovering the unbiased scene graphs from the biased ones. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1581-1590, 2021.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019b
Cong, Y., Yang, M. Y., and Rosenhahn, B. Reltr: Relation transformer for scene graph generation. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2023.
Deng, Y., Li, Y., Zhang, Y., Xiang, X., Wang. J., Chen, J., and Ma, J. Hierarchical memory learning for fine- grained scene graph generation. In European Conference on Computer Vision, pp. 266-283. Springer, 2022.
Im, J., Nam, J., Park, N., Lee, H., and Park, S. Egtr: Extract- ing graph from transformer for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
10

---

Noise-Guided Predicate Representation Extraction and Anflusion-Enhaneee Discretization for Scene Graph Generation
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D., Bernstein, M., and Fei-Fei, L. Image retrieval using scene graphs. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3668-36777 2015.
Li, L.., Chen, G., Xiao, J., Yang. Y., Wang. C., and Chen, L. Compositional feature augmentation for unbiased scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21685- 21695, 2023a.
Jung, D., Kim, S., Kim, W. H., and Cho, M. Devil's the edges: Selective quad attention for scene graph gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18664- 18674, 2023.
Li, L., Xiao, J., Shi, H.,, Wang, W., Shao, J., Liu, A.-A., Yang, Y., and Chen, L.. Label semantic knowledge distil- lation for unbiased scene graph generation. IEEE Trans- actions on Circuits and Systems for Video Technology, 2023b.
Li, R., Zhang, S., Wan, B., and He, X. Bipartite graph net- work with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 11109-11119, 2021a.
Kang, H. and Yoo, C. D. Skew for unbiased scene graph generation. Machine Learning and Knowledge Extraction, 5(1):287-303, 2023.
re-weighting
Kim, B., Lee, J., Kang, J., Kim, E.-S., and Kim, H. Hotr: Bnddto-end human-objec interaction detection with trans- formers. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 74-83, 2021.
Li, R., Zhang, S., Wan, B., and He, X. Bipartite graph net- work with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 11109-11119, 2021b.
Kim, K., Yoon, K., In, Y., Moon, J., Kim, D., and Park, C. Adaptive self-training framework for fine-graine scene graph generation. arXiv preprint arXiv:2401.09786 2024.
Li, R., Zhang, S., and He, X. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pat- lern recognition, pp. 19486-19496, 2022b.
Kingma, D. P. and Welling, M. Auto-encoding varia- tional bayes, 2022. URL https: abs/1312.611
Li, R., Zhang, S., Lin, D., Chen, K., and He, X. From pixels to graphs: Open- vocabulary scene graph generation with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28076-28086, 2024b.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.
Li, Y. and Vasconcelos, N. Repair: Removing representa- tion bias by dataset resampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9572 357219581, 2019.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I,, Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection at scale. International journal of computer vision, 128(7):1956 19981199999911111---11 2020.
Li, Y., Ouyang, W., Zhou, B., Wang, K., and Wang, X. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEFE international con- ference on computer vision, pp. 1261-1270, 2017.
Lin, X., Ding, C., Zeng, J., and Tao, D. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3746-3753, 2020.
Li, J., Wang, Y., Guo, X., Yang, R., and Li, W. Leveraging predicate and triplet learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 28369-28379, 2024a.
Liu, D., Bober, M., and Kittler, J. Neural belief propagation for scene graph generation. IEEE transactions on pattern analysis and machine intelligence, 2023a.
Li, L.,, Chen, L., Huang, Y., Zhang, Z.,, Zhang. S., and Xiao, J. The devil is in the labels: Noisy label correction for robust scene graph generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19869-18877 2022a.
Liu, D., Bober, M., and Kittler, J. Constrained structure learning for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 2023b.

---

Noise-Guided Predicate Representation Extraction and Ennhanion-Enhaneed Discretization for Scene Graph Generation
Liu, D., Bober, M., and Kittler, J. Constrained structure learning for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023c.
Tang, K., Niu, Y., Huang, J., Shi, J., and Zhang, H. Un- biased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3716-3716321 3722,, 2020.
Liu, Z., Miao, Z., Zhan, X., Wang. J., Gong, B., and Yu, X. Large-scale long-tain tailed recognition in an open world. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2537-25466 2019.
Tao, L., Mi, L., Li, N., Cheng. X., Hu, Y., and Chen, Predicate correlation learning for scene graph generation. IEEE Transactions Image Processing, 31:4173 014477--4185 2022.
Luo, S. and Hu, W. Diffusion probabilistic models for point cloud generation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2837-2845,,2 2021.
Teney, D., Liu, L., and van Den Hengel, Graph Graph-structured representations for visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2017.
Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural 2014.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
language processing (EMNLP), pp.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, t., and Polosukhin, At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang. L., Yuan, Z., and Chen, B. Learning to generate an unbiased scene graph by using attribute-guided predicate features. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2581 12589, 2023a.
Wang, L.., Yuan, Z., Lu, Y., and Chen, B. Head-tail coopera- tive learning network for unbiased scene graph generation. arXiv preprint arXiv:2308. 12048, 2023b.
Schroeder, B. and Tripathi, S. Structured query-based im- age retrieval using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 178 188,19,,200 2020.
Wang. Z., Xu, X., Wang, G., Yang, Y.. and Shen, H. T. Quaternion relation embedding for scene graph genera- tion. IEEE Transactions on Multimedia, 2023c.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02500 2020a.
Xu, D., Zhu, Y., Choy, C. B., and Fei-Fei, L.. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-5419, 2017.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic diffcrential equations. arXiv preprint arXiv:2011.134566, 2020b.
Yan, S., Shen C., Jin, Z., Huang, J., Jiang, R., Chen, Y., and Hua, X.-S. Pepl: learning for unbiased scene graph generation. In Pro- ceedings of the 28th ACM International Conference on Multimedia, pp. 265 273, 2020.
Sudhakaran, G., Dhami, D. S., Kersting, K., and Roth, S. Vision relation transformer for unbiased scene graph gen- cration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21882 218933 2023.
perception
Suhail, M., Mittal, A., Siddiquie, B., Broaddus, C.,, Eledath, J., Medioni, G., and Sigal, L. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13936-13999 13945, 2021.
Yang, J., Wang, C., Yang. L.,, Jiang, Y., and Cao, A. Adap- tive feature learning for unbiased scene graph generation. IEEE Transactions on Image Processing, 33:2252-220 2265, 2024.
Yoon, K., Kim, K., Moon, J., and Park, C. Unbiased het- erogeneous scene graph generation with relation- awaree message passing neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3285-3294, 2023.
Tang, K., Zhang, H., Wu, B., Luo, W., and Liu, W. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6619-6628, 2019.
12

---

Noise-Guided Predicate Representation Extraction and Hiffusion-Enhaneee Discretization for Scene Graph Generation
Yu, J., Chai, Y., Wang, Y., Hu, Y., and Wu, Q. Cogtree: Cognition tree loss for unbiased scene graph generation. arXiv preprint arXiv:2009.07526 2020.
A. Implementation Details
We use a pre-trained Faster RCNN (Tang et al., 2020; Ren et al., 2015) for object detection, with the detector frozen during all three tasks. The training process is divided into two phases. First, the basic scene graph generation model (Zellers et al., 2018; Vaswani et al., 2017; Tang et al., 2019) provides coarse-rrained contextual information. which is used for pretraining the Noise-Guided Predicate Representa- tion Extraction module. This is done by leveraging the loss function of the module to constrain the contextual represen- tation of predicates. Next, based on the extracted predicate context representations, we begin to train the two feature enhancement modules. During training, the learning rate is set to 0.001. In the pre-training phase, the batch size is set to 8, and the number of iterations is 60,000. In the feature en- hancement phase, the batch size is set to 8, and the number of iterations is 40,000. All experiments are conducted using four NVIDIA 3090 GPUs, each with 24GB of memory.
Yu, Q., Li, J., Wu, Y., Tang. S., Ji, W., and Zhuang, Y. paromple- language model for fine- fine-grained scene graph generation in open world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21560-21577 2023.
Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5831-5840, 2018.
Zhang, A., Yao, Y., Chen, Q. Ji, W., Liu, Z., Sun, M., and Chua, T.-S. Fine-grained scene graph generation with data transfer. In European conference on computer vision, pp. 409 409-424. Springer, 2022.
Zhang, C., Stepputtis, S., Campbell. J., Sycara, K., and Xie, Y. Hiker-sgg: Hierarchical knowledge enhanced robust scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 28233-28243, 2024.
B. Experimental Analysis
We first integrate NoDIS with existing popular SGG models (Zellers et al., 2018; Zhang et al., 2017; Vaswani et al., 2017: Zheng et al., 2023b) and conduct extensive experiments on three tasks from the VG (Krishna et al., 2017) dataset to val- idate the effectiveness of our method. Existing approaches often use the frequency of different triplets in the dataset as additional prior knowledge, which is weighted against the model's final prediction scores. However, we recognize that in real-world scenarios or with unknown data, such prior information may not be available, potentially affecting the model's performance. Therefore, we do not introduce any prior knowledge or data augmentation techniques during model training or evaluation, aiming to achieve the most reliable results with the simplest approach.
Zhang, H., Kyaw, Z.. Chang, S.-F., and Chua, T.-S. Visual translation embedding network for visual relation detec- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5532-5540, 2017.
Zhang. J.. Shih, K. J., Elgammal. A., Tao, A., and Catanzaro, B. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11535-11543 2019.
Zhang, L.,, Rao, A., and Agrawala, M. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-38444 2023.
As shown in Table I, compared to the best existing methods, our approach achieves relatively better results on mR @ K metrics, indicating that our method strikes a better balance in predicting both head and tail predicates, rather than fa- voring only the head classes. However, our method may show lower @ K scores, as R K reflects local prediction accuracy, while @R@K represents overall prediction per- formance. When the model favors head classes, it leads to improved accuracy for head class predictions, resulting in a higher R@K. K. However, this significantly weakens the model's 's ability to predict tail information. As shown in Fig- ure 6, we can clearly observe that our method significantly improves the prediction accuracy for body and tail classes compared to the baseline model, while still maintaining the baseline model's ability to predict head classes.
Zheng, C., Gao, L., Lyu, X., Zeng, P., El Saddik, A., and Shen, H. T. Dual-branch hybrid learning network for unbiased scene graph generation. IEEE Transactions Circuits and Systems for Video Technology, 2023a.
Zheng, C., Lyu, X.. Gao, L., Dai, B., and Song, J. Prototype- based embedding network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer 2023b. Zhong, Y., Wang, L.. Chen, J., Yu, D., and Li, Y. Compre- hensive image captioning via scene graph decomposition. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 211-229. Springer, 2020.
Vision and Pattern Recognition, pp.
Additionally, for the SGDet task, we compute Recall @ 100 metrics for head, body, and tail classes. As shown in Table 6,
13

---

Noise-Guided Predicate Representation Extraction and Ennhance--Enaaeee Discretization for Scene Graph Generation
existing models tend to have higher accuracy for head class predictions, they are biased toward head categories, over- looking the tail categories with deeper semantic information. By incorporating NoDIS into the existing baseline model, we significantly improve the model's prediction accuracy for the body and tail category, while maintaining its predic- tion performance for the head class. As a result, the model achieves optimal performance in overall mean evaluation. This demonstrates that our approach effectively mitigates biased predictions caused by long-tail distributions, while enhancing the model's prediction accuracy for the body and tail predicate categories at minimal cost.
performed for each module of NoDIS.
C.1. Ablation Study of the Noise-Guided Predicate Representation Extraction Module
First, we perform an ablation analysis of the internal struc- ture of this module. The overall structure is divided into three components: Neighborhood Context Extrac- tion, Noise-Guided Predicate Refinemen,, and Predicate- Prototype Align Module. The Neighborhood Context Ex- traction serves as the foundational predicate context extrac- tion module, providing basic predicate context representa- tions and thus cannot be removed. As shown in Table 8, when the Predicate- Protottye Align module is excluded, the overall performance remains low regardless of whether the Noise- Noise-Guided Predicate Refinement module is used. However, introducing the Predicate- Predicate-Prototype Align mod- ule leads to varying degrees of performance improvement, as demonstrated Rows and 4 of Table This indicates that the Predicate- Predicate-Prototype Align module effectively aligns the extracted predicate context representations with predi- cate prototypes, significantly enhancing the discriminative ability of predicates.
Models
Head
Body
Tail
Mean
G-RCNN
28.6
6.5
0.1
11.7
GPS-Net (Lin et al. 2020)
30.4
8.5
3.8
14.2
NBP (I.iu al., 2023a)
31.7
15.0
8.9
18.5
BGNN (Li al., 202
33.4
13.4
6.4
17.7
GSL et al., 2023c)
33.6
13.5
8.8
18.6
ReIDN (Zhang et al. 2019)
34.1
6.6
1.1
13.9
VTransE (Zhang ct al. 2017)
34.5
7.6
1.1
14.4
Motifs (Zellers et al., 2018)
34.2
8.6
2.1
15.0
MSDN (Li al., 2017)
35.1
5.5
0.0
13.5
VCTree-TDR(Tang al., 2020)
24.5
13.9
0.1
12.8
VTransE-NoDIS (ours)
30:05
15.13
10.84
18.67
Transformer-NoDSS.... (ours)
28.35
19.23
15.16
20.91
PENet-NoDIS (ours)
27.83
19.87
11.03
19.58
Section 3.5 provides detailed explanation of the loss func- tions used in the Koise-Guided Predicate Representation Extraction Module. The basic loss functions L-qa and Teot serve to constrain predicate representations. When neither the Noise- Guide- Predicate Refinement Module nor the Predicate- Protottyee Align Module is introduced, these basic loss functions alone are used to constrain predicate extrac- tion. Therefore, our ablation study focuses on the additional dynamic weighting loss function Lace and the KL diver- gence loss function Lut, which constrains feature distribu- tions.
Table Under the SGDet configuration, we divided the long- tail distribution predicate categories in the VG dataset into three groups: head, body, and tail, and calculated the R@ 100(%) metric for each group.
Finally, we conducted experiments on the GQA (Hudson & Manning, 2019a) dataset across three tasks, as shown in Table 7. Compared to traditional methods, our approach achieves strong performance on all three subtasks. How- ever, we acknowledge that it does not surpass the most recent sate-of-the-art methods ( (Wang et al., 2023b; Biswas & Ji, 2023) on this dataset in terms of evaluation metrics. Nevertheless, considering overall performance, our method demonstrates consistently strong results on both the VG and Open Images datasets, and achieves relatively competitive performance on the GQA dataset as well. This indicates that our approach is effective across different scenarios and re- mains robust under varying scene complexities, highlighting its strong generalization capability.
As shown in Table 9, when only the base loss functions used, the model performs relatively poorly. Introducing the dynamic weighting loss, as seen in the second row, signifi- cantly improves performance. When both loss functions are applied, the model achieves optimal performance. These findings indicate that while adding either loss function in- dividually provides slight performance gains, their effects differ. The dynamic weighting loss focuses on the overall prediction state, resulting in a noticeable improvement in the mR metric. In contrast, the KL divergence loss targets local feature distribution alignment, which may still be influenced by long-tail distributions, leading to biased predictions.
C. Ablation Studies
We conduct ablation studies from two perspectives. First, we independently analyze the components and loss functions of the Moise-Guided Predicate Representation Extraction module. Then, we separately examine the effects of the Feature Enhancement Based on Diffusion module and the Learnable Feature Discretization Mapping module. Our ab- lation experiments are conducted based on the Transformer framework ( (Vaswani et al., 2017), with an ablation analysis
C.2. Ablation Study on Diffusson Steps and Random Sampling Steps in the Feature Enhancement Module
We conducted detailed ablation experiments on the diffusion step size and random sampling step size used in the Feature Enhancement Based Diffusion module. First, for the
14

---

Noise-Guided Predicate Representation Extraction and Hiffusiion-Enaanee Discretization for Scene Graph Generation
PredCIs
SGCIs
SGDet
Methods
mR @ 50/100
F@50/100
mR 50/100
50500000
@R@100/00
5050/100
Motifss (Zellers al., 2018)
16.4/17.1
26.2/27.2
8.2/8.6
13.2/13.8
6.4/7.7
10.5/12.5
+DC (Han al., 2022a)
21.4/22.5
31.7/33.1
9.9/10.4
15.2/15.8 +
9.4/10.7
14.1/16.1
+DHL (Zheng et al., 2023a)
20.4/21.9
-f-
8.4/9.1 -/-
6.6/8.1 -/-
+
+NICE (Li et al., 2022a)
25.4/27.9
34.9/37.8
+
-/-
+NICEST (Li et al., 2022a)
24.2/26.8
34,16/37.23
-/-
-/-
+NoDIS
31.21/32.64
36.44/38.09
15.92/16.65
19.08/19.90
13.76/15.95
17.23/19.81
VCTree (Tang et al., 2019)
16.6/17.4
26.3/27.5
7.9/8.3
12.8/13,4
6.5/7.4
10.6/12.0
+HTCL (Wang et al., 2023b)
32.6/33.9
41.4/42.9
15.7/16.3
20.3/21.0
14.0/15.8
17.3/19.6
+GCL (Dong et al., 2022)
35.4/36.7
39.5/41.1
17.3/18.0
20.0/20.8
15.6/17.8
16.5/19.1
VTranSE (Zhang et al., 2017)
14.0/15.0
22,4/23.8
8.1/8.7
13.0/13.9
5.8/6.6
9.6/10.9
+GCL (Dong et al., 2022)
30.4/32.3
32.8/34.7
16.6/17.4
19.2/20.0
14.7/16.4
15.0/17.2
Transformer (Vaswani et al., 2017)
19.1/20.2
29.5/31.1
9.3/9.7
14.6/15.2
6,717.9
10.7/12.6
+ +DHL (Zheng et al., 2023a)
18.2/20.1
-f-
8.7/9.3
-/-
7.8/8.8
-/-
+NoDIS
31.50/33.44
35.91/37.92
17.23/17.83
20.33/21.05
14.28/16.23
16.73/18.98
Table 7. Comparison with Existing Methods on Three Tasks of the GQA (Hudson & Manning, 2019a) Dataset.
Ieed
Buds
Tall
Head
(a) Recall values for each category with Motifs and after incorporat-
(b) Recall values for each category with Transformer and after incorporating NoDIS.
ing NoDIS.
Ilaad
Body
Tall
(c) Recall values for each category with VTransE and after incorpo- rating NoDIS.
(d) Recall values for each category with PENet and after incorporat-
ing NoDIS.
Figure 6. Recall values for each category on the test dataset before and after incorporating NoDIS, with predicate frequencies decreasing from left right in the dataset.
diffusion process, as shown in Table 10, we experimented with diffusson steps of 50, 100, and 150. Considering the relatively small size of our dataset, commonly used 1000-step setting in traditional diffusion models (Song et al.. 2020a; Chen et al., 2023). During training, a difffusion step is randomly selected from the specified range for step embedding. The results show that
using 50 diffusion steps achieves the best performance in terms of mR @50 and mR@ @ 100. Therefore, we adopt 50 steps in all subsequent experiments.
did not adopt the
Second, regarding the sampling process in this module, we conducted ablation studies on four different random sam- pling step sizes. As the sampling step increases, both train-
15

---

Noise-Guided Predicate Representation Extraction and Contusion-Enhaneee Discretization for Scene Graph Generation
NPR
PPA
R@ @50
R@ 10100/mR@ 100
random steps
R@50/mR@50 50
R@ 100/mR @ 100
X
X
52.92/32.53
55.66/35.16
10
49.96/37.25
53.21/39.97
V
X
59.89/31.76
62.32/33.06
15
47.51/36.81
50.14/39.88
V
47.42/33.19
49.61/36.16 16
20
47.63/36.92
50.26/39.99
V
54.26/35.76
56.63/38.41
25
47.54/36.85
50.15/39.92
Table Ablation Study on Submodules of the Noise-Guided Pred- ieate Representation Extraction Module. NPR refers Noise- Guided Predicate Refinement, and PPA denotes the Predicate- Prototype Align module.
Table 11. The ablation study further explores the of randomly sampled step lengths to constrain the diffuusin process. Following the DDIM-inspired (Song al., 2020a) design, diffusion steps randomly selected for feature reconstruction.
Loce
Lit
R R@ 505mmR@00 550
R@ 1@100/m@@111
X
X
59.46/31.88
61.89/34.22
the discretization mapping approach. With the introduction of a conditional Gaussian noise generator, additional dif- fusion constraint is applied, improving the performance to 39.66%. The aforementioned discretization mapper, as a preprocessing module for the Diffusion reconstruction task, is designed to perform feature mapping on preprocessed predicate contexts. When an additional discrete mapping applied to the representations reconstructed by Diffusion, the model achieves optimal performance.
V
59.18/33.35
61.60/35.98
X
59.84/32.15
62.24/34.40
54.26/35.76
56.63/38.41
Table Ablation Study on the Loss Functions in the Noise-Guided Predicate Representation Extraction Module.
ing and inference time tend to increase to varying degrees. As shown in Table 11, when the sampling step is set to 10, the model achieves favorable performance across multiple metrics, while maintaining reasonable training and testing times. This setting effectively balances performance and efficiency.
From the experimental analysis above, we conclude that the performance gain of a simple Diffusion model on feature reconstruction not significant. This is because the recon- structed predicate representations are unstable and exhibit randomness, which negatively impacts subsequent predicate category predictions. However, introducing additional con- straints into the diffusion process effectively enhances the quality of feature reconstruction, aligning it more closely with the expected feature distribution. Furthermore, directly using the reconstructed predicate representations for proto- type classification can result in instability, as reconstructed predicate representations within the same category may fail to converge, hindering prototype optimization. By incorpo- rating discretization mapping, the alignment between the re- constructed representations and the prototype representation space is improved, facilitating stable category predictions.
diffusion steps
5@50/RR @50 50
R@ 100/mR @ 100
50
49.96/37.25
53.21/39.97
100
50.28/36.86
52.74/39.55
150
49.73/36.90
52.18/39.76
Table 10. In Feature Enhancement based on Difffusion module, an ablation study is conducted on the number of diffusion steps. During the diffusion process, step lengths are randomly selected based on the specified number of steps.
C.3. Ablation Study on Feature Reconstruction Enhancement and Discretization Mapping Module
FED
Randon
Condition
CONG
FDM
Re e:
R 100mmR 1000
We conducted detailed ablation analysis of the Feature Re- construction Enhancement Based on Diffusion module and the Feature Discretization Mapping module. As shown in Table 12, incorporating only the Feature Discretization Map- ping module or the Feature Reconstruction Enhancement Based on Diffusion module leads to a performance drop. This decline attributed to the introduction of additional learnable variable parameters, which fail to converge effec- tively during training. When using the diffusion model for feature reconstruction enhancement, we introduced ran- dom timestep reconstruction constraint to further regulate the reconstruction process, resulting in a slight performance improvement. However, a significant performance boost was observed when combining the diffusion method with
X
42.08/34.75
45.34/37.99
X
54.23/35.75
56.6033.412
X
45.11/36.04
47.95/38.95
A
45,34/36.03
48.13/39.18
45.51/36.18
48.31/39.27
49.98/36.90 52.44/39.66
49,72/36.85
$2,21/39.75
Table 12. Ablation Study of the Feature Enhancement Based on Diffussio (FED) Module and Feature Discretization Mapping (FDM) Module. "Random" indicates feature reconstruction with random time steps, "Condition" introduces the conditional diffu- sion method, denotes providing prior conditions Diffusion. and indicates discretizing the features after Diffusion baatudd reconstruction.
16

---

Noise-Guided Predicate Representation Extraction and Diflusion-Enhaneee Discretization for Scene Graph Generation
D. Quantitative and Qualitative Analysis
representation space. This process effectively aggregates the diversity of predicates within the same category into unified space, enhancing the consistency of predicate rep- resentations within the same category. It also demonstrates that our discretization encoder can effectively learn valid predicate representations for each category.
Quantitative Analysis First, we perform a visual analysis to evaluate the impact of each module in NoDIS on the intra-class variance of predicate representations. As shown in Figures 7a and 7c, the three curves represent the intra- class variance changes of predicates representation within the same class after introducing the Feature Enhancement Based on Diffusion Module and the Feature Discretization Mapping Module. After introducing the Feature Enhance- ment Based on Diffusion Module, we observe a significant increase in intra-class variance across all categories, with an approximately threefold improvement. This indicates that applying the diffusion- based feature enhancement and expanding the visible predicate representation space effec- tively enhances the diversity of within-class predicate repre- sentations, aiding the model in learning more comprehensive representation information. After introducing the Feature Discretization Mapping module for feature aggregation, the intra-class variance significantly decreases, even approach- ing zero. This is because, by applying the discretized feature mapping, we aggregate all scattered features based on their categories. By seeking a unified predicate representation, we group the same- category predicate representations into a single representation space, effectively controlling intra- class variance and ensuring the consistency of intra-class representations. The feature aggregation process enhances the decision layer's ability to learn and capture key infor- mation about the uniqueness of predicates within the same category. As shown in Figures 7b and 7b, we calculate the variance between the input features to the decision layer and the classification weights at the decision layer. After feature aggregation through discretized mapping, we observe that the aggregated features closely resemble the weight distri- bution of the corresponding category in the decision layer, with significant reduction in variance. This demonstrates that the aggregation of features strengthens the effective rep- resentation of predicates within the same category, which in turn adjusts the decision layer's 's classification weights and alleviates decision-making difficulty and confusion.
Qualitative Analysis We conducted a visual qualitative assessment of the model's prediction results on the PredCIs task using the test dataset. As shown in Figure 9, we per- formed comparative analysis between the baseline model and the predictions obtained after incorporating our method. Taking the third row of Figure 9 as an example, the baseline model (PENet(Zheng et al., 2023b), Transformer Vaswaaii et al., 2017)) predicts the relationship between "tree" and "hill" as "on." While "on" is a valid predicate between these two objects, it lacks semantic richness as a head predicate category and fails to provide deeper insight into their re- lationship. In contrast, our method successfully generates more specific and accurate tail predicates, such as "growing on" and "covered in", with confidence scores exceeding 90%.
Additionally, examples like the first and second rows of Figure 9, such as "bus parked on street" and "wheel attached to train," our method produces more precise and semantically informative tail predicates for a variety of ob- ject pairs. It also assigns higher confidence scores to these predicates. These results demonstrate that our method ef- fectively understands and identifies the relational states be- tween objects. Instead of being dominated by generalized head predicates, it assigns object pairs with tail predicates that convey more precise semantic information.
E. Limitation Analysis
This paper introduces the Noise-Guided Predicate Repre- sentation Extraction and (NoDIS) learning method, which is the first to apply diffu- sion models for feature enhancement to mitigate the biased prediction problem caused by long-tail distributions. How- ever, our experiments reveal that while diffusion models exhibit strong generative capabilities, they are somewhat uncontrollable in generating results, especially when used for feature enhancement. As a result, relying solely on diffusion models for feature enhancement yields minimal improvement. To achieve more consistent representations, additional constraints must be incorporated into the diffu- sion process regulate the distribution of generated fea- tures. However, introducing more constraints significantly weakens the diffusion model's generalization ability and generative performance. Therefore, designing a diffusion process for feature-level enhancement that retains the ro- bustness and generalization capacity of the diffusion model
Discretization
Additionally, we perform a visual analysis of the aggrega- tion of predicate representations within the same category in the Feature Discretization Mapping module. In this module, We design a learnable discretization mapping encoder that maps predicate features of the same category into a unified space by calculating the distance between these features and the encoder's representations. Figure 8a shows the distance relationships between 50 randomly selected samples and the encoder's 's representation space. The closer the distance, the higher the score, and the darker the color. Figure 8b illustrates the predicate discrete encoder representation as- signed to each sample. By comparing Figures 8a and 8b, it is evident that after discretization mapping, each sample is distinctly assigned to the corresponding predicate category
17

---

Noise-Guided Predicate Representation Extraction and Cettusion-Enhanedd Discretization for Scene Graph Generation
(a) Intra-class variance variation curves of PENet before and after introducing the Feature Enhancement Based Diffusion Module and Feature Discretization Mapping module.
(b) Variance curves between predicate representations within the same category and their corresponding decision layer weight distri- butions in PENet after integrating the NoDIS module.
(c) Intra-class variance variation of Transformer before and after introducing Feature Enhancement Based Diffusion Module and Feature Discretization Mapping module.
(d) Variance between predicate representations within the same category and their corresponding decision layer weight distri- butions in Transformer after integrating the NoDIS module.
Figure 7. Based on different models, the variance change curves of intra-class variance before and after introducing each module of NoDIS. A larger variance indicates greater distributional difference between the features.
3.0
1.0
0.8
Fredicets Represeratiin Each Samale
Predicats Reprserrttnnn for Each Samale
6.
-.2
-.2
Representation Each
Represontation Each
(a) Score Determination Between Similar Samples When Aggregat- ing Predicate Features via Discretized Feature Mapping.
(b) Sample Indexing for Aggregating Predicate Representations of Same Category via Discretized Feature Mapping.
Figure 8. Visualization and Evaluation of Distance Metric Scores for Aggregation of Same-Class Predicate Representations in Discretized Feature Mapping.
will be key focus of our future work.
18

---

Noise-Guided Predicate Representation Extraction and Diflusion-Enhancel Discretization for Scene Graph Generation
PENe
PENet NoDIS
Thansformer
Transfermer NoDIS
Brutedon
Suitect
Serrest
Sabeet
Iatel
bnbed
PENet
PENet NoDES
Transfermer
Transformer NoDIS
2igh
Gustsdabuid
S-lght
Sovizzaiidd
Atras
Twcted
Dhaildigg
S-tai
Tiched
Strai
Twdted
buckding
Stain
7 wiled
I-aght
Sppola
Higgtt
Appole
PENet NoDil
Transformer
Transformr-NoDDS
Siikig
plaaso
Surin
2yplra
2ylasa
Sylan
Qualitative comparisons conducted the test dataset based the PredCIs task. Black arrows and descriptions indicate correct predicate relationships, red ones represent incorrect predicate relationships, and green ones signify more accurate and superior predicate relationships.
Figure
19


Reviews:
[{"review_id": "HRTs6j38A2", "reviewer": "Reviewer_gBLn", "summary": "The paper introduces a diffusion-based feature enhancement approach to broaden the visual space of predicate representation and improve feature diversity. It first extracts entity representations using a baseline model and refines them with a Transformer for contextualization. Gaussian noise is then applied to the contextualized predicate features, and a diffusion-based model estimates and removes the noise. To address the discrete distribution challenge inherent in diffusion-based models, a mapper module is introduced for predicate classification. The method is evaluated with three different baselines and significantly improves F@K.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your time and effort in carefully reviewing our paper and providing constructive feedback. We are also grateful for your recognition of our work.\n\n- #### *For Weaknesses*: Thank you very much for pointing this out. We classify subject and object entities using the same method as the baseline, which relies on ROI features extracted by Faster R-CNN and determines the entity category using a simple linear layer. We will update Section 3.1 of the paper to include an explanation of this part.\n\n### If you have any further questions or concerns, please let us know, and we will provide additional clarification.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria is appropriate for scene graph generation.", "theoretical_claims": "The paper does not have any theoretical claims. It essentially borrows the variants of Diffusion-based modelling into scene graph generation, especially to model the predicate representation.", "experimental_designs_or_analyses": "The experimental designs follow the standard protocol of SGG. They have demonstrated their performance with sufficient ablation studies.", "relation_to_broader_scientific_literature": "The paper finds an interesting application of diffusion-based denoising in enhancing the visual space of predicates.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper successfully applies the diffusion-based denoising in classifying the predicates of scene graphs. The motivation to employ diffusion-based denoising is well-written and also their proposed method is discussed coherently. Proper ablation studies and evaluation with SOTA baselines demonstrate the superiority of the proposed method. \n\nHowever, the paper lacks in discussing the classification of subject and object. Since they are contextualizing baseline entity features, the entity classification results will be different from the baseline models. Also, a overview of the method should be included in the caption of Figure 2.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJc0uJyg4Y", "reviewer": "Reviewer_giRZ", "summary": "The paper proposes NoDIS, a method designed to address bias in SGG arising from long-tail predicate distributions. The contributions are: 1) it introduces a noise-guided predicate representation extraction technique that employs conditional diffusion models to increase the diversity of predicate representations within the same category. 2) it develops a diffusion-enhanced discretization module to unify similar predicate representations, simplifying decision-layer complexity and improving prediction accuracy. Evaluations on VG and GQA datasets demonstrate that NoDIS outperforms existing unbias methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely appreciate your time and effort in carefully reviewing our paper and providing constructive feedback. We are also grateful for your recognition of our work.\n- *For Weakness 1*: Thank you very much for pointing that out. This work is the first to apply diffusion models for feature enhancement in the SGG task. To facilitate reproduction by other researchers, we have provided detailed implementation details in the appendix(Section: A. Implementation Details), including hyperparameters such as learning rate, number of iterations, and batch size. Additionally, we have submitted the original code as a supplementary file. Upon acceptance of the paper, we will upload the code to GitHub and include the repository link in the paper for broader accessibility.\n- *For Weakness 2*: Thank you for your constructive feedback. We followed the same training procedure and evaluation criteria to validate the reliability of our method on Open Images V4 and V6, as shown in the table below. Compared to recent approaches, our method demonstrates a significant advantage on the Open Images dataset, achieving the best performance in wmAP_rel, wmAP_phr, and score_wtd. This highlights our method's strong capability in both relationship detection and phrase-level consistency assessment, effectively and accurately capturing relationship details between objects. A detailed comparison and analysis will be included in the final paper.\n|Dataset|Method|mR@50|R@50|F@50|wmAP_rel|wmAP_phr|score_wtd|\n|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n|OI V4|RelDN|70.4|**75.7**|73.0|36.1|39.9|45.2|\n|OI V4|GPS-Net |69.5|74.66| 72.0 |35.0|39.4|44.7|\n|OI V4|BGNN |**72.1**|75.5|**73.8**|37.8|41.7|46.9|\n|OI V4|**Transformer-NoDIS(ours)**|70.34|74.84|72.52|**38.21**|**42.35**|**47.19**|\n|OI V6|DBiased|42.1|74.6|53.8|34.3|34.4|42.3|\n|OI V6|PENet|39.3|**76.5**|51.9|36.6|37.4|44.9|\n|OI V6|SGTR|42.6|59.9|49.8|38.7|37.0|42.3|\n|OI V6|CSL|41.7|75.4|53.7|34.3|35.4|42.9|\n|OI V6|BCTR|48.8|68.6|57.0|36.0|**39.0**|42.5|\n|OI V6|SQUAT|-|75.8|-|34.9|35.9|43.5|\n|OI V6|**Transformer-NoDIS(ours)**|**48.93**|74.11|**58.94**|**38.87**|**38.95**|**45.95**|\n- *For Minor Suggestion*: Thank you for your suggestions. In Section 3.2, we randomly initialize learnable Query Tokens based on a Gaussian distribution to extract independent predicate information from the entity representation space. By leveraging a coarse-grained denoising process, entities are treated as noise, allowing the Query Token to extract predicate representations independently of entities, thereby preventing entity influence on predicate classification. For the ablation study in Table 4, we will restructure it to analyze different aspects, including the denoising method, conditional introduction, denoising steps, and feature discretization mapping. Due to page limitations, we have simplified the overall experimental results. Additionally, we will include our failed experiments in the final paper appendix. We sincerely appreciate your thorough review of our paper!\n- *For Suggestion 1*: Thank you very much for your constructive suggestions. Due to space limitations, we have not included more failure cases and analysis. We will add additional visualizations of both successful and failed cases, along with further analysis, in the appendix. We sincerely appreciate your thorough review of our paper! \n- *For Question 1*: Thank you for your question. Our method differs from diffusion-based approaches like SD and DiT. Instead of a full diffusion model, **we adopt the diffusion concept and use only three Cross-Attention layers and three Linear layers in the noise prediction module**. The table below presents a comparison of overall model parameters. Additionally, we conducted experiments not only on the VG dataset but also on large-scale datasets such as Open Images and GQA (results in reviewer FDij). Our method outperforms existing approaches, demonstrating an effective trade-off between resource efficiency and performance while also proving its robustness.\n    |Method|Params(M)|\n    |:------:|:------:|\n    |EBM|322.2|\n    |VCTree-TDE|361.3|\n    |VCTree-EBM|372.5|\n    |BGNN | 341.9|\n    |**VCTree-NoDIS(ours)**|**306.06**|\n    |**Motifs-NoDIS(ours)**|**314.8**|\n    |**Transformer-NoDIS(ours)**|**295.42**|\n- *For Question 2*: We appreciate your question. We previously attempted online feature enhancement based on VAE and GAN, but these methods often encountered gradient explosion or vanishing issues during training. Our analysis indicates that VAE and GAN heavily rely on pre-trained weights. Moreover, **both methods(VAEs, GANs) model the overall pixel information of an image, whereas the SGG task determines relationships based on localized ROI features**. As a result, these methods fail to train effectively in this task and cannot enhance feature representation.\n\n### If you have any further questions or concerns, please let us know, and we will provide additional clarification.", "claims_and_evidence": "1. The authors claim improved generalization by diversifying predicate representations and reducing prediction ambiguity via discretization mapping. These claims are convincingly supported by experiments on VG and GQA datasets.\n2. The quantitative analysis demonstrates improvements over baselines, especially in terms of mR@K, indicating effective bias mitigation.\n3. Ablation studies effectively validate the impact of each proposed module.", "methods_and_evaluation_criteria": "1. standard benchmark datasets (VG and GQA) aligns well with community standards.\n2. the evaluation metrics (R@K, mR@K, F@K) are conventional and sensible for measuring performance in SGG, particularly mR@K for capturing both head and tail predicate predictions.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound:\n1. ablation studies systematically demonstrate the contributions of each module (Noise-Guided Predicate Refinement, Diffusion-Enhanced Feature Enhancement, and Discretization Mapping).\n2. Comparisons with multiple state-of-the-art methods across established benchmarks validate the effectiveness of the proposed method comprehensively.", "relation_to_broader_scientific_literature": "The paper situates itself within recent SGG literature, clearly discussing related prior methods like Motifs and RelTR, and identifying their limitations, especially regarding biases and handling tail classes. The paper differentiates itself by introducing novel diffusion-based methods and discretization techniques, improving over previous augmentation and feature enhancement methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. Clearly motivated approach to tackling long-tail distributions and bias.\n2. Innovative integration of diffusion models for feature diversification.\n3. Strong empirical validation and ablation studies.\n\nWeaknesses:\n1. Complexity in the methodology might limit reproducibility without extensive supplementary details.\n2. Limited exploration on generalizability beyond standard benchmarks, and also can explore the results in PSG and OpenImage dataset.", "other_comments_or_suggestions": "Minor suggestions and typos:\n1. Typos/Minor corrections:\n - Section 3.2: clarify more explicitly the role of the Query Token initialization.\n - Table formatting (eg, Table 4 in the ablation studies section) could be improved slightly for readability.\n\nSuggestion:\n1. Provide additional qualitative visualization examples or failure cases for more transparency", "questions_for_authors": "1. Diffusion Complexity: given the computational cost associated with diffusion models, can you clarify how scalable your approach is to larger-scale datasets or real-world scenarios?\ner will help assess the generalization and robustness of the method beyond benchmarks.)\n2. Comparison with Other Generative Models: Why was diffusion specifically chosen for feature augmentation over alternative generative models such as GANs or VAEs? If use GANs or VAEs, what's the performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mFBH8ygAua", "reviewer": "Reviewer_FDij", "summary": "The paper addresses bias in scene graph generation, especially the difficulty of learning fine-grained predicate labels under long-tailed distributions. The authors propose NoDIS, which has two core aspects: First, Noise-Guided Predicate Extraction expands predicate representations by applying a single-step noise addition and iterative denoising, then further augments this diversity through a conditional diffusion mechanism. This helps generate richer, more diverse representations for both frequent (head) and infrequent (tail) predicates. Second, Feature Discretization Mapping learns to aggregate these expanded predicate features into a discretized representation. This step reduces ambiguity for the classification head by unifying predicate samples of the same category into a more consistent cluster. Empirical evaluations on the Visual Genome and GQA benchmarks demonstrate that adding NoDIS boosts the performance on rare classes. Ablation studies confirm that both the diffusion-based augmentation and the discretization module are integral to the performance gains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely appreciate your time and effort in carefully reviewing our paper and providing constructive feedback. We are also grateful for your recognition of our work.\n- *For Weakness 1*: Thank you for bringing this up. Figure 2(b) illustrates both the training and inference processes, making it relatively complex. We have now optimized the diagram by simplifying redundant module structures, resulting in a clearer and more concise representation.\n- *For Weakness2*: Thank you for pointing this out. We follow the existing noise design of SD and DiT, adopting a linear noise schedule. To further investigate the impact of diffusion steps on performance, we conduct an ablation study with 50, 100, and 150 steps, as shown in Table 1. Additionally, to examine the effect of step size during the denoising process, we perform another ablation study with step sizes of 10, 15, 20, and 25, as presented in Table 2. A more detailed ablation analysis will be provided in the final paper.\n|Diffusion steps|random steps| R@50/mR@50 | R@100/mR@100 | F@50/F@100 |\n|:------:|:------:|:------:|:------:|:------:|\n| 50 | 10 | 49.96/**37.25**  | **53.21**/**39.97**  | **42.68**/**45.65** |\n| 100 | 10 | **50.28**/36.86  | 52.74/39.54 | 42.54/45.20 |\n| 150 | 10 |  49.73/36.90 | 52.18/39.76  | 42.36/45.13 |\n\n    |random steps|R@50/mR@50|R@100/mR@100|F@50/F@100|\n    |:------:|:------:|:------:|:------:|\n    |10|**49.96**/**37.25**|**53.21**/39.97|**42.68**/**45.65**|\n    |15 | 47.51/36.81| 50.14/39.88  | 41.48/44.42 |\n    |20 | 47.63/36.92| 50.26/**39.99**  | 41.60/44.54 |\n    |25 | 47.54/36.85|50.15/39.92|41.52/44.46|\n- *For Weakness 3*: Thank you for your suggestion. On the VG dataset for the PredCls task, using Transformer as the baseline, our method outperforms the approach published in TMM 2024 by 0.35 and the one in TIP 2024 by 0.95 in F@100. In terms of overall performance, with PE-Net as the baseline, our method surpasses the CVPR 2024 approach by 0.73 in mR@100. Additionally, on the Open Images dataset, based on the score_wtd metric (as referenced in Reviewer giRZ's response), our method improves upon the best existing approach by 0.29 on OI V4 and by 1.05 on OI V6.\n- *For Suggestions*: Thank you for your suggestion. We have optimized the content accordingly and will present it in the final paper. We sincerely appreciate your thorough review of our paper!\n- *For Questions 1*: Thank you for pointing this out. In Section 4.3 and Table 4 of the paper, we briefly analyze the necessity of incorporating a random denoising strategy for supervised optimization. During the forward diffusion process, **our noise scheduling strategy follows that of SD and DiT**. To better enforce consistency in the generated representations of the enhanced model, we introduce an additional denoising strategy by randomly sampling n steps for denoising as an extra constraint. During inference, we adopt the DDPM denoising strategy to iteratively remove noise step by step. **Experimental results on different diffusion and denoising step selections are presented in For Weakness2**. We will include this analysis in the final version of the paper with a more detailed discussion.\n- *For Questions 2*: Thank you for pointing this out. Our initial approach considered using the K-nearest neighbors (KNN) method. However, since Diffusion generates diverse features in the early stages, predicate representations within the same category exhibit significant variation, making it ineffective to constrain them using KNN. Instead, we maintain a component similar to a CodeBook to learn a central representation from the representation space of multiple samples within the same category. This central representation is designed to have strong generalization capabilities. In contrast, KNN and clustering methods are too restrictive and struggle to model diverse representations effectively.\n- *For Questions 3*: Thank you for pointing this out, and we apologize for not including it in the paper due to space limitations. The table below compares our method with existing approaches on the PredCLS task of the GQA dataset. Our method achieves further performance improvements over existing methods. More comprehensive evaluation results across multiple tasks will be included in the final version of the paper. Additionally, we have supplemented performance improvements on the Open Images V4 and V6 datasets (results in Reviewer giRZ), which will also be included in the final version.\n| Method  | R@50 | mR@50  | F@50  | R@100 | mR@100  | F@100  |\n|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| Motifs | **65.3** | 16.4 | 26.2 | **66.8** | 17.1 | 27.2 |\n| Motifs-DC |61.3|21.4|31.7|62.7|22.5|33.1|\n| Transformer |65.2|19.1|29.5|66.7|20.2|31.1|\n| Transformer-DHL|-|18.2|-|-|20.1|-|\n| **Transformer-NoDIS(Ours)**|46.54|**30.47**|**36.83**|48.45|**31.90**|**38.47**|\n\n### If you have any further questions or concerns, please let us know, and we will provide additional clarification.", "claims_and_evidence": "1. The paper states that existing SGG approaches overlook intraclass diversity and interclass consistency in predicate representations, leading to biased predictions favoring head classes. They show that prior efforts primarily focus on loss reweighting or sampling strategies, and only partially address the wide range of predicate semantics. Quantitative results (particularly in mR@K metrics) show that standard baselines often fail to predict tail predicates accurately.\n\n2. The proposed NoDIS method, by applying noise-guided extraction and a conditional diffusion model, effectively expands the representation space of predicates, improving the recall on long-tailed predicates. Comparisons on the VG dataset indicate that the average performance on tail categories (mR@K) is consistently higher compared to baselines. Their ablation study further isolates the impact of the diffusion approach on increasing within-class variance of predicate features.\n\n3. The discretization mapper helps unify predicate features that belong to the same category, improving classification accuracy and reducing confusion among semantically similar predicates. The authors show a measurable drop in intraclass feature variance once discretization is applied and an alignment between the learned representation distributions and the final decision layer. This is reinforced by the improvement in F@K and mR@K metrics.\n\nWhile the paper does not provide an in-depth theoretical proof that this particular combination of modules is uniquely optimal, their experimental findings and ablations make a convincing empirical case that NoDIS alleviates biases and achieves strong performance on standard SGG benchmarks.", "methods_and_evaluation_criteria": "The authors propose two-phase design: (1) object detection, (2) predicate classification enhanced by noise-guided extraction and conditional diffusion. Noise injection and denoising are used to create more distinct predicate embeddings (intra-class diversity). A discretization mapping collapses these diverse embeddings for each category, reducing confusion at the classifier stage. The chosen evaluations are well aligned: the authors address data skew and demonstrate improvements on established unbiased metrics (mR@K, F@K).", "theoretical_claims": "The paper’s theoretical contributions are mostly method-centric rather than purely formal proofs. The authors do mention a KL-divergence loss to align representation distributions and reference how the diffusion model preserves or shifts distributions across timesteps. While they do not include a formal proof of correctness or convergence for the diffusion approach, the claims rely on standard diffusion modeling assumptions. I did not see any outright flaws in their theoretical arguments, though the proofs and formal rigor focus on standard diffusion derivations and established VAE/GAN-based feature augmentation concepts rather than novel mathematics.", "experimental_designs_or_analyses": "The authors compare performance on three tasks (PredCls, SGCls, SGDet) using widely adopted metrics (R@K, mR@K, F@K).\nAll in all, the experimental protocols appear sound and consistent with common SGG benchmarks.\nOne area for improvement might be clarifying whether their random noise schedules significantly impact results or whether different hyperparameter choices matter (e.g., number of diffusion steps).", "relation_to_broader_scientific_literature": "- Long-tailed recognition: Instead of conventional class-rebalancing or cost-sensitive methods, they utilize a generative model (diffusion) to expand underrepresented classes.\n- SGG bias reduction: Past works have used sampling, reweighting, or knowledge distillation. NoDIS is somewhat unique in employing a diffusion-based pipeline to construct richer embeddings.\n- Feature discretization: The idea of quantizing or discretizing features with a VQ-VAE–style approach to unify sample clusters seems also a novel adaptation in SGG.", "essential_references_not_discussed": "The references to mainstream SGG and diffusion-based generation are mostly in place.", "other_strengths_and_weaknesses": "Strengths:\n1. The conceptual combination of diffusion-based expansion plus discretized representation is novel, improving both the expressiveness (helpful for tail classes) and consistency (for simpler classification).\n2. Improved tail performance is clearly demonstrated by the metric gains on mR@K.\n\nWeaknesses:\n1. Figure 2(b) is indeed somewhat hard to parse, due to the overlapping gray/dotted arrows and the large number of parallel lines (the textual explanation helps, but the graphic is busy).\n2. Hyperparameter sensitivities (e.g., noise schedule, number of diffusion steps, discretization granularity) are not deeply explored. In practice, these can matter greatly for diffusion-based methods.\n3. The inclusion of more recent baselines would strengthen the comparisons and provide a more comprehensive evaluation.", "other_comments_or_suggestions": "- Experimental table (Table 1) might benefit from consistent significant figures of results.\n- If possible, consider making the figure for NoDIS more straightforward by reducing the complexity of parallel branches or clarifying the ordering of steps in the visual layout.", "questions_for_authors": "1. How sensitive are results to different noise schedules or varying the number of denoising steps?\n2. At inference, once you generate expanded predicate features, are they discretized solely by a nearest-neighbor approach to a learned embedding codebook? Would an alternative approach (e.g., approximate nearest neighbors or distinct heads for each class) help or hinder the method?\n3. Have you considered deeper comparisons or breakdowns of tail performance specifically on GQA? Because GQA has many predicate categories, this might better showcase the benefit of diffusion-based augmentation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VwuNOkneXG", "reviewer": "Reviewer_yxqX", "summary": "This study proposes a Noise-Guided Predicate Representation Extraction and Diffusion-Enhanced Discretization (NoDIS) technique to solve the long-tail problem inherent in the existing dataset for learning the Scene Graph Generation (SSG) model. The main contribution is that the existing proto-type learning method used diffusion pliocytosis as a method for expanding the expression space of the predicate feature. Additionally, feature gathers as a decision boundary that can occur as the representation space is determined, and a discretization mapper is introduced to prevent inference performance degradation. This study was verified on the benchmark for SGG and proved effective in alleviating the long-tail problem.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely appreciate your time and effort in carefully reviewing our paper and providing constructive feedback. We are also grateful for your recognition of our work.\n- *For Weakness 1*: Thank you for bringing this up. First, we compared and analyzed our method against state-of-the-art approaches using the same evaluation framework on the VG dataset (in Section 4.2 and Table 1). Additionally, **we conducted comparisons on the Open Images V4, V6 (results in Reviewer giRZ), and GQA (results in Reviewer FDij) datasets**. Finally, we performed an ablation study by breaking down our method and analyzing its effectiveness from multiple perspectives, **including inter-module effectiveness (in Table 2), the impact of the diffusion method (in Table 4), loss function design (in Table 3), diffusion step size (in Reviewer FDij), and denoising step size (in Reviewer FDij)**.\n- *For Weakness 2*: Thank you for pointing this out. Current diffusion-based methods are primarily used for generation tasks. Some studies also apply diffusion for data augmentation by generating additional pseudo-samples to enhance tail information and improve model performance. We have included this discussion in the related work section of our paper.\n- *For Weakness 3*: Thank you for pointing this out. **Although existing LLMs demonstrate strong image understanding capabilities, they widely suffer from hallucination issues, largely due to data bias**. For example, when asked about the relationship between \"person\" and \"short\" in an image, LLaVA incorrectly responds: \"The person is standing next to a short.\" However, when guided by NoDIS, LLaVA correctly outputs: \"The person is wearing shorts.\" We will include more details and additional results in the appendix. Therefore, mitigating biased outputs caused by data bias remains a critical challenge. Moreover, while LVLMs have extremely large parameter sizes, NoDIS contains only around 300M parameters (in reviewer giRZ), requiring significantly less GPU memory for training. Optimizing and experimenting with smaller models can better support large models and help alleviate biased predictions.\n- *For Question 1*: Thank you for pointing that out. We previously used t-SNE for visualizing the representation space. However, we considered that t-SNE itself may have some randomness (as it requires dimensionality reduction). Additionally, after feature discretization mapping, our method clusters multiple samples of similar predicates into a unified representation space. As a result, visually, it appears as a distinct small region, whereas PE-Net appears as a more scattered large region, which does not look aesthetically pleasing. Therefore, instead of using t-SNE for visualization, we calculated inter-feature relationships and presented the results numerically. We chose not to include the t-SNE results in the paper, but we can certainly add them in the appendix of the final paper.\n- *For Question 2*: Thank you for pointing this out. In SGG tasks, no existing work has utilized Diffusion models to enhance features for expanding the tail representation space. Instead, most approaches rely on pretrained Diffusion or GAN models for data augmentation, which incurs higher computational costs. Some methods expand feature representations by training a decoder and applying a distillation-like approach, as briefly mentioned in the Introduction. We will include a detailed discussion of these methods in the Related Work section.\n- *For Question 3*: Apologies for the omission due to space constraints. **We not only evaluated our method on GQA but also conducted experiments on Open Images V4 and V6**. **The GQA results can be found in our response to Reviewer FDij**, while **the Open Images V4 and V6 results are provided in our response to Reviewer giRZ**. These results demonstrate that our method achieves outstanding performance across multiple datasets, highlighting its strong generalization capability.\n- *For Question 4*: Thank you for pointing this out. Two-stage methods primarily rely on Faster R-CNN's entity detection capabilities to extract entity representations and construct relationship information by independently training a relationship detection module. In contrast, single-stage frameworks based on the Transformer architecture perform object detection and relationship classification simultaneously. While single-stage methods are simpler, they require higher training costs. Moreover, existing single-stage methods are relatively scarce and generally underperform compared to two-stage methods, as shown in the table.\n|Method|Stage|mR@50|mR@100|\n|:-------:|:-------:|:-------:|:-------:|\n|RelTR|One-stage|10.8|-|\n|SSR-CNN|One-stage|8.4|10.0|\n|Iterative SGG|One-stage|8.0|8.8|\n|Relationformer|One-stage|9.3|10.7|\n|SGTR|One-stage|12.0|15.2|\n|**Transformer-NoDIS(ours)**|two-stage|**16.91**|**19.25**|\n\n### If you have any further questions or concerns, please let us know, and we will provide additional clarification.", "claims_and_evidence": "- In this paper, the proposed NoDIS can improve prediction of feature. This is consistent with my intuition, but I have questions about whether it can be argued in general. It can be said that the diversity of feature representation may increase in the process of adding noise and de-noise during the diffusion process, but there is a lack of clear evidence (the effect of NoDIS can be confirmed by the ablation of Tab. 2 in this text, but I think it is too fragmentary evidence). As in the PE-Net study used as a baseline in this paper, if the results of feature representation through t-SNE are attached, the trust in the argument will be improved.\n- The Learnable Feature Discretization Mapping Module proposed in Section 3.4 is shown to avoid confusion between inter-classes that may occur during the diffusion process. If so, it is thought that the change in feature distance between inter-classes is also necessary, as reported in Figure 1 (b) of the text for the change in variance between the intra-classes. If the module worked effectively, the feature distance between the inter-classes would increase further.", "methods_and_evaluation_criteria": "- The proposed model was evaluated on the existing two-stage-based SGG model, which can be seen as fair progress in terms of performance comparison with the existing two-state model.\n- The benchmark data used in the experiment, Visual Genome, and GQA datasets, are used as major data for SGG performance evaluation, which can be considered valid as data sets for performance comparison experiments.\n- All three subtasks (PredCls, SGCls, SGDet) performances that are key to SGG performance comparison are reported, and R@100 values per head, body, and tail class are well reported for proof of long-tail problem mitigation.\n- The Implementation Detail aspect for reproducibility of experimental performance is somewhat inadequate. Values such as learning rate scheduler and feature dimension used in the experiment are missing.\n- It seems that the experimental results on the GQA dataset are missing. The paper says that it was used as performance evaluation data in sec. 4.1, but we can't find the experimental results in either the main or the supplement.", "theoretical_claims": "- This paper was not accompanied by any specific mathematical proof.\n- Although the paper argues for feature representation enhancement through diffusion policy, the theoretical basis for this is lacking.\n- All proofs of the claims have been performed as inductive proofs, but I believe that additional strong experimental evidence of the claims is needed.", "experimental_designs_or_analyses": "- The Visual Genome and GQA datasets used in the experiment are used as major data sets for SGG performance evaluation, and there seems to be no problem in selection.\n- Models set as performance comparators are two-stage-based models and are suitable as performance comparators.\n- GQA is mentioned as the usage dataset, but not found in the actual experimental table. Without experiments on additional benchmark datasets, the claims of model's general performance improvement are highly problematic.\n- It is reasonable to report the recall values of each head, body, and tail class to prove the Long-tail problem mitigation, but the comparison group is not appropriate. For the proposed effect of NoDIS, it is considered correct that baseline, baseline + prototype, baseline + NoDIS, and baseline + all are used to prove the effectiveness.\n- The above problem is also shown in Tab. 2 of ablation study, which shows better performance when using only prototype than when using NoDIS only. This can be judged as the main contribution of the performance increase by the prototype learning method rather than the proposed NoDIS.\n- Tab. 6, 7 in Appendix is the same experiment as in text Tab. 2, 3. If it's unnecessary, it's right to remove it.", "relation_to_broader_scientific_literature": "- Although the research on Scene graph is well organized, the general content of Diffusion is far from improving feature representation in the case of Diffusion studies. To reinforce the research claim, the reinforcement of feature presentation through the Diffusion policy should be added to the relatiob work.\n- The proposed paper focused on two-stage-based SGG models, but recent research trends have mainly focused on one-stage, which can train SGG models end-to-end. In addition, with the development of the Large Language Model (LLM), this study feels somewhat old compared to the study dealing with long-tail research through open-vocabulary with LLM.", "essential_references_not_discussed": "- There is a missing one of the latest papers dealing with the long-tail issue based on two-stage, the same as in this study. Please refer to it.\n\nKim, Hyeongjin, et al. \"Scene graph generation strategy with co-occurrence knowledge and learnable term frequency.\" Proceedings of the 41st International Conference on Machine Learning. 2024.", "other_strengths_and_weaknesses": "Strength \n- The introduction of a diffusion policy for solving the long-tail problem, which is the challenge problem of the existing scene graph dataset, is evaluated creatively.\n- The paper explains the proposed method in detail, and the formula and algorithm table are well organized.\n\nWeakness\n- Experiments to prove effectiveness on the proposed NoDIS are somewhat lacking\n- Lack of related work content. In order to claim the improvement of feature presentation through the diffusion policy claimed in this paper, research related to this must be added.\n- It feels somewhat old compared to open-vocabulary studies using the latest one-stage or LLM.", "other_comments_or_suggestions": "- Many figure diagrams are missing descriptions of x-axis and y-axis. There is an inconvenience in interpreting the diagram.\n- The formula is well written, but it is difficult to follow because there are many identical expressions. For example, in the case of Eq. 5 , it is difficult to distinguish between pre- and post-operation T.\n- The same expression in Eq. 10 is ambiguous. My personal understanding is that you only use the Q_p vectors that you have concatenated when constructing T, but we need to use a better expression. Mathematically, it seems that we only use the first element.\n- The letter and arrow are overlaid at the top (c) of Figure 2, which needs to be corrected.\n- Tab. 2 looks better to replace with Tab. 6 from Appendix. Both experiments are the same, and for Tab. 6, both NPR and PPA are reported to have unused baseline performance, making it easier for experimental evaluation.\n- Unification is necessary when referring to Figure in the text. Some are denoted as Figure xb, while others are denoted as Figure x(b).", "questions_for_authors": "- To verify the effectiveness of the proposed NoDIS, experiments on the change of feature distance between t-SNE and inter-classes such as in PE-Net are needed.\n- What is the basis for improving feature representation when using the Diffusion policy? If you have that study, it is recommended to add it to the related work. This enhances the reader's understanding of the thesis. \n- Is the experiment on GQA missing? As far as I've reviewed, it's nowhere to be seen in the text and Appendix. This takes a toll on the suggestion method generalization argument.\n- The latest SGG research trend is dominated by the one-stage method, and I wonder why you adopted the two-stage method. Is there any particular reason?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received overall positive reviews during the initial review period. The reviewers appreciated the core contributions and the novelty of the proposed approach. Some weaknesses were noted, including limited discussion of subject-object classification, missing GQA results, and suggestions to improve visualizations and explore hyperparameter sensitivity. The authors provided a rebuttal that effectively addresses many of these concerns. The AC concurs with the reviewers that this paper presents a novel approach to the SGG community and recommends acceptance. The authors are encouraged to consider the suggestions in the final version, particularly around clarifying figures and expanding results on additional datasets.
Decision: Accept (poster)