Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Authors: Kibum Kim
Kanghoon Yoon
Yeonjun In
Jinyoung Moon
Donghyun Kim
Chanyoung Park
Venue: ICLR 2024 poster (2024)
Keywords: Scene Graph Generation
Scene Understanding
Imbalanced Classification
Self-training
Long-tailed Problem
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This work proposes a novel self-training method for scene graph generation that assigns pseudo-labels for unannotated triplets to enhance the scene representation, which is challenging to adopt due to the unique nature of scene graph generation.

Abstract: Scene graph generation (SGG) models have suffered from inherent problems regarding the benchmark datasets such as the long-tailed predicate distribution and missing annotation problems. In this work, we aim to alleviate the long-tailed problem of SGG by utilizing unannotated triplets. To this end, we introduce a **S**elf-**T**raining framework for **SGG** **(ST-SGG)** that assigns pseudo-labels for unannotated triplets based on which the SGG models are trained. While there has been significant progress in self-training for image recognition, designing a self-training framework for the SGG task is more challenging due to its inherent nature such as the semantic ambiguity and the long-tailed distribution of predicate classes. Hence, we propose a novel pseudo-labeling technique for SGG, called **C**lass-specific **A**daptive **T**hresholding with **M**omentum **(CATM)**, which is a model-agnostic framework that can be applied to any existing SGG models. Furthermore, we devise a graph structure learner (GSL) that is beneficial when adopting our proposed self-training framework to the state-of-the-art message-passing neural network (MPNN)-based SGG models. Our extensive experiments verify the effectiveness of ST-SGG on various SGG models, particularly in enhancing the performance on fine-grained predicate classes.

Published as a conference paper at ICLR 2024
ADAPTIVE SELF-TRAINING FRAMEWORK FOR FINE- GRAINED SCENE GRAPH GENERATION
Kibum Kim 1 Kanghoon Yoon '+ Yeonjun In'
Jinyoung Moon Donghyun Kim²
Chanyoung Park
KAIST 2ETRI 'Korea University
[kb. kim, ykhoon0B, yeonjun. in, cy. park) @kaist. kr
jymoonBetri. re.ieirre.ee kr, o @_kimekorea.ac..
ABSTRACT
Scene graph generation (SGG) models have suffered from inherent problems regarding the benchmark datasets such as the long-tailed predicate distribution and missing annotation problems. this work, we aim to alleviate the long-tailed problem of SGG by utilizing unannotated triplets. To this end, we introduce a Self-Truining framework for SGG (ST-SGG) that assigns pseudo-labels for unannotated triplets based on which the SGG models are trained. While there has been significant progress in self-training for image recognition. designing a self-training framework for the SGG task is more challenging due to its inherent nature such as the semantic ambiguity and the long-tailed distribution of predicate classes. Hence, we propose a novel pseudo-labeling technique for SGG, called Class-specific Adaptive Thresholding with Momentum (CATM), which is a model-agnostic framework that can be applied to any existing SGG models. Furthermore, devise graph structure learner (GSL) that is beneficial when adopting our proposed self-training framework to the state-of-the-au messagg-passing neural network (MPNN)-based SGG models. Our extensive experiments verify the effectiveness of ST-SGG on various SGG models, particularly in enhancing the performance on fine-grained predicate classes. Our code is available on https://githubb.c cowp:/iaaaaoooa/aaaaaaaaaaaaaaal 107/torch-ST-SGC
INTRODUCTION
Scene graph generation (SGG) is a task designed to provide a structured understanding of a scene by transforming scene of an image into compositional representation that consists of multiple triplets in the form of (subject, predicate, object). Existing SGG methods have faced challenges due to inherent issues in the benchmark datasets (Krishna et al. 2017 Lu et al.) 2016). such as the long- tailed predicate distribution and missing annotations for predicates (see Fig. Specifically, the long-tailed predicate distribution in SGG refers to a distribution in which general predicates (e.g. "on") frequently appear, while fine-grained predicates (e.g. "walking in") are rarely present. Owing to such a long-tailed predicate distribution inherent in the dataset, existing SGG models tend to make accurate predictions for general predicates, while making incorrect predictions for fine-grained predicates. However, scene graphs primarily composed of general predicates are less informative depicting a scene, which in turn diminishes their utility across range SGG downstream applications. Besides the challenge posed by the long-tailed predicate distribution, benchmark scene graph datasets (c.g.. Visual Genome (VG) (Krishna et al. (017)) encounter the problem of missing annotations. Specifically, missing annotations provide incorrect supervision (Zhang et al.) 2020) to SGG models as unannotated triplets are carelessly assigned to the background class (i.e.. bg), even though some unannotated triplets should have been indeed annotated with another class (See Fig. T aa)) For example, treating the missing annotation between person and sidewalk bg may confuse SGG models that are already trained with a triplet (person, walking in, sidewalk). In addition, the prevalence of triplets with the bg predicates between person and sidewalk throughout the dataset would exacerbate the problem. Addressing the missing annotation problem especially important as a large of volume of benchmark scene graph datasets includes the bg class (e.g., 95.5% of the triplets are annotated with the bg class VG dataset.
"Equal Contribution Corresponding Author

---

Published as a conference paper at ICLR 2024
Long-talled Predicate Distribution
(2) Re- brain
Ground Truth ... Maxing Annotation
Probubiiity
person
welkingiin in
SGG model
sidewall
bogr
predicate
reseting wathing d
boy
on
Missing A Anooaatio Problem
person
hair
Lirmmotaled (bockqurandd
hea
4.5%
Minsing
True background
(1) Paaubb-aaell
(a) Inherent Noise in Visual Genome Dataset
(b) Self-training for SGG
Figure 1: (a) Problems in VG scene graph dataset, and (b) self-training framework for SGG.
To alleviate the challenge posed by the long-tailed predicate distribution, recent SGG methods have presented resampling (Desai al.) 2021; Li et al., (02I), reweighting (Yan et al. 2020, Lyu et al. 2022), and various debiasing strategies (Tang et al., 2020; Guo et al.) 2021] Chiou et al, 2021 Dong et al.] 2022). However, these approaches simply adjust the proportion of the training data or the weight of the training objective without explicitly increasing the diversity of the data, which eventually leads to the model's overfitting to minority classes. IE-Trans (Zhng et al. 2022] is the most recent method proposed to increase the variation of the training data, either by replacing general predicates (i.e., the majority) with the fine-grained predicates (i.e., the minority) or by filling in missing annotations with fine-grained predicates. This study demonstrates that discovering informative triplets that have not been annotated helps alleviate the problem caused by the long-tailed predicate distribution, and improves the model's generalization performance. However, IE-trans involves many incorrect pseudo-labels in the training process as it relies solely on the initial parameters of the pre-trained model to generate pseudo-labels of the entire dataset in a one-shot manner. Hence, IE-trans fails to fully exploit the true labels of unannotated triplets.
In this paper, we aim to effectively utilize the unannotated triplets in benchmark scene graph datasets by assigning them accurate pseudo-labels. To this end, we introduce a self-training framework for SGG, called ST-SGG, which assigns pseudo-labels confident predictions among unannotated triplets, and iteratively trains the SGG model based on them (Fig. (b)). ST-SGG reduces the number of incorrect pscudo-labels by iteratively updating both the pseudo-labels and the SGG model at every batch step. This in turn mutually enhances the quality of pseudo-labels and the SGG model, which results in an effective use of unannotated triplets. While there has been significant progress in self-training for image recognition (Sohn et al. 2020 Xie et al., 2020b Lee et al.] 2013 Xu ee al. 2021). designing a self-training framework for the SGG task is more challenging due to the uniqueness of the SGG nature, where the following factors that need to be considered when setting proper threshold that determines confident predictions for unannotated triplets:
Semantic Ambiguity of Predicate Classes: A group of predicate classes in benchmark scene graph datasets are highly related, e.g., "on", "standing on" and "walking on," " which share simi- lar semantics. This implies that the prediction probabilities for predicate classes are not sharpened for a specific predicate class (Fig. (b)), and thus setting a proper threshold for determining a confident prediction is non- -oivial This problem becomes severe due to the fact that unannotated triplets in benchmark SGG datasets are annotated with the bg class, and this accounts for 95.5% of the entire triplets in VG dataset, resulting in SGG models to inevitably produce highly con- fident predictions for the bg class. Hence, pseudo-labeling methods used for image recognition (e.g.. FixMatch (Sohn et al. 2020)) that assign the same threshold (e.g.. 0.95) for all classes is not suitable for the SGG model (as demonstrated by Motif-,-con in Sec. 3.3 Therefore, it is crucial to develop a pseudo-labeling strategy specifically designed for the SGG task, which is capable of autonomously determining the criteria for how reliable its model predictions are for each class, i.c., class- specifc adaptive threshold.
The long-tailed nature of benchmark scene graph datasets poses a challenge in designing self-training framework for SGG models. More precisely, an SGG model trained on long-tailed datasets tends to mainly generate highly confident pseudo- labels from the majority classes, and retraining the model with them would further exacerbate the bias towards the majority classes, causing self-training to fail (as demonstrated by Fig, 2 of Sec. (.3). Therefore, for the SGG task, it is crucial to develop a pseudo-labeling strategy for self- training that assigns accurate pseudo-labes labels to the minority classes, while preventing bias towards the majority classes.
Long-tailed Predicate Distribution:
To cope with the above challenges of pseudo-labeling in SGG, we propose a novel thresholding strategy for the SGG task called class-specific adaptive thresholding with momentum (CATM),
2

---

Published as a conference paper at ICLR 2024
which adaptively adjusts the threshold by considering not only the bias towards majority classes, but also the highly related predicates and the presence of the bg class. Sec. 3.3 we empirically demonstrate that the above factors are indeed mandatory for designing self-training framework for SGG. is worth noting that ST-SGG to any existing SGG models including resampling and reweighting methods such as (Li et al.) 202T and (Sotiris Kotsiantis 2006).
model-agnostic framework that can be applied
Additionally, we particularly focus on building upon meessage-psssngg neural networks (MPNN)- based SGG models, as they have recently shown to be the state-of-the-art (Li al. 2021 Yoon al. 2023). MPNN-based SGG models aim to learn structural relationships among entities based on the message propagation on scene graph, which is shown to improve the representation quality of en- tities and relationships. We discovered that enriching the given scene graph structure, i.e., adding missing edges and removing noisy edges, is beneficial when adopting proposed self-training framework to MPNN-based SGG models, particularly when setting the class-specific thresholds for pseudo-labeling unannotated triplets. Hence, we devise a graph structure learner (GSL) that learns to enrich the given scene graph structure, and incorporate into our MPNN-based ST-SGG frame- work, which allows the messages to be propagated based on the enriched structure learned by the GSL. We show that this results in decreasing the model confidence on the bg class while increasing that on the other classes, both of which are helpful for setting the class-specific threshold. Through extensive experiments on VG and Open Images V6 (OI-V6), we verify that ST-SGG is effective when applied to existing SGG models, particularly enhancing the performance on fine- line-grained pred- icate classes.
To the best of our knowledge, this is the first work that adopts self-training for SGG, which is challenging due to the difficulty in setting the threshold for pseudo-labeling the SGG nature.
We develop novel thresholding technique CATM and the graph structure learner (GSL), which are beneficial for setting thresholds when ST-SGG is applied to any SGG models, e.g., Motif (Zellers al., (2018) and VCTree (Tang al. 2019). and MPNN-based models, e.g., BGNN (Li et al. 2202 and HetSGG (Yoon et al. 2023 respectively.
Through extensive experiments on VG and O1-V6, we verify the effectiveness of ST-SGG compared with the state-of-the-ard debiasing methods, particularly enhancing the performance on fine one-grained predicate classes.
RELATED WORK
In this section, briefly describe the related works and present the focus of our research. Please see Appendix. A for the extended explanation of related works.
Self-nraining In the field of image recognition, self-training is one of the prominent semi- supervised approaches that utilize a large amount of unlabeled samples. The main idea is to as- sign aseudo-labels to unlabeled samples whose confidence is above specified threshold, and use them for model training to improve the generalization performance of the model. Recent works have focused on reliving the confirmation bias (Arazo et al.) (2200) arising from using incorrect pseudo-labels for model training, which in turn undesirably increases the confidence of incorrect predictions. Hence, numerous confidence-based thresholding techniques are proposed to accurately assign pseudo-labels to unlabeled samples (Xie al.) 2020a; [Sohn al. 2020 Xu al., 2021] Zhang et al., 2021: Lai et al. 2022, Guo et 2021 Wei et al. 2021). However, none of the above methods addresses the unique challenges of applying self-training framework in the context of SGG (we will discuss more details of the challenges in Sec. 3.3). In this work, we propose a novel thresholding technique that considers the inherent nature of SGG, where the semantic ambiguity of predicates and the bias caused by long-tailed distribution are present.
Scene Graph Generation. Existing SGG methods point out that SGG models accurately predict general predicates, while rarely making correct predictions for fne-grained predicates. (Desal et al. (2021); Li et al. (2021) and Desai et al. (2021): )) Li et al.] (2021) aim to alleviate the long-tail tailet predicate distribution problem by using resampling and reweighting, respectively. Several debias- ing methods are presented by Tang et al. (2020); Dong et al. (202); Guo et al. 22:2T); Chiou et al. (202T). Most recently, IE-trans (Zhang et al. (202)) proposed replace general predicates with fine- grained predicates and fill in missing annotations with fine-grained predicates to mitigate the prob- lem caused by the long-tailed predicate distribution. However, IE-trans fails to fully exploit the true
3

---

Published as a conference paper at ICLR 2024
labels of unannotated triplets as it uses the initial model parameter of the pre-trained model to gen- erate pseudo-labels. On the other hand, we propose a novel pseudo-labeling technique for SGG to effectively utilize the unannotated triplets, which helps reduce the number of incorrect pseudo-labels by iteratively updating both the iseudo-labels and the SGG model. Further, we extend ST-SGG for
message-passia neural network (MPNN)-based SGG models (Zellers et al. 2018
Tang et al. 2019
Xu et. 2017 Yang et al. 22118) Lin et al. 2020; [ et al. [202] Yoon et al. 2023
Shit al. 2022)
which are considered as sete--oftth-aaae SGG models utilizing the advanced SGG architecture.
3 SELF-TRAINING FRAMEWORK FOR SGG (ST-SGG)
3.1 PRELIMINARIES
Notations. Given an image I, a scene graph G is represented as set of triplets {(s,, Pi, 0;) M where M is the number of triplets in G. A subject s; is associated with a class label St,c C and bounding box position S2,6 E R4, where C' is the set of possible classes for an entity. Likewise, an object o is associated with Oi. E C and ogP IR. A predicate P: denotes the relationship between s; and o, and it is associated with a class label Pi.c E CP, where CP is the set of possible classes for predicates. Note that CP includes the "background" class, which represents there exists no relationship between S; and O, (i.e., Pt. = bg)).
SGG Task. Our goal is to train an SGG model fo : I G, which generates a scene graph G from an image I. Generally, an SGG model first generates entity and relation proposals using a pre- trained object detector such as Faster R-CNN (Ren et al. 2015). Specifically, an entity proposal x" represented by the output of feedforward network that takes bounding box, its visual feature, and the word embedding of the entity class. The predicted probability of the entity class pe IR C*I is estimated by an entity classifier f3 (i.e., p = =f fz(x)), )). Moreover, a relation proposal x between two entities (i.e., subject and object o) is represented by the union box of the visual feature of the two entity proposals. The predicted probability of the predicate class p E R/("l between two entities is estimated by a predicate classifier f% (i.e., pd f (xx+9)). Finally, the SGG model selects the most probable triplets ((s; Ps:
o.)) o.)) MM as a generated scene graph.
3.2 PROBLEM FORMULATION OF ST-SGG
We introduce self elff-trainin framework for SGG (ST-SGG), aiming to exploit a large volume of unannotated triplets, which is formulated under the setting of semi-supervised learning.
Given B batches of scene graphs {GI, Gb, -..; Gn), each batch G., contains set of annotated
triplets G = {(s, Pa, o.) Ip) #f bg). and set of unannotated triplets G {(s., Pt, o:)lp: = bg). Note that only small portion of triplets is annotated, i.e., G > |G For example, only 4.5% of relationships between objects are annotated in the VG dataset (Krishna et al.) 2017).
To exploit the unannotated triplets, we adopt self-training approach for SGG that assigns pseudo- labels to predicates in the unannotated triplets. Given a model prediction p" E IRIC"), we define a confidence & of the model prediction by the maximum value of p", i.e., q max(p") E R, and denote the corresponding predicate class q R(C", which is one-hot vector. We assign q to predicates in the set of unannotated triplets G if the model confidence is greater than a threshold т, and otherwise leave them as bg. Given the labeled triplets in G and pscudo-labeled triplcts in G we retrain the SGG model at every batch step (i.e., iteration). Formally, the loss for training the SGG model under the self eeff-training framework is divided into the following three losses:
L EigbçB
Eicag [-pe log))")) + Beeak 1a [-p?" log(p!)
+3 E E 14.27 --q. log(p.))
Loss for unmolated predicales
Loss for bg class
Loss for labudo-labeled
labeled predicates
where 3 is coefficient for controlling the weight of the pseudo-label loss, and b E R C*I is one- hot vector that represents the bg class. Despite the fact that there may exist true relationships rather than bg in the unannotated triplets, most existing SGG models simply consider all the unannotated triplets belonging the bg class, which means the third loss is merged with the second loss (with 3 = 1) in Equation] I It is important to note that ST-SGG is a model-agnostic framework that can be applied to any existing SGG models.
4

---

Published as a conference paper at ICLR 2024
3.3 CHALLENGES OF SELF- TRAINING FOR SGG
Training an SGG model based on Equation T facilitates the self-training of the model by exploiting unannotated triplets with pseudo-labels. However. unless the threshold T is carefully set, SGG models deteriorate since they are prone to confirmation bias, which is caused by confidently assigning incorrect pseudo-labels to unannotated triplets. In Fig. 2 we empirically investigate the effectiveness of various thresholding techniques by applying them to ST-SGG. Specifically, we first pre-train Motif (Zellers et al.) 2018) on the VG dataset, and subsequently fine-tune it using Equa- tion [hile employing various thresholding techniques including constant thresholding (Sohn et al., 2020) (i.e., 7 = ) fixed class-specific thresholding (i.e., T = Teth), class
fixed class-specific thresholding (Wei et al., [021) that considers the long-tailed distribution (i.e., T = T.",, and class-specific adaptive thresholding (uu et al. 2021 (i.e., T = T Please refer to Appendix B.2 for details on the thresholding techniques.
We have the following
Cum. et of poeudd- libels
three observations: 1)
Menn recatig 100
Rocalli 100
Motif-750m Motif-Te
and Motif-r show lower performance than even Motif-vanilla (without self-training) over all met- rics, failing to benefit from the unannotated triplets through self-training. This implies that thresholding techniques that are widely used in image recognition are not suitable for ST-SGG. We argue that this mainly due to highly related predicate classes, the presence of bg class, and the long-tailed distribution. As these prob- lems co-exist, determining an appropriate threshold becomes difficult when adopting self-training to SGG models. 2) In 2ig.(2(b) we further report the behavior of Motif-r to investigate the reason for its failure. We observe that the training progresses, the number of majority predicate classes being assigned as pseudo-labels increases, while minority predicate classes are not assigned at all. This implies that although we set the different thresholds for the majority and minority predicate classes to address the long-tailed problem, it is non-trivial due to the fact that other issues, such as the semantic ambiguity of predicate classes and the presence of the bg classs, are entangled with the setting of a proper threshold in the nature of SGG. We observe similar behaviors for Motif-~com and Motif-7 3) The performance of Motif-72da decreases at the early stage of training, but its final performance is similar to that of Motif- Vanilla the later stage. We attribute this to the incorrect pseudo-label assignment in the early training steps and the saturation of the adaptive threshold in the later training steps, which prevents the model from further assigning any pseudo-labels. This implies that a naive implementation of the class-specific adaptive thresholding technique fails to fully exploit the unannotated triplets in self-training.
2013
1010
ltrration (0)
Hieration (
(a) Performance over iterations
(b) of of pseudo-lahels of Motif-r
Figure 2: (a) Performance of Motif and its self-trained models with various thresholding techniques. (b) The number of pseudo-labels per predicate class when Motif-r,. is trained on VG.
In summary, it is challenging to determine an appropriate threshold for adopting ST-SGG due to the inherent nature of SGG, mainly incurred by the semantic ambiguity of predicates and the tailed predicate distribution.
4 CLASS-SPECIFIC ADAPTIVE THRESHOLDING WITH MOMENTUM (CATM)
To address the aforementioned challenges of applying self-training to the SGG task, we devise a novel lseudo-labeling technique for SGG, called Class-specific Adaptive Thresholding with Momentum (CATM), which adaptively adjusts the class-specific threshold by considering not only the laiged predicate distribution but also the model's learning state of each predicate class.
4.1 CLASS-SPECIFIC ADAPTIVE TIRESHOLLING
Our approach to adjusting the threshold involves using the model's confidence of the predicate pre- diction on unannotated triplets, as these predictions reflect the per class learning state of the model. We argue that relying the model prediction to determine the threshold has two advantages. First, it addresses the long-tailed nature of predicates that causes significant differences in learning states across predicate classes. Second, although model prediction probabilities for certain predicate
5

---

Published as a conference paper at ICLR 2024
may not be sharpened (or confident) due the semantic ambiguity of predicate classes, it enables us to set an appropriate threshold value based on the model predictions on other instances of the same predicate class. A straightforward approach to estimating the model prediction- thresh- old would be to compute the average of the model's confidence in the validation set, and set as the threshold at every iteration or at regular iteration interval. However, it requires a significant com- putational cost, and the learning state of the model between iteration intervals can not be reflected (Refer to Appendix C.2).
To this end, we employ the exponential moving average (EMA) (Wang et al., 2022) adjust the threshold at each iteration, in which the threshold is either increased or decreased depending on the confidence of the prediction and the threshold at the previous iteration. More formally, denote the threshold of predicate class at iteration I, and use Pe to denote the set of predicates in unannotated triplets in all B batches, i.e., {G] ..... G ..... G that are predicted as belonging to the predicate class c. Our main goal is assign pscudo-labels to unannotated triplets while filtering out incorrect pseudo-labels. Specifically, we increase the threshold T. for given a predicate class if the confidence I of the prediction is greater than the previous threshold - (See the first condition in Equation 2 On the other hand, if the confidence i for all instances of the predicate class e is lower than the previous threshold - we decrease the threshold for the predicate class c, since the current threshold r is likely to be over-estimated (See the second condition in Equation 2 The threshold of the predicate class at iteration 6, i.e., T is updated based on EMA as follows:
use T/ to
(1 X A ---1 + + Xine REEPY [k),
if i E P!! where q: 2 TC (Té increases)
(1 11- X ro-1 Adoe, Eggyy [6:], if ip 0. (r--) remains)
if < r--1 for all i P. decreases)
where Xine and Adee are the momentum coefficients for increasing and decreasing the threshold, respectively. Note that =h is updated only if there exists at least one predicate that is predicted as belonging to the predicate class in any of the batches, and otherwise it is maintained (See the third condition in Equation 2)
As the class-specific threshold reflects the learning state of the model for each predicate class at each iteration, the above EMA approach that adaptively adjusts the class-specific threshold captures the differences in the learning states of the model across the predicate classes. This in turn allows the self-training framework to be applied in the SGG nature. Moreover, the EMA approach is superior to the naive method as it establishes a more stable threshold by considering the model's learning state at every iteration and the accumulated confidence in samples. It is also computationally efficient and does not require computing the confidence on additional datasets, such as a validation set.
4.2
MOMENTUM
However, we observed that the same momentum hyperparameter is used for all predicate classes (i.e., setting the same Xine and dec for all c C²), unannotated triplets mainly pseudo-labeled with majority predicate classes, whereas minority predicate classes receive less attention (See Fig. aa in Appendix E.7). We argue that this incurs confirmation bias and aggravates the problem of long-tailed predicate distribution in SGG. To cope with this challenge, we design the class-specisic momentum, which sets different Ai and X for each e E C² based on the frequency of predicate classes in the training set. The main idea is to increase the threshold of majority predicate classes more rapidly than that of minority predicate classes so that unannotated triplets are pseudo-labeles with minority predicate classes as the training progresses. Conversely, when decreasing the threshold, we want the threshold of majority predicate classes to decrease more slowly than that of minority predicate classes. More formally, let Ni...., Ncrr be the number of instances that belong to the predicate classes C1,...C(er] in the training set, respectively, which is sorted in descending order (i.c., Ni N2 > ... >> Nier)). We set the increasing momentum inn
dee
coefficient and decreasing momentum coefficient as Xine- = ( [N Adec (NeCF 11 et N1 where ine, dec [0, 1] control the increasing and decreasing rate, respectively, and N.
and Nqer +1-e) are the imbalance ratio and reverse imbalance ratio of class For example, assume that N₁ are given three classes with [N1, N2, N3] = [50, 40, 10) where class 1 and 2 are head predicate classes, and class 3 is a tail predicate class. In this case, the imbalance ratio is [1.0, 0.8, 0.2). and the reverse imbalance ratio is [0.2, 0.8, 1.0). Hence, the increasing rates of class 1 and class 3 are
Ni

---

Published as a conference paper at ICLR 2024
X (1.0)a"m and A (0.2)a respectively, which implies that the threshold of class I in- creases more rapidly than that of class 3. On the other hand, the decreasing rates of class and class 3 are A = (0.2)a² and Ag = (1.0)a respectively, which implies that the threshold of class I decreases more slowly than that of class 3. Please refer to Fig. 10 of Appendix E.8 regarding the change of A and Adec based on a te and dec. As a result of setting the class-specific momentum coefficients, a small number of pseudo-labels are being assigned to head predicate classes, while tail predicate classes are assigned more actively (See Fig. ((b) of Appendix[E.
which relieves the long-tad long-tailed the problem of predicates. We believe that pseudo-labeling the unannotated triplets with more tail predicate classes than head predicate classes is acceptable since the original training set already contains a large volume of instances that belong to head predicate classes.
4.3 GRAPH STRUCTURE LEARNER FOR CONFIDENT PSEUDO-LABELS
We discovered that enriching the given scene graph structure, i.e.,
BUNN USL BONN o/ GSL
adding missing edges and removing noisy edges, is beneficial when adopt- ing our self-training framework to MPNN- SGG models. In Fig. we report the confidence of predicate classes after training BGNN LLi (et al. (02I). a SOTA MPNN-based SGG method, on the fully connected scene graph (i.e., BGNN w/o GSL)
Confidence
0.00
0.04
Figure 3: Impact of applying GSL on the model confidence of BGNN (L et al. 2021).
and on scene graph whose structure enriched by a graph structure learner (i.e., BGNN w/ GSL). graph structure learner (GSL) is method to learn relevant and irrelevant relations between entities in a scene graph, allowing
MPNN-based methods to propagate messages only through relevant relations. We observe that BGNN with GSL generates relatively low confident predictions on the "bg" class, allowing the model make more confident predictions the remaining predicate classes, which is particularly beneficial when setting the class-specific thresholds for pseudo-labeling unannotated triplets. To this end, devise GSL that learns to enrich the given scene graph structure, and incorporate it into our MPNN-based ST-SGG framework. Please refer to Appendix D for more details. Note that consider only the relevant relations predicted by GSL as candidates for pseudo-labeling
5 EXPERIMENT
We compare ST-SGG with state-of tiie--oftthhrrttte methods that alleviate the tailed problem on com- monly used benchmark datasets, VG and O6-V6. We report only the result on VG due to the page limit. Please refer to the result on OI-V6 in Appendix F.3, More details of the experimental setups described in Appendix F and
Note that ST-SGG involves CATM by default.
5.1 COMPARISON WITH BASELINES ON VISUAL GENOME
In Table I we apply ST-SGG to widely-used SGG models, such Motif (Zellers et al. 2018) and VCTree (Tang et al. 2019), and compare them with baselines. Based on the result, we have the following observations: 1) ST-SGG exhibits model-agnostic adaptability to SGG models. Motif+ST-SGG and VCTree+ST-SGG improve their performance terms of mR @K and F@K, implying that ST-SGG greatly increases the performance on tail predicates while retaining that of head predicates. ST-SGG that employs debiasing methods is competitive with ofe-state-of-art SGG models. Specifically, we employ resampling and I-Trans (Zhang et al.] 2022) to Motif un- and
der ST-SGG. In terms of F@K, Sotif-RRsamp-+ST-SSG the performance of DT22ACBB and PCPL. in SGCIs and SGDeet despite the simple architecture of Motif. This implies that utilizing unannotated triplets with simple debiasing methods is powerful without advancing the architecture of SGG models. 3) Compared to a previous pseudo-labeling method, ie., IE-Trans, ST-SGG+I-Trans achieves better performance in terms of mR@) K and F@K. This demonstrates that ST-SGG assigns accurate pseudo-labels to unannotated triplets, and relieves the long-tailed problem. We attribute this to the fact that pseudo-labeling in an iterative manner is more accurate than pseudo-labeling in a one-shot manner.
outperform
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance (%) comparison of ST-SGG with sfate-of-the art SGG models on three tasks.
Resam. denotes the re-sampling (Li et al. 2021) method. The best performance is shown in bold.
Method
BC0S0 1001
MIE 100
PradCis
WIE 1DU
SOCk
78:307116
SGiet
FEST 1I
MEEE- Degg soalaae DICT
16.2/174 17.6
150-36.
PCPL
Yan
5088/526 6
35:2/37.8
186-190
23.2225.2
9.5/1177
115/144
KERN
63/8/576 6
17.34 19.2
27.9 22.9
36.7/37.4
94//1..0
15.0155
22.1/298
64/73
10.4111.
GBNeT
066//182 2
22.1/24.1
33.2/35.5
37.3/38. 38.0
12:7/15.
18951995
26.3/20.
7../55
11221133
PB-Ne
619/572
31.5/35.8
42.4/45.0
39.4/40.7
17.5/189
34.5/258
30.7/35.2
124/1455
1777-205
Meiir (7rlem
1015]
6.1/67.1
17 1922
75:01770.9
96.97%6.
90296.
1657183
31.9756.4
64/36
10 1177512
+ST-500
614/555
27:47300 26.
15:1755. 15.3
19.92 17.8
12
18:2/1911
+Knom
07.1/54
26.17
16.8
1377
19.0/11.
14.8
10-9/125
15.6118
21 Reyd
06.2751.4
25.52
27.7
15.17
17.8/199
16.9/20.3
5./955
11.0
Modko Aagr---goooee
(DLFE
INICE
52.5/543 /5.52
25.1/592
26.91336
29:91 32.5
33.0/337.6 37.55
38.5 441.3
33.17 34.00
33.3/33.1
15.5-17. 17.9
15.21 15.0
20.5.21.5
22.1 1235
25.4-29.
27.81313
12.2/
11.2/138
17.015 19.5
16.8-18.5
Traes PAlTE
54.7560 7677/77
35.6
16:87179
21.2/32.4
22.8/31.3
50.37 52.8
32.5735
42.5
31,27322
1800:19. 19.3
32.8/26.1
25.7739.8
12.9/188
17.3/200
Ve ar N019
65.76722
12:22 18.6
22.577704
30.1 18.8
9.57 10.3
2/3/8.5
+Kooom
[J 100
61377855
23.3. 29.3
12.77 46.5
14.77 16.5
11.77184 18.4
19.9720Y
29.7715.9
102/111
15.--133
+ittor
5407570
12.27 56.6
41.3745.
15.2/35.
1642 18.5
22222258
12.3/3468
ITDE
47.2/116
25.4 25.7
33.9
25.4/279
13.21 14.0
16.5/186
19./2332
9.3/11.1
12.6 15.0
51.75535
27
20.0
22.7/363
11:9/138
15.511
INICE
55.0
30.7135.0
37.81
195/3113
11E Trang
536/550
30.3133.0
39.6 41.9
32.91 33.8
16.57 196
23.6
25.42205
11.57 34.0
158-115
Trass Phin
540/55.
30.27
39.7/ 41.6
37.2/38.3 38.5
1900 20.6
26.5
25
11 13.7
18.7
32.5/563
32.9/38.6
40.3/43.
36.3/37.3
2000
20/7/46.
12 15.1
157/1K5
Table 2: Performance (%) comparison of ST-SGG with state-of-the art MPNN-based SGG models on three tasks. Resam, denotes the re-sampling (Li et al. 202T) method.
ProiCis
SGCS
SCINE
BGNN
29.2/31.7
38.3741.5
36.97 36.1
14.57160
20.973 32.5
38,9734.7
114:13.3
16.5/19.2
BRPSN Teae 30od0
-ST-S13G-GB
68.9/514
/362
46.3/43.5
53.5/34.7
19.0/194
35.43345
36.5/314
19.4/2177
Zhea stal Aore
S457 56.7
2527 12.4
18.44412
23.2/34.1
16.7710
21:9125.1
24:2/2008
11:24 13.1
15.51 1S..
(315
31231403 1-27
33.31345
15.71169
21:2132:1
24:2726.1
10.22
14.81
ct 2013
SUD/ 0LL
3007 32.2
3557415
31.6338 38.5
15.71137
22:2/363
38.2734.5
11:5:135
16.7 19.6
-51-800-658
134
46.1/428
35.51
16.2/191
26.1/25.1
76.9/319
12,9/
17.4
2hane eLal 1017
11:17 14.5
339/541 51.
16.71 16
22.03 22.07-25.
13.4/374
1/ 14.9
15..155 13.4
15-Tuza
52.15541
3038
36.7 14.9
33.0133.19
16-3117-3
21.87
23:9/256
10.77
14.8
Moreover, we investigate the re- sult for each predicate. Fig. 4 shows that ST-SGG greatly im-
[4]
Thi
Racala1100 LiGfiee
proves the performance for the tail
predicates while retaining that for the head predicates by pseudo- labeling the unannotated triplet a tail predicates. It is worth noting that predicting fine-gre mme-grained predi- cates (i.e., tail predicates) is more important than predicting general predicates tail predicates are informative in depicting scene. In this regard, S ST-SGG is an effective framework that generates informative scene graphs by utilizing unannotated triplets.
Figure 4: (a) Performance comparison per class. The black line indicates the number of pseudo-labeled instances. (b) Perfor- mance comparison head, body, and tail predicate classes.
5.2 COMPARISON WITH MPNN -BSSED MODELS ON VISUAL GENOME
Table 2 shows the result of the state-of-art MPNN-based SGG models. Herein, apply the graph structure learner (GSL) to existing MPNN-based models including BGNN (Li et al. 2021 and HetSGG (Yon et al. 2023). We have the following observations: 1) The mR@ K and F@K K of BGNN+ST-SGG, and HeISGG+ST-SGG
are increased compared to those of BGNN and HeISGG respectively, which implies that ST-SGG effectively utilizes unannotated triplets when applied to MPNN- SGG models. 2) When MPNN-based models employ ST-SGG +GSL, they achieve state-of-arr performance among SGG models, outperforming the models that solely use ST-SGG. This implies that GSL identifies relevant relationships, and the relations to which pseudo-labels are to be assigned.
5.3 ABLATION STUDY ON MODEL. COMPONENTS OF ST-SGG
In Table 3
we ablate the component of ST-SGG to analyze the effect of each component. We select Motif and BGNN trained with resampling (Li et al. 202T) as the backbone SGG model, and train the following models for SGCIs task. 1) ST-SGG w/o EMA: we remove the EMA of adaptive
threshold, and apply the fixed threshold zllk used in Sec. 3.3
ST-SGG w/o X dde, we remove
8

---

Published as a conference paper at ICLR 2024
the class-specific momentum by setting A and Agee to 0.5. When the Motif backbone is used, we observed that the performance of ST- SGG w/o EMA severely decreases in terms of R@K and mR @ K compared to Vanilla Motif since the fixed threshold cannot properly as- sign correct pseudo-labels. This implies that adaptively updating the threshold through EMA is important for ST-SGG. Beside, compared with ST-SGG, ST-SGG w/o X Xdee shows
Table 3: Ablation study of ST-SGG.
Back-
Medel
SGCh
Vanilla Musi Zellen ol 2016
36.13370
137/14.7
199/31.
5T:5GG wes FXEX
256/374
60/73
9.5-126
ST.SGG we A A ST-S0G
33.03446
135/143
192//292
35477449
169/180
22./2388
Varilla BGVN el el. [111] we
369/3011
14./160
21M/22.5
3011/31.2
ST-S0G
ST-SOG
3237334
180/1900
251/738.6
ST-SGGIGLL
33.57347
18.0/194
13.4/449
inferior performance over all metrics. This degradation is due to the fact that removing class-specifie momentum incurs bias to majority classes as shown in Fig. of Appendix [E.7 This implies that adjusting the threshold in a class-specific manner, which rapidly/slowly adjusts the threshold for head/tail predicates is important to alleviating the long-tailed problem. When the BGNN backbone is used, similar results are observed. Additionally, we confirmed that ST-SGG+GSL generrlly out- performs ST-SGG in terms of @ K. mR @ K, and F@K on the BGNN backbone, implying that enriching the scene graph structure through GSL helps identify the relations to which laeudo-labels are be assigned. We further analyze the effect of a iue and dec Appendix E.8. and Appendix E.9
1t-
waliiigga
parkidd on
molorrycle
street
bear
SNOW
parkeed on
Matll STSC
sodd 010)
nolovyyle
stzeet
beor
SNOW
1000
yet 200 Heretion (c)
Sebrayck 2004
bor 7 SEEN
wollinng
(e)
(b) Case 1: True B Background
(c) Case 2: Missing A Annotation
Figure 5: (a) Adaptive threshold values over iterations, and (b) examples of pseudo-labels of ST- SGG and IE-Trans assigned to the different cases. Conf. denotes the confidence, i.e., q.
5.4 QUALITATIVE ANALYSIS ON CATM
We conducted the qualitative analysis to demonstrate the effectiveness of our proposed CATM in assigning accurate pseudo-labels. In Fig 5(a), we observed that the range of the class-specffic threshold values defined by CATM is diverse. Interestingly, we found that the threshold is not re- lated to the frequency of classes. This shows the difficulty of determining appropriate class-specific thresholds for pseudo-labeling, and we argue that this is the main reason of the failure of the thresh- olding techniques used in image recognition in the SGG task. Moreover, let consider a triplet (motorcycle," ?, street) shown in Fig. 5b)) which is case of true background, i.e., it should be labeled with the bg class because the motorcycle in the blue box is far from the street in the green box. We observe that while ST-SGG predicts it correctly as bg. IE-Trans incorrectly predicts it as parked on. This is the due to the design of TE-Trans that tries to assign pseudo-labels to as many unannotated triplets as possible (Zhang et al. 2022). Next, let us consider triplet (bear, ?, snow) shown in Fig. 5ic), which is a case of missing annotation, i.e., it should be labeled with a class other than bg because the bear in the blue box is close to the snow in the green box. We observe that TE-Trans initially predicts it as on. and then re-assigns it to bg, due to the behavior of IE- Trans that only pseudo-labels predicates belonging to the minority classes; as on belongs the majority class, it is re-assigned to bg. On the other hand, ST-SGG predicts it correctly as walking in, which indicates that ST-SGG is framework that effectively utilizes unannotated triplets by providing accurate pseudo-labels.
6 CONCLUSION
Although self-training has shown advancements in computer vision applications like image recognition, its application in SGG has received limited attention. Simply adapting existing self-training methods for SGG ineffective due to challenges posed by the semantic ambiguity and the long-tailed distribution of predicates. Consequently, we propose novel self-training framework specifically designed for SGG, called ST-SGG. This framework effectively harnesses unannotated triplets even in the SGG nature, resulting in substantial improvements in fine- tne-grained predicates. Our work offers new perspective on training SGG models by leveraging existing benchmark scene graph datasets, which often contain missing annotations. For the limitation of this study, please refer Appendix G
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-00004, Develop- ment of Previsional Intelligence based Long-term Visual Memory Network, and No.2022-0- 00077).
ETHICS STATEMENT
In compliance with the ICLR Code of Ethics, to the best of our knowledge, we have not encountered any ethical issues throughout this paper. Moreover, all the datasets and pre-trained models utilized in our experiments are openly accessible to the public.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of experiment results, we describe the details of datasets, experimental setting, and implementation details in Appendix F and [ Furthermore, we provide source code in https::/gthuu..cmm//aaaaaol models.
along with accessible datasets and our pre-trained
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN, pp. 1-8, 2020. doi: 10. 1199UCNNN46005 202.00 200.00...22202.77.7 09003444.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: arXiv:/308.3 300.3432, 2013.
Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embeddee routing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6163 6171, 2019.
Meng-Jine Jiun Chiou, Henghui Ding, Hanshu Yan, Changhu Wang, Roger Zimmermann, and Jiashi Feng. Recovering the unbiased scene graphs from the biased ones. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1581-1590, 2021.
Alakh Desai, Tz-Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations:
The devil is in the tails. In ICCV, pp. 15404-15413, October 2021.
Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In CVPR, 2022.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalancee semi semi-supervised learning with adaptive threshold- ing. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8082-8094. PMLR, 17-23 Jul 2022. URL flctps:/prrocestt wgw.oocceeinn........... mlr mlr.press/v. 62/ gwwoee.e.....oooooo html
Yuyu Guo. Lianli Gao, Xuanhan Wang, Yuxuan Hu, Xing Xu, Xu Lu, Heng Tao Shen, and Jingkuan Song. From general to specific: Informative scene graph generation via balance adjustment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16383- 16392, October 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax arXiv preprint arXiv:1611.011 11111101144 144, 2016.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International jiternationalouraaal journal of computer vision, 123:32-73, 2017.
10

---

Published as a conference paper at ICLR 2024
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset. Sha- hab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. In- ternational Journal of Computer Vision, 128(7):195-1198 2020.
Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refine- ment pseudo labels for multiclass-imbalancee semi-supervised learning. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pp. 4090-4099, 2022. doi: 10. 119/0C/PR553377...2 2022.400072222
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Lin Li. Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18869-18878, 2022.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. CVPR, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2014. URL http: / aawiivvvi..../....a.aaaaaaaavvvv org. abs babs/// /1405. 0312 cite
arxiv:1405.00 100000311Commmen 1) updated annotation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2983 2017.
Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. CVPR. pp.
2020.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. European conference on computer vision, pp. 852-869, 2016.
Xinyu Lyu, Lianli Gao, Yuyu Guo, Zhou Zhao, Hao Huang, Heng Tao Shen, and Jingkuan Song.
Fine-grained predicates learning for scene graph generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19467-19475, June 2022.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei Li, Ji- azhen Pan, Sahand Sharifzadeh, Georgios Kaissis, Volker Tresp, and Bjoemn Menze. Relation- former: unified framework for image-to-graph generation. In ECCV 2022.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020)., 2020.
Panayiotis Pintelas Sotiris Kotsiantis, Dimitris Kanellopoulos. Handling imbalanced datasets: A review. pp. 25-36, 2006.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on com- puler vision and pattern recognition, pp. 661 96288, 2019.

---

Published as a conference paper at ICLR 2024
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. CVPR, pp. 3716-3725, 2020.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. Freematch: Self delf-adeetive thresholding for supervipe learning. arXiv preprint arXiv:2205.07244 2022.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing selinig framework for imbalanced semi-supervissed learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10857- 10866, June 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6256 6268. Curran Associates, Inc., 2020a. URL httppsss/pprrccoeddgg....cot nww.ipeurppp.. wctpsscccceeeeee wwtppcpppppppp..c.c
neur
files/
Qizhe Xie, Minh- Thang Luong. Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 10687-10698, 2020b.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. CVPR, 2017.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In Proceedings of the 38 th International Conference on Machine Learning, 2021.
Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian- Sheng Hua. Pepl: Predicate-correlation perception learning for unbiased scene graph generation. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 265-273, 2020.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. ECCV, 128(7))670-685, 2018.
Kanghoon Yoon, Kibum Kim, Jinyoung Moon, and Chanyoung Park. Unbiased heterogencous scene graph generation with relation-aue -lwar aware message passing neural network. 2023.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In Computer Vsion-ECCC220 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pp. 606-623. Springer, 2020.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5831-5840, 2018.
Ao Zhang, Yuan Yao. Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Fine-grained scene graph generation with data transfer. In ECCV, 2022.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, 2021. URL https:/7 openreview wet. tttp:/dddddd=33dvv/e 3=
H. Zhang, F. Chen, Z. Shen, Q. Hao, C. Zhu, and M. Savvides. Solving mnnsiing-annotttion object detection with background recalibration loss. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. doi: 10. 109/CASSP440776.22020 9053738.
12

---

Published as a conference paper at ICLR 2024
Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11535-1 1543, 2019.
Chaofan Zheng, Xinyu Lyu, Lianli Gao. Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22783-22792, June 2023,
13

---

Published as a conference paper at ICLR 2024
Supplementary Material
Adaptive Self-training Framework for Fine-grained Scene Graph Generation
Related Work
15
Regarding Challenges of Applying Self-training for SGG
16
B.
Thresholding Techniques Image Recognition Task
16
B.2
Details of Thresholding Techniques in Sec. 3.3
16
3
Detailed Discussion on Result
17
ST-SGG
18
Algorithm
18
Naive Model Prediction-basee Adaptive Thresholding
18
Details on Graph Structure Learner (GSL)
19
D.I Graph Structure Learner
19
D.2 Training
20
Experiment on Visual Genome
20
Dataset
20
Experimental Setting
20
Implementation Details
E,4
Result with Reweighting Methods
21
Zero-shot Performance Analysis
21
E,6
Effect of Initial Value for Class-specific Threshold
22
Effect of Class-specific Momentum
22
E 8
Effect of anc and dec
23
E,9
Effect of 3
23
10
Complexity Analysis
24
Experiment on Open Image V6
24
Dataset
24
F2
Experimental Setting
24
F3
Comparison with the state-or of art models
24
Limitation & Future Work
25
14

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Teainnnri Self-training is one of the prominent semi-supervised approaches that utilize large amount of unlabeled samples, and it has been widely studied in the field of image recognition. The main idea is to assign pseudo-labels to unlabeled samples whose confidence is above a specified threshold and use them for model training to improve the generalization performance of the model. Recent works have focused on reliving the confirmation bias (Arazo et al. (020) arising from using incorrect pseudo-labels for model training, which in turn undesirably increases the confidence of incorrect predictions. Hence, numerous confidence-based thresholding techniques are proposed to accurately assign pseudo-labels to unlabeled samples. Specifically, UDA (Xie al., 2020a) and Fixmatch (Sohn et al.) 2020) employ a pre-defined constant threshold during training. However, these approaches fail to consider the learning state of model, aggravating the confirmation bias as the model predicts with higher confidence at the later part of the training step. To resolve the issue, Dash (Xu et al., (021) presents an adaptive threshold that is decreased as the iteration goes on, and Flexmatch (Zhang et al. 2021) employs a curriculum learning approach to reflect the learning state of the model into the pseudo-labeling process. In addition, Lai et al.] (2022); Guo et al. (2021); D); (Wei et al. (202T) assume the situation with an extremely imbalanced class distribution which the model is prone to a severe confirmation bias. They present thresholding techniques, which change the threshold according to the number of assigned aseudo-labele per class, to reduce the bias towards majority classes. However, none of the above methods addresses the unique challenges of applying self- self-training framework in the context of SGG. In this work, we propose a novel thresholding technique that considers the inherent nature of SGG, where the semantic ambiguity of predicates and the bias caused by long-tailed distribution are present.
Scene Graph Generation. Existing SGG methods point out that SGG models accurately predict general predicates, while rarely making correct prediction for fine-grained predicates. There are two mainstream SGG approaches for addressing the biased prediction issue. One line of research aims to alleviate the long-tailed predicate distribution problem. Specifically, Desai et al. (2021): Li et al.. (2021) develop triplet-level resampling methods to balance the distribution of general and fine-grame grained predicates. Yan et al.] (2020;; Lyu et al.] (2022) propose reweighting losses that consider interclass dependencies
predicates to improve the performance on fine-graines predicates. Additionally, Tang et al.] (2020) introduces a causal inference framework, and Dong al.] (2022) uses the transter learning upproaches to mitigate the bias towards general predicates. Guo et al. (202T); Chiou et al. (202T) present post-processing methods to address the bias. Most recently, TE-trans (Zhang al. 2022) proposed replace general predicates with fine-grained predicates and fill in missing annotations with fine-grained predicates to mitigate the problem caused by the long-tailed predicate distribution. However, IE-trans fails fully exploit the true labels of unannotated triplets as it uses the initial model parameter of the pretrained model to generate pseudo-labels. On the other hand, we propose novel pseudo-labeling technique for SGG to effectively utilize the unannotated triplets, which helps reduce the number of incorrect pseudo-labels by iteratively updating both the pseudo-labels and the SGG model.
Another line of research focuses on designing advanced SGG architectures to improve the gener- alization performance of SGG models. The main idea is to refine the representations of entity and relation proposals obtained from the object detector with message-pasee
neural networks (MPNN).
Specifically, some works enhance the representations by employing sequential models such as RNN
and STM to capture the visual context (Zellers et al., 2018 Tang et al. 2019 Xu al. 2017).
Rather sequentially capturing the context from neighboring entities, recent works propagate mes- sages from the neighbors using graph neural networks whose representation includes structured context between neighbors (Yang et al.] 2018 Lin et al. 2020 Li et al. 22021 Yoon et al.) 2023).
Relformer (Shit et al. 2022) develops one-stage SGG framework using relational transtormer to represent local and global semantic in image as a graph. In this work, we propose a model-agnostic self-training framework for SGG to facilitate the use of many unannotated triplets in the scene graph. We further extend ST-SGG for MPNN-based SGG models based on graph structure learning (GSL), which accurately assigns pseudo-labels based on the presence of relationships.
Difference from IE-Trans. TE-Trans (Zhang et al. (022) is the most recent work that addresses the inherent issues in scene graph datasets aiming at alleviating the long-tailed problem. Specifically, IE-Trans performs an internal transfer that replaces general predicates with fine-grained predicates within the annotated triplets, and an external transfer that fills in unannotated triplets, i.e., back- ground classes, with fine-grained predicates. More specifically, IR-Trans utilizes pretrained SGG
15

---

Published as a conference paper at ICLR 2024
model, such as Motif (Zellers al., 2018), VCTree (Tang al. 2019). and BGNN (Li et al. 2021 to generate the confidence for all possible predicates, and sorts them in descending order. Subse- quently, it conducts the internal transfer the top 70% of annotated triplets and external transfer on the top 100% of unannotated triplets (i.e., all unannotated triplets). It is worth noting that Trans assigns pseudo-labels based on the initial parameter of the pre-trained model, and trains a new model from scratch.
On the other hand, ST-SGG focuses only on the effective utilization of unannotated triplets to alleviate the long-tailed problem. That is, we only focus on the "external transfer." In order to accomplish this, ST-SGG learns class-specific thresholds to determine which instances should be useudo-labeles at every batch, and retraining the model's parameters using the dataset enriched with pseudo-labeled predicates.
Herein, we clarify the difference between ST-SGG and the external transfer of IE- IE-Trans, they both serve a similar purpose, which is to make use of unannotated triplets.
1. While ST-SGG sets different threshold for each predicate class, IE-Trans assigns pseudo- labels to predicates based on their confidence values without considering class-specific thresh- olds. This approach overlooks the diverse distribution of confidence across classes, leading to imprecise pseudo-labeling. For example, Fig. (
(a), we observed that the top- 1p--1% % confidence of carrying is around 0.4, which is relatively high among confidences of overall predicate classes, while that of painted below 0.05, which is relatively low. This implies that when performing external transfer to unannotated triplets sorted in descending order by the confidence, IE- Traass would rarely pseudo-label triplet with painted on, its confidence is generally low. We conjecture that, to avoid this problem, Tr-nsa performed external transfer on the top 100% of unannotated triplets (i.e., all unannotated triplets) if there is at least slight overlap between two bounding boxes. In fact, Figure 7.(b) of the IE-Trans paper Zhang et al. (2022) shows that IE-Trans is only successful when the external transfer is performed on the 100% of unannotated triplets, while the performance degrades significantly otherwise. To make it worse, assigning pseudo-labes to all unannotated triplets leads to the generation of a significant number of incorrect aseudo-labels as the true background relations exist in the scene graphs. On the other hand, ST-SGG addresses these problems by proposing CATM that considers the different confidence distributions across the classes and the presence of true background.
2. ST-SGG is computationally more efficient compared with IE-Trans. ST-SGG only needs to check whether the confidence produced at each batch is greater than the class-specific thresholds to compute the loss for pseudo-labeled triplets. On the other hand, IE- TE-Trans needs to perform an additional data transfer stage, which produces confidence of all possible relationships in the dataset to perform the internal and external transfers. Due to the large number of triplets in the scene graph dataset, this additional stage requires a lot of computational time during the model's inference. This will be further discussed in detail in Appendix E.
B REGARDING CHALLENGES OF APPLYING SELF-TRAINING FOR SGG
B. THRESHOLDING TECHNIQUES IN IMAGE RECOGNITION TASK
As self-training has been widely explored in the field of image recognition, straightforward ap- proach to designing ST-SGG tion. we evaluate the existing thresholding techniques (Sohn et al. 2020 Wei et al. 2021 Xu et al.) 2021 when they are applied to ST-SGG.
adopt existing thresholding techniques. In the following subsec-
B.2 DETAILS THRESHOLDING TECHNIQUES SEC. 3.3
1. Constant thresholding (Sohn et al. 2020 Xie et al. 2020a): T con is a pre-defined constant threshold used in Fixmatch (Sohn al.] 2020). Due the fact that the SGG model predicts the bg class with significantly higher confidence than the remaining classes, setting global threshold such 000 0.95 would fail. Thus, set the threshold as the top- % confidence among the confidences for all classes computed on the validation set.
2. Fixed class-specific thresholding (Sohn al. 2020)= T - is a variant of constant threshold- ing that defines the threshold in a class-specific manner. The threshold for each predicate class is set to the top- 1 % confidence per predicate computed the validation set as in Fig. ((a).
16

---

Published as a conference paper at ICLR 2024
Motif-z-'
Motif-r
top- 1% confiderce
fregg
(a)
(b)
Motifr
Motif-r
Motif-rade
Threhbold ower' theaat on
#oo paeurlo-lbbl
12 8
atstoddata unde
#nd tre3ato
has
weating
sugadity
Gllnddy
20 TKl
LTM 17A 203
200
758 yoi 155 US8 1999 261 Iteration (1) (d)
28 5t 15s LoW 1253 150 1790 2000 Iteration (1)
Iberation (1) (c)
(e)
Figure 6: (b). Static thresholds -he and T (c) The number of pseudo-labels for each predi- cate, where Tht and Taul thresholds are used for ST-SGG, respectively. (c) Adaptive threshold over iterations.
Class wreguency-weighted Fixed class-specific thresholding (Wei et al. 2022 ): T 7 is
designed to consider the nature of long-tailed predicate distribution in SGG to prevent the model from assigning pseudo-labels mainly to majority classes. We set T as the linear combination of Tell and the normalized frequency of predicate class as in Fig. 6(b), which imposes a penalty for assigning pseudo-labels to majority classes.
4. Class-specific adaptive thresholding (Xu et al. (2021); Guo & Li 2022 ): The is a thresh- old that adaptively changes according to the learning state of the model. Following Dash (Xu eet al. 2021). ). we gradually increase the class-specific threshold starting from 7 to reduce the number of incorrect pseudo-labels the training proceeds (Please refer to G()).
B.3 DETAILED DISCUSSION ON RESULT
Here, delve deeper into the results of Sec 3.3 and explore the reasons why each thresholding technique failed to improve the performance of SGG during self-training.
Despite Motif-r containing reliable confidence levels for each class (i.c., top- % confidence per predicate class), we observed that the model only assigns pseudo-labels to predicates belonging to the majority classes (Please see c)), leading an increased bias towards the majority classes. This indicates that the long-tailed problems must be addressed to design a self self-training framework for SGG. To address this, T. was introduced to penalize assigning pseudo-labele with majority pred- icate classes, but it resulted in a significant performance drop in terms of Recall @ 100 compared to Motif-rehs in Fig. a). This is because, as shown in Fig. Motif-r." cannot produce pseudo-labels with majority and minority predicate classes and rather generates pseudo-label, with intermediate classes, leading to the bias towards intermediate classes. is natural that when the model is biased towards the intermediate classes, the Recall @ 100 for majority classes decreases since it degrades the generalization performance for majority classes. This result implies that simply penalizing the majority classes is not solution for providing appropriate thresholds for each predicate class.
On the other hand, Tash adjusts the threshold over iterations to control the number of pseudo-labels. However, as depicted in fig. Motif-7adh exhibits a bias towards the majority classes in the early iterations and later on, due to the saturated thresholds. it ends up not assigning any pseudo-labele resulting in performance similar that of Motif without self-training i.e., Motif-Vanilla. From the experiment, we validated that a new thresholding technique is required for ST-SGG, considering the loigetailed problem and the nature of SGG.
17

---

Published as a conference paper at ICLR 2024
Algorithm ST-SGG
Require: Pretrained SGG model fe. increasing/deercasingg rate a inc adee, coefficient of loss for pseudo-labeled predicates B. maximum iteration T, and sorted number of instances that belongs to each predicate class N1, N2,..., Nicel-
Compute class-specific momentum: Xime ja Adee MMe (iopi4i-c ja Ve E CP Ni
2: for = 1 to T do
3:
Given a batch of images I,,... LB.
4: G G1 G 1... G and G
[G] G y Gb
5:
for all (s,, Ps, O,) G do
6: Compute d: = max (x. and assign pseudo-label q c.
7:
end for
8:
if use GSL then
9:
Compute $8.0
using GSL.
10:
L B 1-p. log (og(p,))
B G 2
Pi log(p?)
bg
Loss for annotated predicates
Loss for bg class
11:
+ E G (á 14,274 ")
[-q log(p
Loss for pseudo- laheled predicates with GSI.
12:
else
13:
L = BieGA 1-p, log(p
+ EEGy 1
P log (pg)
Loss for anmotated predicales
Loss for bg class
14:
+ E G 27 I-q: log (p.)
Loss for pseuda labeled predicates
15:
end if
Update class-specific adaptive threshold
Refer Algorithm 2
17:
Train the SGG model fe w/ gradient descent:
18:
9.+1 + 0₂ - d0,
de
19: end for
C ST-SGG
C.I ALGORITHM
For better understanding of ST-SGG, we provide the details of the procedure Algorithm I and Algorithm In Algorithm 1, process batch of images by first segregating the batch into notated scene graph G and an unannotated scene graph G (Line 4). During each batch iteration, evaluate confidences and assign pseudo-labels all the unannotated predicates in G (Line 6). Finally, we calculate three separate losses for the annotated predicates, background predicates, and pseudo-labeled predicates (Line 13). Specifically, for annotated predicates, we employ the standard cross-entropy loss. For unannotated predicates, if their confidence is above the current threshold 7. we calculate the cross-entropy using the assigned pseudo-label as supervision. If the confidence is below the threshold, the predicate is classified within the background relation class. Notably, update the thresholds dynamically after each batch, as explained in Algorithm 2.
C.2 NAIVE MODEL PREDICTION-BASED ADAPTIVE TURESHOLLIIN
In Sec we mentioned that straightforward approach to estimating the model prediction-bart based threshold would to compute the average of the model's confidence the validation set, and set as the threshold at every iteration or at a regular iteration interval. Herein, we implement the naive model reddiciion-based thresholding by setting the class-specific threshold as the averaged confi- dence computed on the validation set. We computed the confidence with 100 regular intervals since it is impractical to the class-specifie threshold from the validation the expensive computation cost. In Fig we observe that the performance of the naive thresholding method significantly degrades between and 100 iterations. We attribute this degradation to the
every iteration due to
18

---

Published as a conference paper at ICLR 2024
Algorithm 2 CATM
Require: Set confidence q for G increasing/decreesinng rate a inc, dec, and sorted number of
instances that belongs to each predicate class Ni. N2,NN...... Neerj. for in C* do
2:
if 3i E 'D where di 2 76-1 then
3:
T/ = (1- X)mo) To ---f + Aime
EpEPY
[6:]
(T increases)
4:
else if 0: < _ for all E Pu then
5:
T/= (mm) Te --1 + Adec Eiepy [a:]
(Tc decreases)
6:
else if p = 9 then
7:
T/2 T (T--1 remains)
8:
end if
9:
end for
return [T]. Tenp]
Appess MesiNa
R.1100
30-
Figt00
C 12 0
20
15
Figure 7: Performance over iterations when naive model prediction- thresholding is applied on SGCIs task.
fact that in SGG it is crucial to reflect the learning state of the model at each iteration, but the naive approach failed to do so. Therefore, the naive approach is inefficient and ineffective compared to the EMA-based approach in terms of reflecting the learning state as well as time complexity.
D DETAILS ON GRAPH STRUCTURE LEARNER (GSL)
D.1 GRAPH STRUCTURE LEARNER
we formally describe the GSL which adopted the MPNN-based SGG model. The main idea of GSL to enrich the structure of the scene graph by discovering relevant neighbors or removing irrelevant neighbors.
For more details of the graph structure learner mentioned in Section 4.3
Based on the subject and object representations (i.e., x8, xº), GSL uses MLP to generate scalar 88.0 E [0, 1] representing the link probability the relation between a subject s and an object o follows: 0.0
(MLP((x x where [:] is the concatenation operation. A straight- forward approach for improving the representation of the relations would be to assign a weight to message based on the link probability 5* their importance. However, as s is soft value that lies between 0 and I, this approach fails to completely prevent irrelevent messages from being propagated among entities, i.e., this approach essentially treats a graph as a fully connected graph where the edge weights are between and 1. To this end, we propose to sample relevant relations from a scene graph based on 800 so that messages are only allowed be propagated through the sampled relations. However, as sampling is discrete process that is not differentiable, which hinders the end-to-end training of our model, we apply the Gumbel-softmax reparameterization trick (Maddison et al. 2016 Jang al.) 2016)
treat the messages differently according to
g3.0 Berouuli 1+esp[-(kg (t.o +c)/7) where Bernoulli(-) is the Bernoulli approximation,
Gumbel(0. 1) is the Gumbel noise for reparameterization, and т is the temperature hyper- parameter. the forward pass, we sample the relation between s and o, if 3a.0 > 0.5, while in the
backward pass, we employ the straight- gradient can be passed through the relaxed g..o That is, only for and with 35.0 > 0.5, allow
gradient estimator (Bengio et al. 2013) so that the
message passing between and o, and consider the relation between between s and o as a candidate for pseudo-labeling.
19

---

Published as a conference paper at ICLR 2024
D.2 TRAINING
We train the GSL by optimizing the following binary classification loss:
B
Gol
LGLL =
Σ>
S )7 log(s ())66)))
6=1 k=1
where y* is the binary value that equals to relation exists between subject Sy and object Og. and otherwise 0 (i.e., when bg), and y is a hyperparameter. As the number of bg greatly outnumbers that of the remaining classes, i.e., class imbalance, we use the focal loss (Lin et al. (017)) instead of the binary cross-entree cross-entropy loss.
E EXPERIMENT ON VISUAL GENOME
E.1 DATASET
We follow the commonly used pre-processing strategies that have been extensively employed for evaluating SGG (Yoon et al. 2023 Li et al.) 2021 Zellers et al. 2018 Xu et al., 2017). The Visual Genome dataset comprising TO8K images is divided into a 70% training set and 30% test set, with 5K images from the training set utilized for the validation set. Based on the frequency of occurrence, only consider the top 150 most frequently occurring object classes and the top 50 predicate classes. Following the pre-processing steps, the average number of objects per image is 11.6, while the average number of predicates is 6.2.
E.2 EXPERIMENTAL SETTING
Evaluation Metric. We evaluate SGG models on three metries: (1) Recall@K (R@K) is the conventional metric, which calculates the proportion of top-K predicted triplets that are in ground truth. (2) mean Recall@K (mR@K) calculates the average of the recall for each predicate class, which is designed to measure the performance of SGG models under the long-tailed predicate class distribution. (3) F@K K calculates the harmonic average of R R@K and mR @ K jointly consider R@K and mR @ without trade-off between these metrics. Recent SGG studies (Li et al.) 2021] Zhang al. (022) addressing the long-tailed distribution focus enhancing the performance of the minority predicate classes (measured by mean R R@K, () since they usually include more informative descriptions in depicting scene (e.g., "walking in", "playing"). than the majority classes (c.g., "on", "has") does. However, there is a trade-off between R@ K and mR KK other words, if a model deliberately lowers the number of predictions for head predicate classes (e.g.. "on"") while increasing for tail predicate classes (e.g., "standing on". "walking on", and "walking in"), we would encounter decrease in R K and an increase in mR @ K. That being said, we can deliberately increase R@K K the expense of reduced mR @ K and vice versa. Thus, we focus on enhancing the F@ K, considering the trade-off between Recall @ K and mean Recall K.
Evaluation Protocol. We evaluate under three conventional SGG tasks (1) Predicate Classifica- tion (PredCIs) provides the ground truth bounding box and the class of entities, and then evaluate the performance of SGG models in terms of recognizing the predicate class. (2) Scene Graph Clas- sification (SGCIs) only provides the bounding box, and requires the model to predict the class of entities and predicates between them. (3) In Scene Graph Detection (SGDet), SGG models generate entity proposals, and predict the classes entities and predicates between them.
Baselines. We include the baselines that can be classified into 1) model-agnostic framework and 2) specific models. For the model-agnosti framework, we compare ST-SGG with unbiased SGG models such as TDE (Tang et al., 2020). DLFE (Chiou et al. 202T). Re-sampling (Li et al. 2021), NICE (Li et al. 2022). and IE-10 Ir-Tsa (hhang al.) 2022). Moreover, we also include the van- ant of IE- Transs i.e.. I-Trans), which excludes the bseudo-labeling mechanism (i.c.. E-Trans) on unannotated triplets, to further compare the effectiveness of its pseudo-labeling with ST-SGG. For the specific model, we include the state-of-the-art models such as DT2-ACBS (Desai et al. 2021)
PCPL Yan et al. 2020), KERN (Chen et al. 2019, and GBNet (Zareian al. 2020).
20

---

Published as a conference paper at ICLR 2024
E.3 IMPLEMENTATION DETAILS
Following previous studies (Tagg et al. 2020 Yoon et al. 2023 Li et al. 2021; Zhang et al. 2022
employ Faster R-CNN (Ren al. 2015) with ResNeXt-101 network as the object detector, whose pretrained parameters are frozen while training the SGG model. In SGDet task, select the top 80 entity proposals sorted by scores computed by object detector, and use per-class non-maxinnn non-maximal suppression (NMS) at IoU 0.5. For ST-SGG, we conduct grid search for the rate in momentum "M and a a with an interval of 0.2 (Sec. 4.2). and set the coefficient of the loss for pscudo-labeled predicates to 1.0 (Equation I in the base SGG model. On the other hand, we set the 3 to 0.1 when adopting the re- re-weight loss in Appendix E.4 Following the self-training framework, we initially train the SGG model based on the annotated triplets and then
(Xie al. (017) backbone
re-train it using both annotated triplets and pseudo-labele triplets. We set the initial value of (t) for all predicate classes to 0 based on our observation that the performance of ST-SGG is not sensitive to the initial value. We set the maximum number of pseudo-labeled instances per class to 3 in an image. Similar to IE-Trans, we give pseudo-labels on unannotated triplets only when the bounding boxes of the subject and object overlap. For the graph structure learner, we set the temperature r to 0.5 and y in focal loss to 2.0. For each experiment, used the A6000 GPU device.
Table 4: Performance (%) comparison of ST-SGG with Reweight loss (Rwt) approach on three tasks. The best performance shown in bold.
Moddl
PredCC
SGCB
SGDtt
Ko 100
I0 100
F0050/ 109
R3/50/100
5KK 100
FO55// 100
R:050/ 100
mk 100
PO:50/100
20-5/531 53.0
13./3162
00:37-43.7
11:17
179/1955
12:7134-3
15.4 15:6-325
1422323
1611315
+3 Trass
49.5/51.5
36.0/39.0
41.71443
25.0/25.0
20.5221.
23.7/24.9
23.2//2.11
14.7
15.0/21.1
1E Tuus
45.5150.5
35.8/3911
41.21
2914/303 30.3
21.0/222
245//266
23.4/27.2
199/175
18.2/21.3
ST-00
90.0
36.//6L7
42.37 44.9
76.1
24.4735.5
160/ 19.2
18.//11.
BGNN al Ruoi
57.2/59.7
25.7/26.1
35.5/35.2
345//358
25.8/29.3
14.8/17.2
-ST830
53.2/55.5
29.3/32.1
37.874007
32.7733.8 33.9
167
22.0723.7
23/6/2711
11447443
15.4
-k-Trm
24.3 19.7
15.
41.174114
79.07 101
306/31.8
34.47
11 11.87:51
1987 16.8
16.9 7011
Tius
49.5/50.4
36.5/39.5
41.31 46.2
29.5/ 30.6
205/21.7
24.2/25.4
21.2/3455
14.2/
17.07 19.9
47./49.7
37.3/40.1
41.9/44.4 44.4
29.17
21.1/33.6
24.5/25.8
181//182
17.3/30.5
E.4 RESULT WITH REWEIGHTING METHODS
In addition to the experiments with re-sampling (Li et al. 202T) and I-Trans Zhang et al. 2022) approach, which is presented in Table [ of the main paper, we further conduct experiments for ST-SGG with a conventional re-weight loss (Rwt) and show the results in Table 4 Note that we follow the conventional re-weight loss used in IE-Trans (Zhang et al. 2022). We observe that Mo- N//BGNN++WHSST-GGG generally improves the performance on mR @ K and F@ K compared to MotiZ/BGNN+Rwt. Furthermore,
achieves even better per- formance. These findings, consistent with the insights discussed in Section [..] support that ST- SGG with a simple debiasing method (i.c., re-weight) shows the effectiveness of utilizing unanno- tated triplets, resulting in further alleviating the long-tailed problem.
E.5 ZERO-SHOT PERFORMANCE ANALYSIS
Table 5: Zero-shot (zR) Performance when Motif and VCTree backbones are used.
Method
xR@ 100
Method
zR@50/ / 100
Motif
16.3 (6.3/19.4
VCTree
16.7 / 19.7
Motif- Resuf-Resupling
17.0/ 19.5
VCTreo+Rssampinng
16.3 / 19.4
Motif-IE-Tean
13.0 15.9
VCTicc+IE-Teas
14.0 16.5
Motif-ST-SGG
18.0/21.0
VCTre+ST-SGGG
/9.8/21.0
Following the zero-shot settings ( (Tang et al. 2020), we compute the recall@K values for specific (subject. predicate, object) pairs, which are not included in the training set but emerge in the testing phase. reveaas that debiasing techniques, including resampling and JE-Trans, do not enhance the zero-shot performance of SGG models. To elaborate, the limitation of the resampling approach lies its simplistic alteration of predicate class frequencies, which does not augment any variance in the data. As for IE- EE-Taas. it generates pseudo- only for minority predicates classes, which restricts the ability of finding zero-shot triplets. Conversely, the application of ST-SGG shows improvements in the zero-shot performance for both Motif and VCTree models. This improvement suggests that ST-SGG generate pseudo-labels that are instrumental in enhancing the model's
21

---

Published as a conference paper at ICLR 2024
Mori-12
Moni-rf-- -
Notii-g
of has
0.6
0.2
25
20
120
130
202
Thrshood
Motil
Mcbi-17 =
Kotil 1
mounteed on
25
10
75
100
125
110
175
203
McXi-1 :2
Moor 1f
Kotif 12"
0.8
0.1
i5
50
75
100
L25
150
175
203
ageinst
Theshold
1.
Mori-12
Motifrr
Katifr? 12
Itoraton
Figure 8: Adaptive threshold values over iterations on SGCIs task when rf=1, =0 and TO 7.ls. Motif backbone is used.
generalization to unseen triplet patterns. The effectiveness of ST-SGG in this context underscores its potential robust tool for improving the generalization capabilities of SGG models, where the ability to predict novel patterns crucial.
E.6 EFFECT OF INITIAL VALUE FOR CLASS-SPECIRE THRESHOLD
We the initial value of To for predicate classes to 0 based on our observation that the perfor- mance of ST-SGG is not sensitive to the initial value. We further showcase the effect of the initial value of the class-specific adaptive threshold. Specifically, we train Motif with bi-level sampling, and consider three cases: initial thresholds for all predicates are one (i.e., Motif-r 1), zero (i.e., Motif-r 0), and the mean value of the confidence computed on the validation (i.e., Motif- -f= in the main paper.). In Table 6 we observe that the performance of ST-SGG is not sensitive to the selection of the initial threshold in terms F@ K. Moreover, in Fig. 8 CATM even- tually adjusts thresholds for all predicates in narrow range. These results imply that CATM rapidly finds the proper threshold wherever they started from and, in turn, makes the value Te stabilize within a few iterations.
Table 6: Performance comparison when - = 1, -2 O, and T = Tcls. Motif backbone is used.
SGCIs task
Method
R@50 100
mR@50/ 100
F@50
Motif-Resampling
36.1 37.0
13.7 14.7
19.9/2100
ST-SGG w/ r-
32.2 33.6
17.3 18.5
22.5 23.9
ST-SGG w/ T =0
33.4 34.9
16.9 18.0
22.4 23.7
ST-SGG w/ dl
31.9//3.6
16.5 18.1
21.8 23.5
E.7 EFFECT OF CLASS-SPECIFIC MOMENTUM
We conducted an analysis on the effect of class class-specific momentum which is described in Section4.2 of the main paper. Fig. D a) shows that the SGG model without the class-specific momentum pro- duces significantly more pseudo-labels for majority predicates compared to minority predicates, exacerbating the long-tailed problem, which leads to a decrease in performance, as shown in Table of the main paper. On the other hand, Fig. ()) demonstrates that unannotated triplets pseudo-
22

---

Published as a conference paper at ICLR 2024
Cum. # of pseudl- labels
Majority class
Minority class
Cum. of pseudo- labels
Majority class
Minority class
6009
6000
6000
5000
$500
5000
3000
5000
grumiag
4000
4000
2500
2000
2000
2000
1000
1000
1000
1000
400
2050
2570
500
2500
3050
2500
2559
550
Iteration (t)
iueration (t)
iteration (4)
iteration (t)
(a) Without Class-specific Momentum
(b) With Class-specific Momentum
Figure 9: The cumulative number of assigned pseudo-labels of several predicate classes in the ma-
jority and minority classes (a) without class-specific momentum (i.c., A Adee 0.5) and (b) with class-specific momentum. Unannotated triplets are mainly pscudo-labeled with majority (i.e., head) predicate classes in (a), whereas minority (i.c., tail) predicate classes are assigned more in (b).
labeled more with predicates from minority classes compared to majority classes. This demonstrates that applying class-specific momentum relieves the long-tailed problem by generating the pseudo- labels annotated with minority predicates classes.
E.8
EFFECT OF inc AND a dec
1.0
1.0
1.0
16.2
16.7
18.5
19.1
19.5
17.9
0.8
0.8
08
17.9
18.0
20.8
20.5
20.7
21.5
0.6
0.6
0.6
18.9
19.9
21.7
23.2
21.8
21.0
1nc
0.6
20.5
23.4
21.6
22.6
22.4
0.2
42
20.2
23.6
23.1
21.8
0.0
Majoriy Class
Minority
Majorty
Minucity Class
0.0
21.0
20.9
50.0 19.8 20.4
0,0
02
0.5 dec
α
Figure 10: Aire and A according to the change of a ine and Figure 11: Effect of ane and dee with F@ 100 performance.
deo respectively, which represented by lines' color.
Fig. 11 shows the performance (F@ 100) according to the change of a ine (y-axis) and a dec (x- axis), which control the increasing and decreasing rate in EMA, respectively. Note that the smaller the value of ine or dee is, the more aggressive the threshold of minority classes increases or decreases, which is described in detail in Fig. 10 We observe that high value of te (row 1, 2) leads to decrease in performance since a high value of a the severely restricts the increase of the threshold for minority classes, resulting in incorrect pscudo-label assignment for minority classes. Compared to the performance the high value of 'ne the high value of dee shows relatively competitive performance. This suggests that a gradual decrease in the threshold for minority classes acceptable since it keeps assigning the confident pseudo-labels. However, the best performance achieved with a inc = 0.4 and dec = 0.4, rather than dee = 1.0 or 0.8, indicating that in the self- self-training for SGG task, it is beneficial to appropriately increase decrease the threshold while assigning more pseudo-labee labels on minority classes.
E.9 EFFECT OF B
Fig. 12 shows the performance with respect to the coefficient of the loss for the pseudo-labeled predicates i.c., 3 in Equation [ Here, we also employ the Motif backbone network trained with bi-level resampling techniques (Li et al.) 2021). We observed consistent performance of ST-SGG ranging from 0.1 to 1.0 for the coefficient
Performance Metric
35.0
32.5
22.5
20.0-
17.5
Facto
3. which indicates that ST-SGG not sensitive to the coefficient of the loss for the pseudo-labeled predicates. Moreover, we noticed performance drop when 3 becomes too large, suggesting that it is preferable to prioritize the existing annotated predicates over the pseudo-labeled predicates to train SGG models.
T
15.0
1.0
3.0
Figure 12: Sensitivity of B in
Equation 1
23

---

Published as a conference paper at ICLR 2024
E.10 COMPLEXITY ANALYSIS
Table 7 shows the total training time of Motif-Vanilla, IE-Trans, and ST-SGG re- quired for 20,000 training iterations. Note that the 20,000 iterations include "Pseudo- labeling." " and "Re-training." " We observed that ST-SGG requires shorter training time than IE-Trans for pseudo-labeling and re-training. This implies that the iterative pseudo-labeling process of ST- SGG is computationally more efficient than the pseudo-labeling process adopted by IE-Trans.
Table 7: Training time of Motif-backbone models dur- ing 20,000 iterations with batch size A6000 GPU device is used.
Trainig time PredCls task (20,000 iterations)
Model
Self-training
Pre Pre-training
Total
Pacudo-labeling
Re- Re-training
Vanilla
2h 21m
2h 21m
IE: Trans
1h 39m
2h 21m
6h 21m
ST-SGG
2h 42m
Sh 03m
Please note that ST-SGG utilizes the confidence originally produced by the backbone SGG model during the training stage, resulting in very low additional computational cost. For example, when Motif-vanilla generates the confidence compute the loss for annotated triplets, ST-SGG compares with the leamed thresholds and determines whether to include in the loss for pseudo-labele labeled predicates. In other words. ST-SGG only requires additional computations for calculating the gra- dients of the loss for pscudo-labeled predicates. On the other hand, IE- Trans has additional inference stage after training Motif-Vanilla, and stores confidence for every entity pair in the dataset to obtain the rank of all confidence.
F EXPERIMENT ON OPEN IMAGE V6
F.1 DATASET
We closely follow the data processing of previous works (Li al. 2021 Yoon et al. 2023) for Open
Image V6 (0I-V6). After preprocessing, OI-V6 is split into 126,368 train images, 1,813 validation images, and 6,322 test images, and contains 301 object classes, and predicate classes.
F.2 EXPERIMENTAL SETTING
Evaluation Metric. Following (Zhang et al. 2019), we additionally evaluate SGG models on three
metrics: wmAP rel is weighted mean average precision of relationships, wmAP is weighted mean average precision of phrase, and the final score computed by score 0.2 R@50 + 0.4 X wmAP 0.4 X wmAPppr. phr Specifically, wmAPre! evaluates the average precision of subject, pred- icate and object, where both the ground truth boxes of subject and object have an IOU greater than 0.5 with the ground truth bounding boxes, and compute the weighted sum of the average precision scaled with each frequency of predicate class. wmAP pAr is similarly computed wmAP, Prel- but it computes single bounding box enclosing both subject and object with IOU greater than 0.5.
Evaluation Protocol. Following the previous works (Li et al. 2021 Lin al. 2020 Zhang al. 2019), we evaluate on Scene Graph Detection (SGDet) task, where SGG models generate entity and relation proposals, and predict the classes of entities and the class of predicates between them.
F.3 COMPARISON WITH THE OF-TT MODELS
Table g shows the result in SGDet task on Open Image datasets. We have the following obser- vations: 1) ST-SGG generally improves Motif, Motif with the resampling, and BGNN in terms of mR @ 50 and F@500 implying that the long-tailed problem is alleviated by our pseudo-labeling ST-SGG shows competitive performance among those with Mo- tif backbone in terms of R 55 and wmAP. It is important to note that R 50 and WmAFT] metrics primarily emphasize the performance of head predicates, which is in contrast the objective of ad- dressing the toilet problem. This implics that our proposed framework retains the performance on head predicates while achieving the state-of-art performance on tail predicates. This implication
method. 2) Interestingly.
'The WmAP metric is computed by APral weight, where the weight based on the frequency in the data.
24

---

Published as a conference paper at ICLR 2024
Table 8: Results on Open Images V6 (Kuznetsova et al. 2020). Bold represents the best performance, and 0.4 underline represents the second best.
Mcu1951300
Weight
Model
R.8500
mR
F-8:50
wmAPFr
wmnFfp
100
ReIDD/Zhuss
bang
platt
interant paae
sus
GPS-Ne 1m
A0r0
+TDR Tang
Figure 13: APrel performance compari- son per class. The blue line and Weight (i.e., right y-axis) represent the frequency ratio the test data.
+Ress PRNE
4Resamp-
25.2
can be further explored by referring to Fig. 1 which displays the performance AP for each class. Specifically, for the head predicates, + achieves competitive results, particularly in the case of the contain predicate. For the tail predicates, Motif+ -O-if+ST-SGG significantly enhances the performance, particularly in the cases where Motif struggles to make accurate predictions, such as ski predicates.
G LIMITATION & FUTURE WORK
Although our current work (including existing SGG models) focuses on utilizing benchmark scene graph datasets, ST-SGG has potential leverage external resources. For instance, ST-SGG can benefit from leveraging localization datasets (Lin et al., 2014) containing only the bounding box information without annotation. Specifically, obtained from localization datasets in the second and third terms in the loss (Equation I i.c. add them to G This approach is promising in utilizing resources enrich the benchmark scene graph datasets, where acquiring annotated triplets is costly.
include these unannotated bounding boxes
25


Reviews:
[{"review_id": "CkdIukrEm4", "reviewer": "Reviewer_jErj", "summary": "The paper proposed the ST-SGG framework to address the long-tailed predicate issue in Scene Graph Generation (SGG). It incorporates the CATM pseudo-labeling method and a Graph Structure Learner (GSL). Experimental results confirm improved performance on fine-grained predicate classes.", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- ST-SGG serves as a model-agnostic framework, meaning it can be applied to various existing SGG models. This aspect has the potential to expand the applicability of the self-training in SGG.\n- Based on experimental results, the proposed framework seems to effectively alleviate the issues of long-tailed distribution. The performance improvements are primarily concentrated on fine-grained predicate classes.", "weaknesses": "- In some experiments, the R@k values significantly decreased after employing the proposed framework. The paper lacks some explanations for this phenomenon.\n- There's a concern if the proposed framework might, in some scenarios, sacrifice a considerable amount of overall performance to achieve improvement in fine-grained predicates.\n- The paper lacks a clear depiction of the overall framework structure. Please clearly demonstrate how the different components interact and how pseudo-labels are generated and applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsZuZXxDf5", "reviewer": "Reviewer_NQzC", "summary": "This paper proposes a novel self-training algorithm of training scene graph models based on un-annotated triplet loss. The distinction between un-annotated and background relationship class is drawn with the help of a class-specific dynamic threshold. Proposed method has been validated in Visual Genome dataset with two classic SGG baseline (MOTIF and VCTREE) and with four recent-most SGG baselines (Resam. IT-SGG, BGNN, and HetSGG). Proposed method increases the performance of the minority classes with slight decrease of majority classes.", "questions": "The authors determine the threshold in an object-agnostic fashion considering only the relationship classes. However, the same relationship label has different semantic meaning to different pair and therefore an object-conditional viewpoint of relationship distribution may improve the result more. \n\nTo clarify more, in the loss calculation phase, if we consider the viability of a prediction conditioned on its object, we might have better choice on the threshold. For example, the prediction probability of 'on' will be higher for 'building-window' pair, however, it will be extremely low for 'man-pizza' pair. Is it possible to use such commonsense knowledge in the threshold choice of the loss equation?\n\n**Post-rebuttal rating:** My concerns and queries have been properly addressed and therefore, I keep my original rating of 8.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths\n\n**1. Well-motivated:** The paper's motivation is well-stated and clearly disseminated to the reader. Exploring vast un-annotated triplets in the image databases is one of the desired direction of open-set annotation dataset and Visual Genome is one of the most useful resources for conducting such exploration. This paper successfully established this motivation in their introduction. \n\n**2. Novel self-training loss:** The proposed simple yet novel and effective loss is one of the major strengths of the paper. The discussion regarding challenges of fixed-threshold based distinction between background class and un-annotated class is interesting and highly relevant. The proposed class-specific thresholding solves the problem efficiently and demands further exploration. \n\n**3. Less worsening of majority classes:** Traditional debiasing schemes of SGG usually hurt the recall classes significantly while improving the minority classes. However, this paper hurts the recall minimally, especially with the baseline MOTIF and VCTree. \n\n**4. Detailed ablation study on thresholding:** Since the main contribution of this paper is a threshold-based self-training algorithm, authors have performed detailed analysis on how to choose appropriate threshold.", "weaknesses": "This paper has the following weaknesses\n\n**1. Comparison with other debiasing methods missing.** The paper demonstrated that their method can improve the diversity of the baseline models. However, such improvement of baseline models is prevalent in SGG literature now. Therefore, a direct comparison with other debiasing methods would shed light more on their performances. For example,  VCTree+ST-SGG should be compared with VCTree+Resample. With that comparison, the readers would have better understanding of their methods performance. \n\n**2. Zero-shot evaluation missing.** Since unannotated triplets are being accounted in the training, this method might improve the zero-shot performance of the baseline model as well. A separate study with zero-shot evaluation would strengthen the claim of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9YsLGAnPYi", "reviewer": "Reviewer_QGUt", "summary": "This paper proposes a self-training framework for the SGG task, which uses the unannotated triplets for imbalanced SGG. With the class-specific adaptive thresholding with momentum algorithm, their model can obtain some pseudo labels to deal with the missing annotations problem. Finally, they verify the effectiveness of ST-SGG on VG and OI-V6 datasets.", "questions": "See Strengths and Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea is well-motivated and intuitive. They provide an in-depth analysis of long-tailed SGG.\n2. The paper is well written and easy to understand.\n3. Extensive experiments and ablations verify the effectiveness of enhancing the performance of tail classes.", "weaknesses": "1.The technical contribution of this paper might be limited. The self-training framework is used for image classification, the adaptive thresholding is used to obtain accurate pseudo labels in many tasks, and the GSL is also not new. The authors should take more discussion about their novelty and variant in SGG.\n\n2.There are some SOTA SGG methods which might get higher performance than this paper. These methods should also be included for comparisons and analyses, such as VETO[1], GCL[2], PE-Net-Reweigh[3], and so on.\n[1] Vision Relation Transformer for Unbiased Scene Graph Generation, ICCV2023.\n[2] Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation, CVPR 2022.\n[3] Prototype-based Embedding Network for Scene Graph Generation, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)