Evidential Uncertainty and Diversity Guided Active Learning for Scene Graph Generation
Authors: Shuzhou Sun
Shuaifeng Zhi
Janne Heikkilä
Li Liu
Venue: ICLR 2023 poster (2023)
Keywords: Active learning
Scene graph generation
Uncertainty estimation

Abstract: Scene Graph Generation (SGG) has already shown its great potential in various downstream tasks, but it comes at the price of a prohibitively expensive annotation process. To reduce the annotation cost, we propose using Active Learning (AL) for sampling the most informative data. However, directly porting current AL methods to the SGG task poses the following challenges: 1) unreliable uncertainty estimates, and 2) data bias problems. To deal with these challenges, we propose EDAL (\textbf{E}vidential Uncertainty and \textbf{D}iversity Guided Deep \textbf{A}ctive \textbf{L}earning), a novel AL framework tailored for the SGG task. For challenge 1), we start with Evidential Deep Learning (EDL) coupled with a global relationship mining approach to estimate uncertainty, which can effectively overcome the perturbations of open-set relationships and background-relationships to obtain reliable uncertainty estimates. To address challenge 2), we seek the diversity-based method and design the Context Blocking Module (CBM) and Image Blocking Module (IBM) to alleviate context-level bias and image-level bias, respectively. Experiments show that our AL framework can approach the performance of a fully supervised SGG model with only about $10\%$ annotation cost. Furthermore, our ablation studies indicate that introducing AL into the SGG will face many challenges not observed in other vision tasks that are successfully overcome by our new modules. 

Published as a conference paper at ICLR 2023
EVIDENTIAL UNCERTAINTY AND DIVERSITY GUIDED ACTIVE LEARNING FOR SCENE GRAPH GENERATION
Shuzhou Sun', Shuaaieng Zhi Janne Heikkilä', Li Li
'University of Oulu, "National University of Defense Technology *Corresponding author: Li Liu (dreamliu204 0 @mmall com
ABSTRACT
Scene Graph Generation (SGG) has already shown its great potential in various downstream tasks, but it comes at the price of a prohibitively expensive annotation process. To reduce the annotation cost, propose using Active Learning (AL) for sampling the most informative data. However, directly porting current AL methods to the SGG task poses the following challenges: 1) unreliable uncertainty estimates and 2) data bias problems. To deal with these challenges, we propose EDAL (Evidential Uncertainty and Diversity Guided Deep Active Learning), a novel AL framework tailored for the SGG task. For challenge I), we start with Evidential Deep Learning (EDL) coupled with a global relationship mining ap- proach to estimate uncertainty, which can effectively overcome the perturbations open-set relationships and tainty estimates. To address challenge 2). we seek the diversity-based method and design the Context Blocking Module and Image Blocking Module to alleviate context-level bias and image-level bias, respectively. Experiments show that AL framework can approach the performance of a fully supervised SGG model with only about 10% annotation cost. Furthermore, our ablation studies indicate that introducing AL into the SGG will face many challenges not observed in other vision tasks that are successfully overcome by our new modules.
to obtain reliable uncer-
INTRODUCTION
Scene Graph Generation (SGG) (Johnson et al., 2015) aims at generating a structured representation of a scene that jointly describes objects and their attributes, as well as their pairwise relationships. SGG has attracted significant attention as it provides rich semantic relationships of the visual scenes and has great potential for improving various other vision tasks, such as object detection (Ren al., 2015; Redmon et al., 2016), image search (Gong et al., 2012; Noh et al., 2017), and visual ques- tion answering (Antol et al., 2015; Zhu et al., 2016). Albeit being an emerging area of research, which can bridge the gap between computer vision and natural language processing, SGG is still underexplored despite many recent works focusing on SGG (Chang et al., 2021; Zhu et al., 2022).
The main challenges that impede the advancement of SGG are twofold. On the one hand, existing datasets for SGG (Krishna et al., 2017: Lu et al., 2016) suffer from many serious issues, such as long- tailed distribution, noisy and missing annotations, which makes it difficult to supervise satisfactory model. On the other hand, existing deep learning-based SGG methods are data hungry, requiring tens or hundreds of labeled samples. However, acquiring high-quality labeled data can be very costly, which is especially the case for SGG. The reason for this is that SGG involves labeling visual <subject, relationship, object> triplets (e.g., <people, ride, bike>) over entity and relationship classes in an image, which can be difficult and time consuming (Yang et al., 2021; Shi et al., 2021; Guo et al., 2021). Therefore, it is highly desirable to minimize the number of labeled samples needed to train a well-performing model. Active Learning (AL) provides a solid framework to mitigate this problem (Yoo & Kweon, 2019; Kirsch et al., 2019; Huang et al., 2010; 2021). It is, therefore, natural to investigate whether AL can be used to save labeling costs while maintaining accuracy, which is the focus of this paper. In AL, the model selects the most informative examples from an unlabeled pool according to some criteria for manual labeling. and then the model is retrained and evaluated with the selected examples. This looks intuitive yet simple, but directly transferring existing AL methods to the SGG task will face several challenges.

---

Published as a conference paper at ICLR 2023
First, existing batch query-based based AL paradigms (Gudovskiy et al., 2020; Kim et al., 2021; Mahmood et al., 2021; Sener & Savarese, 2017: Tan et al., 2021) used for SGG face large number of open- set relationships, i.e., relationships that appear in the unlabeled pool but are absent in labeled data, mainly because of the severe long-tailed distribution of the SGG relationships (Zellers et al., 2018; Tang et al., 2020a;b). We observe that existing uncertainty estimation approaches perform badly in classifying SGG relationships, especially open-set relationships. Inspired by Evidential Deep Learning (EDL) (Sensoy et al., 2018) and its advanced performance in open-set action recognition (Bao et al., 2021), we enhance and incorporate it into our proposed AL framework to estimate the relationship uncertainty. Second, the relationship annotations in SGG dataset are very sparse, result- ing in severe foreground-back background imbalance (Xu et al., 2017; Goel et al., 2022). Foreground- relationships are those within annotated triplets in the dataset, while background-relationsss the ones that are absent between object pairs. Due to the large number of background-relationshiip they can perturb or even dominate the uncertainty estimation. To this end, we propose a relation- ship mining module, Relationship Proposal Graph (RPG), as part of the uncertainty estimation, which works by filtering out mackground-relationssiip to refine the uncertainty obtained by EDL. Third, despite our improved EDL having the capability to generate reliable estimates of relationship uncertainty, its sampling results are still vulnerable to the problems of traditional uncertainty-basee AL methods, i.e., data bias problems (Kim al., 2021; Shen et al., 2017; Luo et al., 2013). More importantly. we also found that uncertainty-busee AL used for SGG will be biased at both context- level and image-level, where the context in SGG refers to the feature space formed by relationship triplets. For this issue, we design the Context Blocking Module (CBM) and the Image Blocking Module (IBM), which are inspired by diversity-based based AL methods. The former can block similar contexts to avoid the context-level bias, while the latter can block redundant images to eliminate the image-level bias.
are
Contributions.
The main contributions of this work are the following: (1) We carry out pioneering study of using AL for SGG to achieve label efficiency without significantly sacrificing performance loss and propose novel framework dubbed Evidential Uncertainty and Diversity Guided Deep Active Learning (EDAL). (2) In the proposed EDAL framework, we introduce novel evidential un- certainty to guide deep active learning and efficient one-shot estimation of relationship uncertainty. In this process, a relationship mining module is designed to avoid the perturbation of uncertainty es- timation by deckground-eelationsiin
In order to effectively mitigate context-level and image-level bias problems induced by AL, we design two modules, CBM and IBM. (3) Extensive experimental results on the SGG benchmarks demonstrate that EDAL can significantly save human annotation costs, approaching the performance of a fully supervised model with only about 10% labeling cost.
2 RELATED WORK
Scene Graph Generation (SGG). SGG extracts a structured representation of the scene by assign- ing appropriate relationships to object pairs and enables a more comprehensive understanding of the scene for intelligent agents (Johnson et al., 2015; Lu et al., 2016; Krishna et al., 2017; Liu et al., 2021: Yin et al., 2018). For supervised training of the SGG task, a massive amount of triplets within images in the form of <subject, relation, object> need to be provided, which involves several sub-tasks including object detection, object recognition and relationship description, and results in an unaffordable annotation cost. To mitigate this, (Chen et al., 2019) proposed a semi-supervised method for SGG, which requires only a small amount of labeled data for each relationship and generates pseudo-labels for the remaining samples using image-agnostic features. However, these pseudo-labels tend to converge to a few dominant relationships. (Ye & Kovashka, 2021) designed a weak supervision framework to reduce the reliance on Labor-intensive annotations with the help of linguistic structures. Recently, (Yao et al., 2021) trained an SGG model in an unsupervised man- ner by drawing on knowledge bases extracted from the triplets of web-scale image captions. Despite showing the promise of label efficient learning techniques in SGG, the above caption- based methods rely on large-scale external linguistic knowledge which fits the target scene. This, to some extent, limits its generalization to other scenes without adequate linguistic priors We explore an alterna- tive approach and propose a hybrid AL framework tailored to the SGG task in order to avoid the expensive labeling cost without access to external knowledge.
Active Learning (AL). AL aims to select the most informative data from the unlabeled pool for annotation to support model training. In vision tasks such image classification and object detec-
2

---

Published as a conference paper at ICLR 2023
Sumple results this AL round
Initil pretraining
Uncertainty based module
Diversity based module
Initial labeled data
SGG Model
Iondental Uncertiinty
RPG Micaale
CBM Modile
Module
Prefimrnsyyy
Retraining model
Oracle
Update
Unlabeled pool
Labeled
(a) The proposed overall framework
Evidential layer
Evidential Uncertainty
background relationships
Context Blocking Module (CBM)
image
SGG Model
remove
contexts
foreground
k aa
data bias at context-level
relationship
low uncertainty
high uncertainty
back Brokground
Remove background relations
Image Blocking Module (IBM)
relationship
Relationship Proposal Graph (RPG)
ground truth
Relation mining
Labeled
pool
Max
similar images
remove
prediction
data bias at image-leve
SG
RPG
uncertainty
(b) Uncertainty based module
(c) Diversity based module
The overall structure of our proposed AL framework EDAL. Following the standard AL setup, EDAL samples data from the unlabeled pool round by round to support the model training, and the quit condition that the label budget exhausted.
Figure 1:
tion, AL has been widely applied with impressive performance. Uncertainty- based methods (Shen al., 2017; Luo al., 2013; Yoo Kweon, 2019; Huang et al., 2021; Gudovskiy et al., 2020; Kim et al. 2021) and diversity-based methods (Mahmood et al., 2021; Sener & Savarese, 2017; Tan et al., 2021) are currently two types of mainstream AL technologies. For the unlabeled pool, the former aims to estimate uncertainty robustly, while the latter samples the data dispersedly possible within their feature space. However, the uncertainty-basee methods tend to bias towards subset of the unlabeled pool, mainly caused by unreliable uncertainty estimation with little labeled data, methods often struggle with large-scale datasets and complex tasks (Mah-
and the diversity- mood et al., 2021; Shi et al., 2021). Inspired by the above facts, propose hybrid AL strategy with elements from both uncertainty-basee and diversity-based methods, which is, to the best of our knowledge, the first hybrid AL method for the challenging SGG task.
Evidential Deep Learning (EDL). The pioneering work (Sensoy er al., 2018) proposes using EDL to estimate reliable classification uncertainty, especially for open-set entities, without loss perfor- mance. The authors design predictive distribution for classification by placing a Dirichlet distribu- tion. Deep Evidential Regression (Amini al., 2020) introduces the evidential theory to regression tasks by placing evidential priors over the original Gaussian likelihood functions. Recently, (Bao al., 2021) used EDL to estimate uncertainty for open-set action recognition. model calibration method is also proposed to regularize EDL training and thus mitigate the overfitting problem. In this paper, we employ EDL to perform uncertainty estimation the proposed hybrid AL framework. Our main motivation lies in its advantages for reliable open-set uncertainty estimation and the fact that a large number of open-set relationships exist for AL algorithms in the SGG task.
3 METHOD
The overall pipeline of EDAL is shown in Figure which is a hybrid AL model composed of uncertainty-based and diversity-based methods. First, the evidential uncertainty estimation method applied with the extracted prior information from the available labeled data samples to estimate the relationship uncertainty (Section 3.1). Diversity-basee sampling driven by context-level and image-level biases is then adopted to refine selected samples to reach the labeling budget (Section 3.2). A pseudo-code of EDAL is given in Appendix A.1.
3

---

Published as a conference paper at ICLR 2023
people:
chai::
low uncertintt
dogg
high uncertiitty
Busting Shall
Unartainty
people
leac
Seyinng (8:87)
stadd
Ulaer
racket
racket
Without CDM: 2.55
Clas
With CBM: 2,02
(c)
Figure 2: The motivations for key designs in EDAL. (a) motivation for EDL: relationship distribu- tions (top) and average output logits of each relationship (bottom). (b) motivation for RPG: fore-
ground relationship (solid lines) and background relationship (dotted lines). (c) motivation for CBM: the target scene contains many similar contexts (sitting on in this example). (d) motivation for IBM:
the sampling results are biased towards partial relationships (Class and 3) in this example.
3.1 EVIDENTIAL UNCERTAINTY BASED SAMPLING
Assume that the unlabeled data of the /-th AL round is (/1, 12. and that the i-th image
1, contains Tı| objects and most T₂| x (ITa) 1) relationship triplets {< _f >, >,< of,,, >}. In Section 3.1.1, we design an uncertainty estimation method for each relationship. Then, in Section 3.1.2, overcome the perturbation of the background-relationshij to the uncertainty estimate by the proposed relationship mining module.
of < 0, o 7
3.1.1 EVIDENTIAL UNCERTAINTY ESTIMATION
The SGG task suffers from severe long-tailed distribution of relationships, as shown in Figure (a). In the VG150 dataset(Xu et al., 2017), the top-5 categories amount to more than half, of which we regard top-5 relationships that the random samples of the first AL round will only cover a small number of categories and, thus, inevitably yield lot of open-set relationships in the unlaheled pool. However, commonly adopted woobability-basee losses (e.g., cross-entropy loss with Softmax probabilities) often provide unreliable uncertainty estimates, such as false over-confidence predictions, for open-set samples (Sensoy et al., 2018). On the contrary, EDL is able to predict reliable uncertainty estimates in this scenario by collecting evidence from each output class by placing a Dirichlet distribution over the class probabilities. For an open-set relationship that does not belong to any available labeled classes, ideally, the model would not be able to collect any evidence that the input belongs to a known relationship. In this case, the parameters of the Dirichlet distribution become all one and lead to an output with a high uncertainty value of 1. Inspired by EDL and its impressive performance in open- set action recognition (Bao et al., 2021), we introduce the modified EDL to estimate the uncertainty of relationships.
the head categories and the rest the tail categories. It is expected
Specifically, we denote an object pair and its ground truth relationship æ; and Ytj. respectively, where j E (1, 2, ........ } is the index of & relationship categories. Hence Vij = 1 x; owns the j-th relationship, and otherwise. The vanilla EDL supervises the 1-th AL model F, with the following loss function to determine the expected probability Pıj that x; belongs to the j-th category:
Pa (1 Pis) (St + 1)
LEDI (F.) = Σ (yig 2
KL (.) (D K 1, 1,)]]
(I)
(1)
where esj is the j-th output logit of I,, and it acts as the collected evidence. aij = Egj + 1. S, E/e Oaj is the strength of the Dirichlet distribution D(p: a;), where p; is simplex representing class assignment probabilities, a; (a....... a.x). Dij = /S is the EDL output for x;. KL :-1-]] is the KL divergence loss used for regularization, and ã; is the Dirichlet parameters after removal of the non- misleadiid evidence from predicted parameters a4 During inference, the uncertainty of z; can be estimated via evidential uncertainty u, k/S, with maximum value of 1. We refer our readers to (Sensoy et al., 2018) for more details of evidential theory and its derivations. Therefore, for image 7, estimate the uncertainty of its triplets and denote the
4

---

Published as a conference paper at ICLR 2023
-. However, even though term (2) of Equation 1 has been used as a regularizer, we found that the above vanilla EDL loss suffers from severe overfitting problem when supervising the SGG model (see Appendix A.2). We argue that the insufficient regularization is due to the large number of background-relationstipp in the SGG task making the prediction of foreground poreground-relationssips indiscriminate, and this intractable problem is shown in 2 (a). Besides, the vanilla EDL loss may be hampered by incidental vanishing gradient
result as U1; [U (ess::Uc]__o... [oo!_
because its gradient is VLEDI = N Σ (f yey) o' (f (z.)) where o is the ReLU activation function (Sensoy et al., 2018), and therefore, o' 0 when f (1:) 0. For the above issues, we find that the evidential-probabiin based cross-entropy loss Lepoe can be used as a strong regularizer when replacing term (2) of Equation 1. Specifically, the final loss of our EDL can be formulated as:
K
(yay Dz))² + Piy (1 Dij)
LEDI (Ft) = Σ
+ +1- log (Dis) Moll
(S+ + 1)
(2):Tappe
(1)
The cross-entropy loss is often paired with softmax, the standard output of classification models, which provides an inflated probability due to its exponential transformation on the output. How- ever, Lepce penalizes hidential-probabbiity, which is second-order probability derived from the parameterized Dirichlet distribution (Sensoy et al., 2018). We argue that Lepce can effectively pe- nalize non-discriminative prediction foreground-relatioonsiii,,, and, thus, can be seen as strong regularizer. More importantly, cross-entropy loss based on evidential evidential-probabbliy can prevent over- confident predictions open-set set relationships without affecting the uncertainty estimates.
3.1.2 RELATIONSHIP PROPOSAL GRAPH
The relationship annotations in the SGG dataset are very sparse and cause severe foreground- background imbalance problem (Xu et al., 2017; Goel et al., 2022). For example, with five objects in Figure 2 (b), there are, in principle, twenty possible relationships, but only four are annotated and therefore belong to foreground relationships (e.g., <woman, holding, racket>, etc.). The large number of backkround--elattonsshn
is dominant and may disturb uncertainty estimation for sparse and valuable foreground ones within the scene. An intuitive idea is to use off-the-shelf methods which can recognize whether relationship belongs the foreground or not, such as Directed Semantic Action Graph (Liang et al., 2017), Confusion matrix (Hwang et al., 2), Statistical prior (Guo et al., 2021) 1), etc. Unfortunately, these methods are driven by a large amount of labeled data, and the limited amount of annotation in the AL setting is not enough for them. To tackle this issue, we further leverage global information across available labeled images to refine the instance uncertainty obtained in Section 1. Concretely, relationship mining method is proposed to infer how likely there exists a dnnggroudd-relations agnostic), which serves weight to refine evidential uncertainty estimation.
r between two object classes p and q (i.e., instance
First, we generate proximity cluster C<p.g> = Op. r, O, >} to accumulate correlation between two object classes with foreground-relationssiin from available annotated triplets, where Op and O₂ are object instances of classes p and q, respectively; r is a inagroound-rllationnss The cluster size [C=pua> is determined by the number of triplets within it. Note that proximity clusters are instance agnostic. Hence triplets from different images can contribute to the same cluster.
from labeled data.
We then extract statistical graph (SG) and probabilistic graph (PG) from proximity clusters to represent bregrrouddbackkrrone probability of a relationship with different neighboring orders. The statistical graph (SG) models the existence of foreground-relan relationships whose nodes represent object categories and the edge weights w are binary values depending on whether the correspond- ing proximity cluster is empty or not. An edge with values of or O indicates that the corresponding relationship belongs to the foreground or background, respectively. Considering the lack of repre- sentativeness of a few labeled data samples, we further construct a probabilistic graph (PG) which infers the potential likelihood of the relationship borhood of the nodes into consideration. Specifically, PG has the same nodes as SG but instead assigns soft edge weights between nodes if they both are connected to shared neighboring node in SG with non-zero edge weight. By extending proximity cluster to second-order neighboring, we define C2 C2p.92 = {< Op: rl, Om >,< Om, r2, O,, >} where O,, Om, O,, are instances of object
foreground by taking the second-ordes neigh-
5

---

Published as a conference paper at ICLR 2023
classes p, m and q, respectively; ri and 12 are two foregroumd-relatioonhhips We can then compute the edge weight RP of PG between classes p and as follows: ppg
PG w P.O
= /ee2.il//Maa (1, pc 122 IC2 <p,
(3)
where i is the number of object classes. The intuition of PG is that we observe nodes within the second-order foreground neighborhood also possibly correlate via ooregroudd-ellatiossii and we create PG to take full advantage of such information from the limited labeled data.
Finally, the Relationship Proposal Graph (RPG) is obtained by merging SG and PG to infer our fi- nal likelihood of owning foreground-relationsshin between two classes. RPG has the same nodes as PG and SG whose weights wR are computed via Max(w p.g' wy(g), where Max(..-) is an operation kernel taking the maximum edge weights between the same two corresponding nodes. RPG can be thought of as class- wise prior information of the existence of foreground-relationshin considering both first and second order adjacent nodes from the pool of labeled triplets. We can use wpgg to weight Ur, to overcome the perturbation of background-relationhiiss Specifically, we denote the weighted uncertainty as U/ Uj, {U<ot_oss x
RPG 00,00 Uzo x RPG o3.00"
RPG - where oद represents the category to which object of belongs. U [ x w a
Please note that as RPG is retrieved using only sampled data during training and its computational overhead is marginal, we, therefore, keep RPG constantly updated whenever new labeled data arrive.
3.2 DIVERSITY-E BASED BLOCKING MODULE
In this subsection, we introduce the Context Blocking Module (CBM) and the Image Blocking Module (IBM) to cope with the data bias issue of the uncertainty- based AL methods.
As to the context-level data bias illustrated in Figure 2 (c), scene may contain many similar con- texts (e.g., <people, sitting on, chair>). We argue that when considering the potential similarity of objects in the learned feature space, not all triplets sharing similar contexts contribute equally the performance of SGG but instead increase the labeling cost and computational expense. Image-level data bias is also shown in Figure (d). The uncertainty-basse sampling will be biased towards the classes where the current model performs poorly (the sampling results will bias toward data from Class 1 and Class 3 in this case). which has been discussed in the existing literature as well (Huang et al., 2021; Mahmood et al., 2021; Sener & Savarese, 2017; Tan et al., 2021). See Appendix A.3 for further discussion of the data bias problem.
To alleviate the data bias problem at the context-level, we propose CBM to block certain triplets/ohject pairs sharing similar contexts. Let the context feature of T in the 1-th AL round be f1g, J1.1 = {sT ST I ( J. where L/,,, is the j-th context feature extracted by the model Ft-1. We then calculate the context-level density of I denoted by DI.C = (D (11)))1)) where Dj..c (E ([((((()-1)) Iff, - fh,,l)-1. Similar con- texts with the largest density value are removed, whose actual number is determined by a pre-set deduplication ratio t, and we update the uncertainty.
{D),.o Di....... ,DW, I.CC
For the data bias problem the image-level, we propose IBM to filter images that contribute simi- larly to SGG. Let the labeling budget at /-th round is &, Although typical AL methods often directly select la samples with the largest uncertainty, in EDAL & (Es > (e) preliminary samples with the highest uncertainties are first chosen, i.c. S₁,7, = {11 12, - ,, -z Similar to the density compu- tation in CBM, we also calculate the image-level density Ds
DI.. Diz, }, where DI.,I = EEA= 1fi,, 1l. Si., is the i-th image feature extracted by the model Fi-1- Fi- nally, we remove l, - b, images with the largest image-level density value to reach the desired labeling budget &, Note that because l, in IBM is the actual labeling budget and & is a hyperpa- rameter related to the degree of image-level bias. We argue that E, should be gradually increased during training, as several existing AL works have shown that the data bias problem will become more and more serious as AL progresses. Specifically, & in our framework can be computed as E 4 + X [4.1 t", and we call it De-bias Temperature Units (DeTu). In the Appendix A.4, we explain why & is modeled in an exponential form rather than constant & l, A1 [24e] (De-bias
6

---

Published as a conference paper at ICLR 2023
DCCK
500k
(a) MotifsNet
(b) VCTree
(c) Transformer
Figure 3: Active Learning performance of EDAL and baseline methods. mR @/20/50/100 represents the average performance of @@ 20, R 50, and R @ 100. We repeat each experiment three times and report the mean (solid line) and standard deviation (shadow).
Constant value, DeCv) linear form & le + A2 [4/ (De-bias Linear Units, DeLu), We will also quantify the advantages of DeTu through the ablation study (Section 4.3).
EXPERIMENTS
4.1 EXPERIMENT SETUP
5-th Al round
Implementations. We show the effectiivness of EDAL on the popular VG150 dataset (Xu et al., 2017) for the
SGG task through extensive experiments. To prepare the AL dataset, we first remove all the relationship annota- tions from the official training (about 56k images) create the initial unlabeled pool 2/ with object annotations only. We randomly sample from if at the rate of P1 in the first round and start the AL framework. For the follow- ing i-th round (i 2 2), sampling with the rate of p; by the proposed method is used to obtain Sie, until the la- bel budget is exhausted. Specifically, in our experiments, p1=0.01, p;=0.01, to 10%, i.e., to label 10% of the data U. For t CBM and
Full supervision
10-th round
round
H1T4
2. We set the label budget
Full supervision
AL round
Figure 4: Qualitative examples of the fully supervised model and our method at the 5-th and the 10-th AL rounds. Dif- ferences highlighted in blue.
+ X |2,e/ 1" in IBM, we set = 0.2, À = 10 and 4. See Appendix B.1 for how these parameters are determined.
Baselines and backbones.
We compare proposed method against the following baseline meth- methods: including Least Confidence
ods: 1) Random sampling. 2) Classical uncertainty-be Sampling (LCS) (Shen et al., 2017), Maximum Entropy Sampling (MES) (Luo et al., 2013), and Margin Sampling (MS) (Culotta & McCallum, 2005). 3) Recent state-of-the-a methods: The
Core-set approach (Sener & Savarese, 2017), Fisher Kernel Self Supervision (FKSS) (Gudovskiy
7

---

Published as a conference paper at ICLR 2023
et al., 2020), Integer Programming Approach (IPA) (Mahmood et al., 2021), Loss Prediction Mod- ule (LPM) Yoo & Kweon (2019) and Temporal Output Discrepancy (TOD) (Shi et al., 2021). Please refer to Appendix B.2 for implementation details. We use three typical SGG backbones in our evalu- ation: 1) Neural Motifs backbone (MotifsNet) (Zellers et al., 2018). 2) VCTree (Tang et al., 2020b). 3) Transformer- backbone (Shi & Tang, 2020).
Evaluation metrics and modes. The first is evaluation metrics for AL. We follow the standard evaluation metric, i.e. performance under fixed labeling budget. Under this metric, the higher the performance, the better the AL framework. Then comes the evaluation modes for SGG. Following MotifsNet (Zellers et al., 2018). we use three evaluation modes: 1) Predicate classification (PredCIs). 2) Scene Graph Classification (SGCIs), and 3) Scene Graph Detection (SGDet).
Table 1:
Ablation study results. The SGG back- Table 2: Performance on open-set relationship bone used here is MetifsNet.
recognition. The SGG backbone used here is Mo- tifsNet.
PredCls R@/20/501000
SGCIs
SGDet Re/20/501100
Method
R@/20/5/1100
Method
PredCIs
SGCIs
SGDet
R0/2U50/100
R0/205500000
Ré6/20/501100
EDAL-EDI
16.0/61.563..
29.0/3..032.9
18.7/24.9/28.8
EDAL-RPG
55.5/619/63.7
1/3330
19,0/55 4/29.3
EDALALCS
50.1/62.4/70
34.3/40.4/43
14.4/20.4/25.
EDAL-IBM
56.7762.2/63.8
29/9/31.7/32.9
4/28/2
EDAL+TOD
53.2/66.0733
34.0742,444 24443
15.8/22.0/27)
EDALAFKSS
$7.3/69.4776
35 35,2/45. 1/46.2
17.1/23.4/22
EDAL-CBM
56.8/62.6/64.3
30.3/32.8/33.6
18.7/25.2/29.3
EDALALPM
55.2/68.4755.
34.7542,69553
16.3/22.7/28.1
EDAL
57.0/63.5/551
31.7/35.0/5588
21.4/27.2/30.3
EDAL
59.7/2.7780.6
37. 97.1/44.648.4
17.3/25.2/0.3
4.2
MAIN RESULTS AND ANALYSIS
In this section, we compare EDAL with other baseline methods. We report the quantitative results in Figure 3 and show some qualitative samples in Figure 4. From the above results, we have several observations: 1) Our pro- posed AL framework shows improvement with clear margin over all baseline methods, Le., EDAL has higher performance under fixed label budgets. 2) The stan- dard deviation (shadows in Figure 3) shows that our pro- posed AL framework has more stable learning process. We believe that this mainly comes from CBM and IBM, which can alleviate the data bias problem at the image and context-level to ensure that the results of each AL round can cover the unlabeled pool better, while the biased data obtained by baseline strategies can obviously mislead the model training. 3) The uncertainty- based methods (e.g.. TOD (Huang et al., 2021), FKSS (Gudovskiy et al., 2020), LCS (Shen al., 2017), etc.) perform better in the first few AL rounds but lag in the subsequent rounds, and we conjecture that the data bias problem is responsi- ble for this, which gets worse as AL progresses. This phenomenon also justifies our claim in section 3 that the hyperparameter & in IBM to alleviate data bias should be gradually increased. Thus, we argue that the exponential deduplication strategy DeTu is key our proposed AL framework. 4) The diversity-base methods (e.g.. IPA (Mahmood et al., 2021), the Core-set approach (Sener & Savarese, 2017), etc.) perform very poorly due to the large dataset in the SGG task. However, our hybrid AL framework can avoid the curse of scale by leveraging uncertainty to obtain preliminary samples. 5) Figure 4 shows that our method can output more mene-rriineemmmnnnntuu relationships, e.g,, <animal, sitting on, shelf> VS <animal, on, shelfs. We argue that this improvement stems from our uncertainty estimation method giving more attention tail categories, which we discuss in more detail in Appendix B.3. 6) Due to space limitations, we only report average performance mR 2200///0100000000 here instead of individual R@/20, R@/50 and R@/100. For the detailed results, see Appendix B.4. Furthermore, Figure 3 shows the results under the most commonly used met- rics for AL, we also report the performance under another metric, labeling budget under expected model performance, in Appendix B.5. Finally, the label budget set in this paper is 10,, but also explored the model performance under more budgets with the results presented in Appendix B.6.
Figure 5: Relationship heatmaps ob-
tained by RPG (a, b. c), simple binary statistics (d. c, f), and the full training set (g). Here we only show 10 object categories for clarity.
4.3 ABLATION STUDY
As discussed in Section and illustrated in Figure 1, our proposed AL framework consists of four key designs, iee, EDL-based uncertainty estimation module, RPG, CBM, and IBM. We
8

---

Published as a conference paper at ICLR 2023
drop the above designs one by one from our framework and report the numbers in Table 1. Note that to keep the framework running, the removed EDL-based module will be replaced by baseline uncertainty estimation method (we use LCS (Shen et al., 2017) here). The above quantitative results show that each design can contribute to our AL framework. In the fol- lowing, will further prove the effectiveness of those designs and explore their functionality.
Ablation on EDLL-based uncertainty estimation mod- ule. This subsection evaluates the robustness of our pro- posed uncertainty method. To evaluate this advantage, we report open-set recognition performances of different certainty methods in Table 2. Our evaluation procedure inspired by (Kopetzki et al., 2021), which emphasizes that a robust uncertainty method should perform well for out-of-distribution (OOD) data. The results show that the EDL-based module outperforms other methods by a clear margin. We argue these improvements are achieved by placing Dirichlet distribution on the class probabilities to avoid over-confident predictions on the open-set relationships. Due to space constraints, we give the implementation details in Appendix B.7.
Figure 6: Normalized similarity of sam- ples at different AL rounds.
Ablation on RPG. As shown in Figure 5, simple bi- nary statistics can only cover a very small number of relationships, especially the early AL rounds. Therefore, argue that in the AL setting, simple bi- nary statistics cannot complete the task of filtering the nackgroundrelaationssiss. In contrast, ages the -ocond-order neighboring to complement the bi- nary statisties. We can find that even in the early AL rounds (e.g., round I (Figure (a)), RPG can still obtain satisfactory relationship mining results, and in the third round, our method can obtain mining results that cover almost all the true relationships in the training set.
62-
60-
unt 2050000
50-
method lever-
56-
Dula (nleprout
Jabel budges
Figure 7: Performance of different deduplication strategies in IBM. The SGG backbone used here MotifsNet.
Ablation on CMB, Figure 6 shows the average similarity of the context features in each sample, which illustrates that CBM can remove duplicate information and provide a more meaningful uncertainty estimate. Furthermore, we find that different AL rounds of CBM (with the same 7) have consistent deduplication effects. This is expected as we believe that the context-level bias is only related to the data itself, which is why we set 7 to a constant value.
Ablation on IBM. We show different deduplication strategies in Figure 7, and we have the following observations. DeCv is a lot behind DeTu and DeLu, which fully supports our claim in Section 3 that different AL rounds have different uncertainty estimation capabilities and should use progressively improved deduplication strategy. At the same time, in the early AL rounds (e.g.. 2-5 rounds), DeLu can be par with DeTu, but in the later rounds, their gap widens. We argue that this reflects the fact that the linear form is no longer sufficient to keep pace with the degree of the data bias problem due to the improvement of uncertainty estimation ability.
5 CONCLUSION
Compared with simple tasks, AL for SGG suffers from a large number of complex open-set relation- ships, boocground-rllatoonshe
that perturb the uncertainty estimation, as well as data bias problems at both the image-level and context-level. For the above challenges, in this work, we have developed a hybrid Active Learning framework, EDAL, for releasing the expensive relationship annotation cost in the SGG task. We have extensively experimented on different types of SGG backbones, and the results in three SGG evaluation modes show that our proposed AL framework outperforms other state-of-the-art methods by a clear margin. EDAL consists of four key components, the EDL-based uncertainty estimation module, RPG, CBM, and IBM, and we have also explained their motivations in detail and explored their contributions through the ablation study. Finally, we discuss the limita- tions of existing metrics in Appendix B.4 and explore the potential of multimodal annotation- active learning for the SGG task in Appendix C, which belongs to future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was partially supported by National Key Research and Development Program of China No. 2021 BEB3IB0300000 the Academy of Finland under grant 331883, Infotech Project FRAGES, the National Natural Science Foundation of China under Grant 62022091 and 62201588, the National Natural Science Foundation of China under Grant 62201603, and Research Program of National University of Defense Technology under Grant 7K22-04.
REFERENCES
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:1 13:14927-14937, 2020.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. VQA: Visual question answering. Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13349-1332 2021.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alexander G Haupt- mann. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Vincent Chen, Paroma Varma, Ranjay Krishna, Michael Bernstein, Christopher Re, and Li Fei- Fei. Scene graph prediction with limited labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2580-2590, 2019.
Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In Association for the Advancement of Artificial Intelligence, volume 5, pp. 746-751, 2005.
Arushi Goel, Basura Fernando, Frank Keller, and Hakan Bilen. Not all relations are equal: Mining informative labels for scene graph generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15596-15606, 2022.
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE transactions on pattern analysis and machine intelligence, 35(12):2916 2929, 2012.
Denis Gudovskiy, Alec Hodgkinson, Takuya Yamaguchi, and Sotaro Tsukizawa. Deep active learn- ing for biased datasets via fisher kernel self-supervision. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 9041-9049, 2020.
Yuyu Guo, Lianli Gao, Xuanhan Wang. Yuxuan Hu, Xing Xu, Xu Lu, Heng Tao Shen, and Jingkuan Song. From general to specific: Informative scene graph generation via balance adjustment. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 16392, 2021.
Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou Active learning by querying informative and representative examples. Advances in neural information processing systems, 23, 2010.
Siyu Huang, Tianyang Wang, Haoyi Xiong, Jun Huan, and Dejing Dou. Semi-supervised active learning with temporal output discrepancy. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 3447-3456, 2021.
Seong Jae Hwang, Sathya N Ravi, Zirui Tao, Hyunwoo Kim, Maxwell D Collins, and Vikas Singh. Tensorize, factorize and regularize: Robust visual relationship learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1014-1023, 2018.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein. and Li Fei- Image retrieval using scene graphs. In Proceedings of the IEEE conference on com- puler vision and pattern recognition, pp. 3668-3678, 2015.
10

---

Published as a conference paper at ICLR 2023
Yoon- Yeon-Yeong Kim, Kyungwoo Song. JoonHo Jang, and Il-Chul Moon. LADA: Look-ahead data
acquisition via augmentation for deep active learning. Advances in Neural Information Processing Systems, 34:22919 19-9999922900,0 2021.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. BatchBALD: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günne- mann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-ber based models reliable? In International Conference on Machine Learning. pp. 5707-5718. PMLR, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International jourraaaanddouuraall of computer vision, 123(1):32-73, 2017.
Xiaodan Liang, Lisa Lee, and Eric Xing. Deep variation-structtre reinforcement learning for visual relationship and attribute detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 848-857, 2017.
Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph gener- ation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11546-11556, 2021.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European conference on computer vision, pp. 852-869. Springer, 2016.
Wenjie Luo, Alex Schwing, and Raquel Urtasun. Latent structured active learning. Advances in Neural Information Processing Systems, 26, 2013.
Rafid Mahmood, Sanja Fidler, and Mare T Law. Low budget active learning via wasserstein distance: An integer programming approach. arXiv preprint arXiv:2106.02000 2021.
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE international conference on computer vision, pp. 3456-3465, 2017.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint (xXiv:/70.0004000 2017.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018.
Yanyao Shen, Hyokun Yun, Zachary C Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. arXiv preprint arXiv: 2017.
Jiaxin Shi and Kaihua Tang. http://iiiiacoonnicacnannascnnne
2020.
Jing Shi, Yiwu Zhong, Ning Xu, Yin Li, and Chenliang Xu. A simple baseline for weakly-supervised scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16393-16402, 2021.
Wei Tan, Lan Du and Wray Buntine. Diversity enhanced active learning with strietly proper scoring rules. Advances in Neural Information Processing Systems, 34:10996-1000 2021.

---

Published as a conference paper at ICLR 2023
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3716-3725, 2020a.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3716-3725, 2020b.
Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative mes- Sage passing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5410-5419, 2017.
Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu, and Yujiu Yang. Probabilistic modeling of semantic ambiguity for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12527-12536, 2021.
Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Comelius Weber, Zhiyuan Liu, Stefan Wermter, and Maosong Sun. Visual distant supervision for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 35816-15826, 2021.
Keren Ye and Adriana Kovashka. Linguistic structures as weak supervision for visual scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- 2021.
nition, pp.
Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao, and Chen Change Loy- Zoom-Net: Mining deep feature interactions for visual relationship recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 322-338, 2018.
Donggeun Yoo and In So Kweon. Learning loss for active learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 93-102, 2019.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural Motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5831-5840, 2018
Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, et al. Scene graph generation: A compre- hensive survey. arXiv preprint arXiv:2201. 00443,, 2022.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7W: Grounded question answer- ing in images. In Proceedings of the ICEE conference on computer vision and pattern recognition, pp. 10995, 2016.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX TO METHOD
A.I PESUDO-CODE FOR THE EDAL
Without loss of generality, we show the sampling process for t-th AL round (t 2 2). Assume that the unlabeled data of the t-th AL round Ur {I,, 12, ..,, and that the i-th image I contain [7:] objects and thus [:] x [[: - relationship triplets {< _ of >; < o of >,
[/. -, 1:1-1 }}.
Algorithm Evidential Uncertainty and Diversity Guided Deep Active Learning (EDAL)
Input: 1,14 = {1,, I2, .. -RuI) T, 2,,,,,, X,
I/ inputs of (-th AL round
Output: St.F.
1: for = to [4,] do
2: U1, {U<ot- U Unot <ot_o
14-113
// estimate the uncertainty of
<o;
each triplet using evidential uncertainty estimation (Section 3.1.1)
3:
WPGG R RPG
RPG
RPG
a.4. 2.4.1,441111 141113
calculate the RPG weight for each triple
in Uz = (U<ot_of>>
4:
w
U of
RPG
<ot _of x
u 0..0
RPG
,U dft 1>
x
w
RPG it 1
// refine the evidential uncertainty (Section 3.1.2)
5: end for
6: Uu. {U Ui.....,UU Fuun
I/ reliable uncertainty obtained by Section 3.1
7: for = 1 [4 do
8: D1.c 12 =ot " co oo -
I/ calculate the density at 9: remove the [[7,] x (1 - 1) x r] context with the smallest density, denoted as Dj,00
context-level (Section 3.2)
= o.............. Di.............. ci
alleviate the context-level bias
10:
get the uncertainty of each context in the Dí.c according to Uu., denoted as UI,Cc U1,C
get the uncertainty of I, U", - PPinel U/
11:
12: end for
Un = U";""" ", ",."",)) Drjan)
// uncertainty obtained by Context Blocking Module 3.2. Compared with Uu., Uig does not contain context-level bias
14: = (
15: select &, samples with the highest uncertainty, denoted as S. S = {11, 12...1.....
16: for = to l, do
17: DI..1 = (D) ... ..,Di.)
I/ calculate the density at image-level (Section 3.2)
18: end for
19: Ds.FF = Di.....,i.!.
20: remove the E2 - le samples with the smallest density, denoted as St,, level bias. Se,e, the final sampling result of ofttth AL round
// alleviate the image-
A.2 LOSS FUNCTION
As shown in Table 3, the vanilla EDL loss function performs very poorly on the SGG task. We think this is mainly because the regularization term (2) of Equation 1 is insufficient. We prove it by reporting the performance Equation 1 without term (2), and the results validate our point that term (2) can only bring about trivial improvement. Instead, loss function can provide the model with better supervision without compromising the estimation of uncertainty. We have discussed this fully in Section 3.
13

---

Published as a conference paper at ICLR 2023
Table 3: Model performance with different loss functions.
PredCls R@20/50/100
SGCIs
SGDet
method
R @ 20/50/1000
R @ 20/50/100
term (1) in Equation 1
36.1/60.7/62.
28. 28.7/31.3/32.4
17.3/23.5/27.5
Equation
55.4/60.9/62.6
29.5/34.9/32.5
18.6/24.1/27.5
Equation (ours)
57.0/63.5/65.1
31.7/35.0/35.8
21.4/27.2/30.3
A.3 DATA BIAS PROBLEM IN ACTIVE LEARNING
Active learning, especially those based on uncertainty estimation, is prone to data bias problems. This has been discussed in much of the existing literature (Huang et al., 2021; Mahmood et al., 2021; Sener & Savarese, 2017; Tan et al., 2021). Here we further explain how active learning produces biased sampling.
Let Q(2.g) and P(a.y) denote the distribution the unlabeled data pool and the selected data ob- tained by an AL method, and suppose their densities are q(z, y) = q(y x))((>) and p(r, y) = ply t))(z), respectively. We use H(h H) to represent the optimal sampling for the original distribu-
tion H under the condition of a given sampling rate, where h obeys the distribution H. Based on this definition, H((2, y) P(a.)) can be calculated as:
H((z, v) ~ (uaa)) If Av [))(()) In(a(y) =)q(x))d.d.
(4)
H((x, y) P(aw) = If g(y <))((>) (n(p x)p(x))d.d,.
(5)
We then use KL divergence DKI(Q(Z.)) || P(a.)) to describe the extent to which P(a,v) covers Q (zx.)):
DKI(Q(Z.)) P(.,)) = H((x,)) v) ~ P(ax))) - H((z, y) Q(.v))
= =Iʃ alv (x)((z) In g(y x)g(ar) dydly. ply x)p(t)
(6)
Therefore, we can obtain the optimal active learning query function QAL by minimizing (KI(Q(rM)) P(xa))::
QAL arg migminnK (I(((..))) Il P(ew))
P(a.))
However, from an example shown in Figure (d), we can see that P(x,p) is biased towards partial categories in practice. Assuming that the optimal sampling of training data under given conditions QAI- Obviously, in P(w.y)- some high uncertainty data Q (H((x, y) ~ P(t.,)) are not queried by QAL. but some low uncertainty data H((x, y) ~ ((x.))) Maa are selected instead. This unreason- able sampling is exactly the data bias problem in active learning.
A.4 DE-BIAS UNIT
In the Image Blocking Module (IBM), an exponential formulation De-bias Temperature Units (DeTu) L, = lg + A [er] 1" is adopted rather than its counterparts of constant form (De-bias Constant value, DeCv) l, & Aı [ee] linear form (De-bias Linear Units, DeLu) & le + X2 [4, (see Section 3.2).
In addition the quantitative result from Figure 7, here we further explain the intuition. The uncertainty-base methods suppose that data with higher uncertainty contains more valuable in- formation, while the biversity-based methods believe that data with lower density obtain richer in- formation. Based on these two viewpoints, we define the information of the data as E(X) to quantify the value of samples, F(X) = Ux/ (Dx + 8), where Ux and Dx are the uncertainty and
14

---

Published as a conference paper at ICLR 2023
density of X, respectively and 5 is an infinitesimal value to avoid the numerical issue. Thus, the information gain of IBM is:
E(S,,.) E(( (Do.,, +6)U 5, -DDs,,, + +5)Us...
(8)
(Ds..) + x (Dst, + +0)
where Sye, is the data sampled without IBM, and St,t, is the results obtained by our proposed AL framework with IBM. We assume two extreme but representative scenarios to support our choice of DeTu. (1) Assume the current model is incertainty-unrelibbee i.e., it cannot provide any valid uncertainty estimation. In this case, the result of the uncertainty method equivalent to random sampling. this case, the IBM will be indistinctive or even play negative role i.e., E(S E(S & O. (2) Assume the current model is uncertainty- uncertainty estimations with the confidence of 1. In this case, considering the fact that uncertainty- based methods suffer from data bias problems and E(S E(S...) 2 0, this indicates that the IBM will always bring a positive effect.
i.e., provides reasonable
The above facts suggest that different blocking strategies should be used in different AL rounds be- cause the uncertainty estimation ability of the model is gradually improved, so the constant strategy (DeCv) can not meet our requirements. At the same time, for the linear strategy (DeLuu, we argue that the DeLu with large slope may have an adverse impact in the early AL rounds due to the unreliable uncertainty estimation, while the DeLu with small slope may not meet the demand of data removal in the later AL rounds due to serious data bias problem. Therefore, we believe that IBM needs a nonlinear data removal strategy, and the results in Figure 7 can also prove our point.
B APPENDIX TO EXPERIMENT
B.1 HYPERPARAMETERS SETTING
For y and À in IBM. Motivated by the fact that data bias problem will become more serious as AL progresses, we argue and experimentally prove that the deduplication ratio should be gradually increased during training. The exponential deduplication function is a natural choice that meets the above expectations. The value of À is set according 77, and the basic principle is to ensure a suitable deduplication amount. In our paper, we keep the deduplication amount at approximately less than 10% via A. We show the exponential functions with different 7 and À in Figure 8. We argue that the should be nearly flat in the early AL rounds and should grow rapidly in the later rounds to combat the increasingly serious data bias problem. As tradeoff, we r and thus A 10-5.
However, for 7 in CBM, we argue that the data bias problem at context-level is only related to the constant value. In Figure 9, show model performance
input data itself and thus should be with different r. Inspired by this experiment, we set J = 0.2 in this paper.
1004, m
0010-
A 10
08
A- 1079,
06
0.04
02
0.00
Figure 8: Deduplication function with different r) and A.
15

---

Published as a conference paper at ICLR 2023
ProdC3s
64
62-
60
15
55-
52
50
2
10
label budget (%)
Figure 9: Model performance with different T.
full dataset
2-nd AL round
3-rd AL round
4-th AL round
5-th AL round
sadeoppie 1005
.c0
11.9
onto q
6-th AL round
7-th AL round
8-th AL round
9-th AL round
10-th AL round
Figure 10: The relationship distributions of full dataset (trainset of VG150) and sampling results of our AL framework.
B.2 COMPARISON DETAILS OF ACTIVE LEARNING BASELINES
To validate our proposed method, we compare a large number of baseline active learning methods, including classical imcertainty-based methods and recent state-of-the-ee methods. Specifically, the comparison details these baselines are as follows. The Core-set approach (Sener & Savarese, 2017) is diversity-based AL method, and we follow the training tricks and hyperparameters in the original paper. Fisher Kernel Self Supervision (FKSS) (Gudovskiy et al., 2020) proposes a low- complexity feature density matching method and calculates the uncertainty of unlabeled data based on it. Note, for fairness. we use the complete unlabeled pool when comparing this method rather than using only part of the pool in the original paper to create artificial data. Integer Programming Approach (IPA) (Mahmood et al., 2021) minimizes the discrete Wasserstein distance in feature space from the unlabeled pool select the core set. Temporal Output Discrepancy (TOD) (Shi al., 2021) calculates the uncertainty on unlabeled data based on losses at different training stages, and it also incorporates the semi-supervised method to improve performance. To keep focus and follow the AL setup, only compare the uncertainty method proposed in TOD.
B.3 ANALYSIS ON LONG-TAILED DISTRIBUTION
Our proposed EDL-based method can provide reliable uncertainty estimates, and in this subsection we show that it also leads to an important product, i.e., alleviating the long-tail distribution problem. Our uncertainty estimation method pays more attention to tail categories due to these relationships and performs poorly in the head distributions of the full dataset and sampling results of our AL framework. We can that the proportion of head categories in the full data is 78%. However, that in our method does not exceed 50%. We think this improvement is very important for the SGG task because it encourages the model
model. Figure 10 shows the relationship
16

---

Published as a conference paper at ICLR 2023
to predict more fine- line-grineed/meninngee relationships, e.g., predict sitting on instead of on. Besides, this advantage has the potential to make our method approach or even exceed fully supervised model performance with less label cost, which we have analyzed in Appendix B.6.
Although our method can alleviate the long-tailed distribution problem, it is worth noting that the head categories in SGG are often common relationships that will appear in various scenes. There- fore, although our method gives priority to tail categories, many head category relationships (on, has, in, etc.) that appear in tail category scenarios are also sampled. As a result, we believe our method will perform better in tasks with a low fusion of head and tail categories.
Pooto
27 10.00
(a) R @220
(b) R @@0
(c) R @ 100
(d) mR /20/50/100
Figure 11: Quantitative evaluation on the VG150 dataset using MotifsNet (Zellers et al., 2018).
B.4 DETAILED EXPERIMENTAL RESULTS
In this section, we provide more details on experimental results for benchmarking purposes with further analysis. In Figure 3, we have shown the average performance of R /2025/00000 due to space limitations. Here we report more detailed quantitative results with different backbones of Motif&Net, VCTree, and Transformer Figure 11, Figure 12, and Figure 13, respectively.
From these figures, we have the following observations and inspirations: 1) In both challenging SGCIs mode and SGDet evaluation modes, the performance of our proposed EDAL is closer to that of using full supervision than other methods. In addition, we great potential of active learning in SGG because all baseline methods achieve competitive performance given 10% of training labels, which further implies that discovering the most valuable data from long-taile tailed imbalanced SGG training data is worth further investigation. This gives us the confidence to continue the topic of this paper in the future. 2) The overall trend of performance under R R@51 and R@ 100 are very similar, and we suspect this is mainly caused by the lack of representativeness of the recall metric (R @ K) with a larger value of K. especially when K is much higher than the actual number of relationships within the scene. To verify our assumption, we compute the histogram of the number of ground truth relationships and all possible relationships per image in the test set of VG150, shown in Figure 14. Where all possible relationships refer to the relationship between all object pairs. Specifically, N objects have (NN 1) possible relationships. We can see that the histogram of relationship statistics can support the above point. This observation shows that to accurately validate the SGG backbone, it is necessary to set an appropriate recall parameter according to the relationship distribution in the scene. 3) Metric R@ K is
e.g., under R @ 50 there is performance difference whether the successfully retrieved relation lies the 1-st or 50-th of the predictions. Moreover, R@ K is K K-dependent, i.e,, inappropriate K may affect the model evaluation. Thus, we aim to design
17

---

Published as a conference paper at ICLR 2023
K-independent and order-sensitive evaluation metric for SGG and which will be our pursuit in the immediate future.
(a) 20
(b) 50
(c) R @ 100
(d) mR @ 2/20/50/100
Figure 12: Quantitative evaluation the VG150 dataset using VCTree (Tang al., 2020b).
Mar
(a) R 20
(b) R 5550
R 100
(d) mR /20/50/100
Figure 13: Quantitative evaluation on the VG150 dataset using Transformer (Shi & Tang, 2020).
B.5 LABELING BUDGET UNDER EXPECTED MODEL PERFORMANCE
Figure shows the evaluation results under the most commonly used metric for active learning, i.e., performance under fixed labeling budget. In Table 4, Table 5, and Table 6, report the performance under another active learning metric, i.e., labeling budget under expected model performance. Under this metric, the less the labeling cost, the better the sampling of active learning approaches.
18

---

Published as a conference paper at ICLR 2023
10+
50H
(a)
(b)
Figure 14: The number distribution of ground truth relationships (a) and all possible relationships
(b). The distribution shown here is counted come the test set of VG150.
see that EDAL outperforms other baseline methods, which means that our proposed active learning framework requires less annotation cost when pursu- ing fixed model performance. In addition, advantage of EDAL: there will be round gap between EDAL and the baseline methods. For ex- ample, when evaluated under the backbone of MotifsNet (Zellers et al., 2018), our proposed AL framework uses 7 rounds while the baselines take minimum of 8 rounds to meet the expected per- formance (the expected performance here is that the @@20 reaches 30 in SGCIs mode). The round gap shows that EDAL can reduce the labeling cost and also save training resources.
From Table 4, Table 5, and Table 6,
experimental results also reveal another important
Table 4: Labeling budget under expected model performance. The SGG backbone used here is MotifsNet (Zellers et al., 2018).
Predicale classification (PredCls)
Scenne Graph Classification (SGCIs)
Sceeee Graph Detection (SGDet)
R@020
R
R@020
R
100
R
R@:50
100
50
55
55
60
60
65
25
30
28
30
30
35
15
18
20
25
25
28
random
5.67
10.0
5,33
9.00
8.00
10.0
3.00
8.33
3,00
5.00
4.67
10.0
3.67
9.00
4.67
9.00
5,00
8.00
Least
6.00
10.0
5.67
9.67
8.33
10.0
3,00
9.00
3.00
4.67
4.33
10.0
4.33
8.33
4.00
10.0
5.00
7.33
Entropy
6.33
10.0
6.00
9.33
8.00
10.0
3.33
8.67
4.00
5.00
4.67
10.0
4.00
7.67
4.33
10.0
5,00
7.33
Margin
6.67
10
6.00
9.00
8.00
10.0
3.67
8.33
3.33
4.67
4.67
10.0
3.67
8.33
4.00
10.0
5.33
7.67
IPA
4.67
8.67
4,33
7.67
6.00
10.0
3,00
7.67
3.67
4.67
4,00
10.0
4.33
7.67
4.00
9.33
4.67
TOD
4.00
7.67
4,00
7.00
5.33
10.0
2.67
7.33
3.00
4.67
4.00
10.0
3.00
7.00
3.67
9.00
4.00
7.33
FKSS
4.67
7.33
4.33
6.33
5.33
10.0
2.67
7.67
3,00
4.67
4.33
10.0
3.67
6.67
8.33
4.33
7.00
The Core Set
5.67
8.67
5.33
8.67
7.00
10.0
3.00
8.67
3.00
4.67
4.67
10.0
4.33
7.67
4.00
9.00
5.00
8.33
EDAL
3:67
6.67
3.67
5.67
5.00
9.67
2.67
6:33
3,00
4.33
4,00
8.33
3.67
7.33
4.00
7.67
4,00
7.00
Table 5: Labeling budget under expected model performance. The SGG backbone used here is VCTree (Tang et al., 2020b).
Predicate classification (PreACls)
Scene Graph Classification (SGCIs)
Gnaph Desection (SGDet)
R8:20
R@:00
100
R:B:00
R@:50
RB (00
R@20
R@50
R@100
50.00
55,00
55.00
60.00
60:00
65.00
30.00
35.00
30,00
35.00
35.00
38.00
15.00
18.00
20.00
25.00
25:00
25.00
random
4.00
9.00
4.00
8.33
6,00
10.00
4.33
9.00
3.67
6.67
6.00
9.67
3.67
9.00
4.67
9.00
5.00
3.00
Least
9.33
9.67
4.00
2.33
5.67
10:00
8.33
7.33
Entropy
4.00
9.00
4.33
9.00
7,00
10.00
4.33
8.67
3,00
6.33
6.00
9.00
4,00
7.67
4,35
8.67
5.00
7.35
Margin
9,00
8.67
4.00
3.00
6.00
9.33
3,67
8.33
9.33
7.67
IPA
4.00
7,33
4.00
7.00
6.00
10.00
4.33
8.67
3,00
6.33
6.00
3.00
4.33
7.67
4,00
9.33
4.67
7.00
TOD
7.33
6.67
2.33
5,00
7.33
PKSS
4.33
6,33
4,00
6.35
5.67
10.00
4.33
8.00
2,33
6:00
5.67
3.00
3.67
6.67
3.67
8,33
4.33
7,00
8.33
7.67
6.00
9,00
EDAL
4.00
6,33
3.67
6.00
5.00
9.00
3.67
7.67
2,00
6.00
5.00
7.00
3,67
7.33
4,00
7.67
4.00
7.00
B.6 RESULTS WITH MORE LABEL BUDGET
In our main paper, set the label budget to be 10%, i.e., label 10% of the data in the unlabeled pool. In this section, we explore the performance of active learning under more label budgets. Specifically, increase the budget to 1 13%, and the results reported in Figure 15.. From these experimental
19

---

Published as a conference paper at ICLR 2023
results, find that in Predicate classification (PredCIs) mode, active learning can only obtain very trivial gains when increasing the budget. However, in Scene Graph Classification (SGCIs) mode and Scene Graph Detection (SGDet) mode, active learning can continue to benefit from the increased budget and even over the full supervised performance. This promising result is exciting for the SGG task, which means that the active learning method can not only save considerable labeling costs but also further improve the model performance. We think this is mainly because the object information used by PredCIs is ground-truth labels, while those used by SGCIs and SGDet are obtained by a biased detector, i.e., the detector is trained on data with the severe long-tailed tailed distribution. In other words, both SGCIs and SGDet try to predict the relationship between object pairs obtained by the bi- ased detector. It is expected that the long-tailed distribution at the relationship-lvel further amplifies distribu- the detector bias. However, active learning has a certain mitigation effect on long-taii tions, which we believe is the reason why active learning methods can outperform the full supervised performance in some cases. In addition, two debiasing modules, CBM and IBM, are included in our active learning framework, which also plays key role in improving model performance in SGCIs and SGDet modes.
Further, continue to increase the label budget to observe performance trends. Specifically, increase the label budget to 20%, and the results are shown in Figure 16. The performance of EDAL begins to converge when the labeling budget reaches certain value and increasing the labeling budget after that cannot improve performance. This inspires us that it is imperative to set a suitable stopping point for active learning.
Table 6: Labeling budget under expected model performance. The SGG backbone used here is Transformer-bart based backbone (Shi & Tang, 2020).
Preficate classification (PredCik
Scene Graph Classifictiio (SGCk)
Scene Gnaph Detection (SGDet)
R@:00
R@:50
R@100
R@:0
R@50
Re
R-020
R@50
R 100
50,00
35.00
60.00
65.00
60:00
65.00
30.00
33.00
30.00
35.00
35.00
38.00
15.00
20.00
22.00
25.00
25.00
30,00
random
3.33
6,00
6.33
10.00
5.00
10.00
3.33
6.00
3.00
7.33
7.33
7.44
3.33
8.67
4.67
7.33
3.67
9.67
Least
2.67
5.33
5:33
10.00
4.33
10.00
2.67
5.33
2.33
7.00
6.00
6:00
2.00
5.33
3.67
7.00
3.00
9.00
Entropy
2.33
6.67
6.00
10.00
4.67
8.33
3.33
6.33
3.33
7.00
6.67
6.89
2.33
8.33
4,00
6.67
3.00
9.00
Margin
3.00
5.67
5.67
10.00
4.67
8.33
4.00
6.00
3.33
7.00
6.67
6.89
2.67
7.67
4.00
6.67
3.00
9.00
IPA
3,00
5.67
5.33
10.00
4.33
7.67
3.67
6.00
3.33
7.00
6.00
6:00
266
7.67
4,00
7,33
3.00
8.33
TOD
2.00
5.33
5.00
10.00
4.00
8.00
2.67
5.67
2,00
7.00
6.00
6:00
2.33
7.00
3.33
6.33
2.00
5.00
PKSS
2,00
5,00
5:00
10.00
4,00
7.67
3.67
5.67
3,00
7.00
6.00
6:00
2.00
8.00
4.35
6.33
3.00
8.00
The Core Set
2.33
5.67
5.33
10.00
4,00
8.00
3.33
6.00
3,00
7.00
6.00
6.00
2.00
7.00
3.67
6.67
2.67
3.00
RDAL
2.35
5.00
4.67
9.00
3.33
7.00
2.67
5.00
2,00
6.00
6.00
6.00
2.00
7.00
3.35
6.33
2.00
8.00
B.7 IMPLEMENTATION DETAILS OPEN-SET RELATIONSIIPP RECOGNITION
VG150, a subset of the VG dataset, contains 150 object classes and 50 relationship classes. There- fore, to V G150, there will be many open-set relationships in the VG dataset, i.e., those labeled as foreground in VG but background in VG150. 50. Our statistical result shows that there are about 68k open-set relationships in the test of VG150. Similar to current work (Zellers et al., 2018; Tang et al., 2020a; Lu et al., 2016) evaluating on VG150 (Xu et al., 2017), to compare the effectiveness of different uncertainty estimation methods on open-set relationship recognition without contamina- tion from data sampling, we conduct evaluations on the Visual Genome (VG) (Krishna et al., 2017) test set. Specifically, we first leverage the full training set of VG150 to train the model and use different strategies to estimate the relationship uncertainty. Then, we then classify the relationships based on the uncertainty, i.e. open-set relationships and non-open-set relationships, according to a fixed threshold (0.5 in our paper). We then classify the relationships based on the uncertainty, ie. open-set relationships and non-open-set relationships, according to a fixed threshold (0.5 in our experiment). Finally, we rank the relationships for each scene according to uncertainty and com- pute the recall of open-set among the top-20/50/100 relationships. The performance on open-set relationship recognition is shown in Table 2.
20

---

Published as a conference paper at ICLR 2023
1000 290
1090
BCCE
(a) MotifsNet
(b) VCTree
(c) Transformer
Figure 15: Active Learning performance of EDAL and baseline methods. mR //R2200000100 repre- sents the average performance of R @ 20, R@50, 50, and R@ 100. We repeat each experiment three times and report the mean (solid line) and standard deviation (shadow). Unlike the label budget set (10%) in our main paper, explore the model performance of more budgets here.
(a) MotifsNet
(b) VCTree
(c) Transformer
Figure 16: Active Learning performance of EDAL under 20% label budget.
C APPENDIX TO CONCLUSION
The annotation expenses of SGG can be decoupled into relationship-level annotation and object- level annotations, of which the former the most laborious. addition, thanks to the existing large-scale datasets of object detection, our primary goal in this paper focus reducing relation- ship annotations by leveraging the proposed active learning system EDAL. However, we found that the object-level annotations can be further decoupled into bounding box annotations and category annotations (see Figure 17). To further improve the practicability of active learning in SGG, it is worth more investigation to explore the joint active learning framework of both object-level and relationship-leww annotations. We believe the correlation multi-modal annotations would bring further benefits to improve the labeling efficiency of EDAL, which serves future work.
21

---

Published as a conference paper at ICLR 2023
SGG annotations
righd, [221, 188, 483, 491], has, (hair, [309, 181, 360, 221]
<ggir, (221. 188, 483, 491D. has, (mouth, [313, 278, 346, 294
<(horse, [10, 35, 191, 495J), has, (leg. [114, 359, 132, 445Jj=
Bounding box annotations [231, 188, 483, 491] [00, 181, 221)
Relationship-leve annotations
Relationship-leve annotations
igirl, hos, hair- "girl, has, mouth -horse, has, leg?
(girl, [221, 188, 483, 491]]
Category annotations
(hair, [309, 360, 221))
(mouth, [313, 278, 346, 294))
girl, hair, leg,
Figure 17: Decoupling of SGG annotations.
22


Reviews:
[{"review_id": "zV0jJ0ojQk", "reviewer": "Reviewer_pmut", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As a structured representation, the scene graph bridges computer vision and natural language processing, and the scene graph generation task is vital for the CV community. However, The annotation process of the scene graph is very complex including bounding boxes, labels, and relationships, so it is necessary to explore how to ensure the robustness of the scene graph model with few annotations. The authors propose a new method based on active learning, which trains a robust SGG model with fewer data by fully exploiting informative samples in the dataset.", "summary_of_the_review": "Based on the above considerations, I believe that this paper has some novelty and will have some positive impact on the field. In the rebuttal stage, the authors should solve my main concerns to polish the final version.", "strength_and_weaknesses": "The positive aspects of the paper include:\n\n1. Using the active learning method to save the annotation cost is a novel direction in scene graph generation and is important for further SGG study.\n\n2. The designs of the proposed modules, i.e., EDL, RPG, CBM, and IBM, are convincing and fully consider the inherent problems of SGG, such as data bias and sparsity of the dataset.\n\n3. The final performance of the proposed method is good, which approaches the performance of a fully supervised SGG model with only about 10% annotation cost for training the SGG model. The abundant ablation studies and detailed comparisons verified the effectiveness of the proposed modules.\n\nThere are some main concerns:\n\n1. As stated in the abstract, the authors aim to reduce the annotation cost. However, the proposed method only reduces the training set for training models, and the method also needs a large labeled dataset to sample an informative training set. Does this approach really save annotation costs throughout data collecting and model training processes?\n\n2. In section 3.1.1, the authors mentioned that “the uncertainty of xi can be estimated via evidential uncertainty ui = k/Si with a maximum value of 1.” Why is the maximum value 1? Is eij greater than or equal to 0? The definition of eij is unclear.\n\n3. The authors should explain the qualitative results in Figure 4 in Section 4.2. What the figure illustrates? \n\nSome minor issues:\n\n1. Inconsistent symbol in Eq.(1) and following text. “S” (standardized) in Eq.(1) and “S” (italic) in text.\n\n2. Inconsistent information in the caption and the image in Figure 4, “3-rd and 10-th AL rounds” in the caption and “5-th/10-th AL round” in the image.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h2X6HtWqOG", "reviewer": "Reviewer_QrN3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an evidential uncertainty and diversity guided deep active learning framework for the scene graph generation task. In particular, uncertainty is estimated by coupling evidential deep learning (EDL) and global relationship mining. Also, a context blocking module (CBM) and image blocking module (IBM) are designed to explore the diversity existing with context-level and image-level bias. The experiments on the VG150 dataset seem to validate the effectiveness of the proposed method.", "summary_of_the_review": "Based on the above statements, I would like to weakly accept the paper at the initial stage. After carefully reading the authors' response, I am happy to see most of my concerns have been well addressed and therefore would like to keep the initial rating.\n", "strength_and_weaknesses": "#### Strengths\n- Overall the paper is well written and easy to follow.\n- It is interesting to see the authors apply active learning in the task of scene graph generation, which is not well explored so far.\n\n#### Weaknesses\n- Is the proposed method able to handle the label noise? Usually we cannot guarantee that the human annotations are 100% accurate and therefore a solid active learning method should also be able to handle the label noise to some extent.\n- Are the orders of the uncertainty based module and the diversity based module exchangeable?\n- Is the proposed task-specific active learning method able to outperform the task-agnostic active learning works like: \n\n[1] Yoo, Donggeun, and In So Kweon. \"Learning loss for active learning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PX3ach0xoBO", "reviewer": "Reviewer_ffZd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method, EDAL, for scene graph generation with active learning. The paper combines an uncertainty-based and diversity-based active learning. The former utilizes evidential deep learning to estimate the relationship uncertainties and the diversity-based sampling is driven by image-level biases. The main idea is to utilize the evidential probability based cross entropy loss as a regularizer for relationship classification. This evidential probability is the second-order probability derived from the parameterized Dirichlet distribution. Additionally, diversity-based on context and image-level bias is incorporated.  ", "summary_of_the_review": "The paper develops an hybrid active learning framework for learning SGG. The hybrid method utilizes both the diversity-based and EDL for uncertainty-based AL. One of the major contributions is modifying the regularization for the relationship recognition task using EDL. The results show their method archives higher mean Recall for PredCls, SGCls and SGDet, without compromising the performance of head classes as measured by Recall metrics within a relationship labeling budget of 10%. ", "strength_and_weaknesses": "The paper aims to develop a hybrid approach utilizing both the uncertainty-based and diversity-based active learning for scene graph generation tasks. The paper utilizes evidential deep learning for its known property of being reliable specially in open-set problems. \nThey took inspiration from EDL originally aimed for open-set action recognition and modified it for relationship recognition. Essentially that is utilizing the evidential probability based cross entropy loss as a regularizer. This evidential probability is the second-order probability derived from the parameterized Dirichlet distribution. The hypothesis is that the cross-entropy loss based on evidential-probability prevents the overconfident predictions in open-set relationships without affecting the uncertainty estimates. This is an intended behavior specially because of sparse relationship triplet labels present in a scene. The experimental designs including baselines corresponding different sampling strategies are convincing. Fig 3 depicting mean Recall values with varying labeling budget is a valuable addition in this paper. It also shows that the improvement of mR values don't come at the cost of Recall values. \n\nFrom the visualizations, it also looks like the method generates more fine-grained (in the case of VG, they are tail classes) relations : e.g. On vs sitting on in Fig 4. I am curious about how the relative performance trends are for a labeling budget over 10% such as 20%-25%.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DTDH31yNhRj", "reviewer": "Reviewer_Jsof", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an active learning-base model for scene graph generation, in order to reduce the prohibitively expensive annotation cost.", "summary_of_the_review": "Regarding the limited contribution and unclear details of this paper, I recommed to weakly reject the paper.", "strength_and_weaknesses": "Strength:\n(1) Adopting evidential uncertainty and diversity guided deep active learning can alleviate the issue of open-set relationships sounds.\n(2) The proposed method can save human annotation costs by approaching the performance of the fully supervised model seems promising.\n\n\nWeakness:\n(1) How to evaluate the robustness of proposed uncertainty method? Please offer more details.\n(2) How to handle the long-tail problem with the proposed model? Please examine the effect of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster