Weakly Supervised Scene Graph Grounding
Authors: Yizhou Zhang
Zhaoheng Zheng
Yan Liu
Venue:  (2021)
Keywords: Weakly Supervised Learning
Scene Graph Grounding
Visual Relation
Computer Vision

Abstract:  Recent researches have achieved substantial advances in learning structured representations from images. However, current methods rely heavily on the annotated mapping between the nodes of scene graphs and object bounding boxes inside images. Here, we explore the problem of learning the mapping between scene graph nodes and visual objects under weak supervision. Our proposed method learns a metric among visual objects and scene graph nodes by incorporating information from both object features and relational features. Extensive experiments on Visual Genome (VG) and Visual Relation Detection (VRD) datasets verify that our model post an improvement on scene graph grounding task over current state-of-the-art approaches. Further experiments on scene graph parsing task verify the grounding found by our model can reinforce the performance of the existing method. 

Under review as a conference paper at ICLR 2021
WEAKLY SUPERVISED SCENE GRAPH GROUNDING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent researches have achieved substantial advances in learning structured repre- sentations from images. However, current methods rely heavily on the annotated mapping between the nodes of scene graphs and object bounding boxes inside im- ages. Here, we explore the problem of learning the mapping between scene graph nodes and visual objects under weak supervision. Our proposed method learns a metric among visual objects and scene graph nodes by incorporating information from both object features and relational features. Extensive experiments on Visual Genome (VG) and Visual Relation Detection (VRD) datasets verify that our model post an improvement on scene graph grounding task over current state-of-the approaches. Further experiments scene graph parsing task verify the grounding found by our model can reinforce the performance of the existing method.
INTRODUCTION
Motivated by various needs, researchers have designed multiple representations to describe visual contents. More specifically, object bounding boxes localize the objects inside image while scene graphs represent object-wiss interactions. Ideally, each bounding box should corresponds to node the scene graph. However, in many cases, such node-object level correspondences are es- tablished, particularly when the information of scene graphs come from non-visual inputs, such as image captions (Wang et al. 2018b), knowledge graph (Zareian et al. 2020b) and commonsense base (Shi et al. 2019).
The lack of node-object level mapping in data results in constraints on various
learning tasks, e.g., scene graph parsing (Xu et al. 2017, Zhang et al., 2017b). VQA (Ghosh et al. 2019) and image captioning (Yang et al.) 2019). the mapping can be learned without extra annota- tions, a comprchensive view of image will be created and benefit number of downstream tasks. Therefore, in this paper, we focus on grounding scene graph nodes to visual objects under weak supervision, where the node-object correspondences not annotated
during training phase.
Although the scene graph grounding problem can benefit plentiful downstream tasks, it has been barely studied. Unlike other weakly supervised learning tasks, which focus single label space (Dietterich et al. 1997, Waan et al., (2188)), the scene graph grounding problem involved with two label spaces: object categories and relation types, which are disjoint but dependent. More specifically, visual relations are highly correlated with visual objects. As result, desirable model should correctly handle the interaction among object categories and visual relations instead of sim- ply learning them independently. Therefore, most of the well-studied weakly supervised learning methods are not suitable for the learning on scene graphs.
Among the few relevant efforts recently spent on this task, Zareian et al. achieve impressive results. They notice the grounding problem when they are trying to handle weakly supervised scene graph parsing. They treat it as a side challenge in weakly supervised learning and propose to tackle it by jointly learning the node-object mapping and visual relation parser under weak supervision. In their method, parser capturing the interaction of relation feature and object feature represents the image as bounding box graph. Then they align such bounding box graph with the scene graph to construct the correspondence between visual regions and scene graph nodes. The mapping found by alignment algorithm further utilized in optimizing the scene graph parser.
However, the graph aligning process results in one core limitation of their method. Given the fact that the graph-matching problem is NP-hard, they must take the trade-off between efficiency and accuracy into consideration. Furthermore, to enable weakly supervised learning, in training stage

---

Under review as a conference paper at ICLR 2021
the authors directly generate supervision signal for the parser from the mapping output by the model itself. And as a consequence, an inaccurate graph alignment in initial training stage could mislead the parser, leading to a performance drop.
In this paper, we propose to formulate the mapping problem as a minimum match problem on bipar- tite graph instead of graph alignment. More specifically, measure the similarity of node-object pair, we propose to learn a cost function incorporating information from both object classes and mutual relations. The lack of node-object level mapping brings challenges the learning process. Therefore, we design an image-graph level distance inspired by Wasserstein 1985]. and propose to train our model by minimizing this distance on corresponding image-grapy -aageggraph pairs.
Our contributions are:
This is the first paper that considers weakly supervised scene graph grounding inde- pendent task and explores it. We argue that weakly supervised scene graph grounding is vital to multi-modal learning and deserves broad attention.
We propose novel framework that can bridge visual objects and scene graph nodes by cap- turing both object classes and relational information and searching the best match among the objects and nodes. By designing image- gaaseggrahhhue level distance, tackle the challenge brought by the lack of supervision.
Empirical results indicate that our grounding method outperforms existing mapping meth- ods. Moreover, also verify the value of our model in enhancing the performance of existing SOTA models on weakly supervised scene graph parsing, an important and bene- ficial downstream task in multi-modal learning.
RELATED WORK
Visual Relation Detection (VRD): The goal of VRD to further understand an image by detect-
ing the relations between detected object pairs. Powered by datasets consisting of visual relation annotations, e.g.. Visual Relation Detection (Lu et al. 2016) and Visual Genome (VG) (Krishna ett al. 2017). substantial advancements are made by recent researches. Several convolutional neural network (CNN) architectures (Zhang al.] 2017b Yin et al. 2018: Li et al.) 2017 Inayoshi et al., 2020) are tailored for recognizing visual re relations. while some other researches (Dai et al. 2017 Zhang et al. 76, Mi & Chen 2020) focus on post-CNN feature inference. Apart from the vision domain, some researchers (Lu et al. [016 Yu et al. 2017 Zhang et al. (2017a) pay their attention to the language domain, employing linguistic priors for better predicate predictions. Recent researches
Baldassarre al. 2020
Peyre al. 2017 explore the problem with weak supervision.
Scene Graph Parsing: Unlike the VRD task, scene graph parsing, first introduced by (Xu et al, 2017). aims parsing structured representation of the given image. A few works (Tang et al. 2019 Wang et al. 2019 Lin et al., 2020 Zellers et al., 2018) parse scene graphs by reasoning over visual context. Furthermore, Yang et al. proposes
Lin
R-CNN (Girshick) 2015) like model with additional graph convolutional networks (GCN) (Kipf & Welling 2017). Additionally, real-world commonsense, as well external knowledge, are utilized to guide the scene graph parsing process
(Zareian et al. 2020b Gu et al. 2019 Chen et al., 2019 Zareian et al. 2020c). Recently, Zareian ct al. proposes a novel graph alignment to parse scene graphs under weak supervision.
Graph Neural Networks (GNN); Recent years, great quantity of research efforts have been de- voted into Graph Neural Networks (GNN) (Scarselli et al.) [2088] Bruna et al. [013 Kipf Welling 2016). By adopting suitable aggregation strategies, GNN can capture features from various aspects, including edges, nodes and the whole graph (Battaglia et al.] 2018).
computer vision fields, graph neural networks can be applied to capture the structural information on both images and scene graphs (Yang et al. 20188) Johnson et al. 2018 Yang et al. (019). More specifically, by aggregating structural information on the input graph, GNN can represent scene graph node visual object as an embedding vector containing relational information of the edges connected to it.
Weakly Supervised Learning (WSL): Weakly supervised learning has been utilized in many com- puter vision tasks like object detection, semantic segmentation and visual relation detection (Zhang et al. 2017b] Huang et al. 2018 Zareian et al. 2020a). Many of them belong Multiple Instance
2

---

Under review as a conference paper at ICLR 2021
Learning (MIL) et al., 1997 Maron & Lozano-Péére Pérez 1998). When applying neural net- works in multiple instance learning, instance- pooling (Liu et al. 2012) and feature-level pool- ing( Wang al.) 2018a) are two common architectures in practice, making MIL compatible with gradient-based optimization strategies.
TASK DEFINITION
A scene graph is structured representation of an image, capturing both visual objects and their relations. We denote an image as I, regarded as set of visual objects {U1, U2, .... Um J. We assume that the features and locations of visual objects can be extracted by pre-trained object detector. And the scene graph describing 7 can be formulated as G = (U, R), where U = {uI, U2,..., Un is the node set (n I m) and R (ri, r2, .... r is the edge set. Each node has label yu which describes the object category it refers to. And each edge is represented as triplet (u4, Ug vo), where it is the source node, Uy is the target node and y, is the relation type.
r2.....
Scene Graph Grounding: Though the graph-image pair (G, 7) can be easily retrieved, the corre- spondences between graph nodes U and visual objects remain unclear. The goal of scene graph grounding is to map each node to visual object e inside the image.
Weakly Supervised Scene Graph Grounding: As manually labeling the ground-truth node-object references is expensive and time consuming, we aims at learning grounding model without relying on such amnotations. In Weakdy Supervised Scene Graph Grounding, we are given a training set D, where the each sample in D is tuple of an image I and its scene graph representation G. However, the ground truth node-object mapping is not provided for any tuple. We aims at learning a grounding model from D.
4 PROPOSED METHOD
To build up the mapping between visual objects and scene graph nodes, we propose to learn a object-node wise metric that incorporates information from both object categories and relations. Given an ungrounded image-graph pair, we measure the similarity of each object-node pair using learned metric and build a bipartite graph among the object set and the node set. this way, we formulate the object-node mapping problem as searching the minimum weight match in a bipartite graph. Unlike the NP-hard graph matching problem in previous work (Zareian et al.) (020a)), our formulation leads to polynomial time solution. The overview of our framework is shown in Fig. Formally, we define the metric on a given object- node pair (u, v) as following cost function:
c(u, v) ACobi v) + (1 A)Crel v)
where Coty (u, v) is the object-class based metric and crei(u, v) refers to the relation based metric.
4.1 OBJECT-CLASS BASED METRIC
Given a scene graph node and visual object v. we define following metric to incorporate object category information:
Coby(u, v) = P(yn # yo) 1 P(y. = yo) where yu is the object category node u and y/ the category of v. Since the ground-truth object- node mapping is not accessible during training phase, we can not acquire the ground truth categories of visual objects. Therefore, we train a neural network to estimate P(y. = yo) through pool based multiple instance learning. The estimate is denoted as P(y, you Ooty) where Oori is the parameter of the network. We optimize the neural network with the following loss function:
LMTI(G, 1;0mns) = Σ
log loga((P ((P()) y; Oots) ) E f})
(3)
DEY(U)
where is the image, U the node set of corresponding scene graph G, YU) the object cat- egory set of U and is pooling operator. For class y existing in the image, the pooling oper- ator aggregates the probabilities p(y, y; Botj) over all objects in the image I. It can be any
Howeeer, due to the label space difference between pre-training dataset and scene graph dataset, we give the category lahel of the visual objects
3

---

Under review as a conference paper at ICLR 2021
Cobj
Otjest Cless
Cost Matrix
Baard Metric
Hone
Solver
Pelation Basad Metii
Jiler
[1
Crel
Figure 1: Overview of Our Framework: Bounding boxes and Rol features of visual objects extracted by a pretrained detector; The object class based metric and relation based metric compute the cost matrix between visual objects and scene graph nodes; A matching algorithm is utilized to predict the grounding results.
Visual Graph Neural Network
Yalk in
Pensn
Pensu
Scene Graph Neural Network
Kide
Ride
Joint Relational Embedding Space Z
Horac
Bikk
Figure 2: Overview of Our Relation based Metric Module: Visual objects and scene graphs fed into two independent GNN model, which map them to the joint relational space Z.
permutation-innvarian operator, such mean. With the learned P(ya Vv), we define Cobj as:
Cobj(u, u; Bobs) P(y. Moo Bobj)
4.2 RELATION BASED METRIC
Apart from object category, the object-wise interaction and relation-object interaction are also im- portant for learning the mapping. as the scene in an image may be complicated (He et al.) 2020). For example, in one image there might be "person" "riding" a "horse" and "person" "pprson""riding "bike". crucial to correctly distinguish them for some downstream tasks like weakly supervised scene graph parsing.
To learn a metric capturing above interaction information from both visual input and scene graph, propose two-stream architecture, shown in Fig. to model the structural similarity. The two streams are two graph neural networks that project visual objects and scene graph nodes into first construct
shared embedding space Z. In visual stream, capture object-wist wiie interaction, fully connected graph where each node is visual object. Then the graph is forwarded into graph neural network, where the i-th layer updates the representation of object as follows:
h h+1 MI.P (AG.PEAAGGEAAATE (MLP) (h, (h, (N))))
(5)
4

---

Under review as a conference paper at ICLR 2021
where + means concatenate, AGGREGATE is aggregation operator, and (u) is the neighbor set of u. We initialize the representation of each visual object as the features extracted by pre- trained visual-encoder. Meanwhile, capture the relation features in scene graph, forward it into another graph neural network proposed in Johnson et al. (2018), which can model the relation- object interactions. The initial representation of node edge is the word embedding of its class name. Its updating rule in each layer is shown in Alg. [
Algorithm Updating algorithm of Scene Graph Neural Network.
Require: scene graph G (U, R), node representation hy of each node u E U and edge repre- sentation h, of each edge E: R
Ensure: Updated node representation h', and edge representation hy
(s Ur) do 2: h4 = 9r (h. Hh, (ha)) a function aggregating information from a triplet)
1: for each edge
3: hr (be))
linear layer with activation)
4: end for
5: for each node do
6: ha AGGREGATE(
E S(u)}) {fs is a linear layer with activation and S(u) is the set of edges whose source node is u}
7:
h AGGREGATE(( (faAT((((((a)))) (h,)]r E (u))) ) (So is a linear layer with activation and (()) is
the set of edges whose target node is u}
8:
K2 MLP(AGGREGATE((MM h())
9:
end for
Given scene graph node G and visual object I, let us denoted their representation in space Z as h(u) and h(v) respectively. Then we can calculate the structural similarity between and v as:
Cralu,, vz Grel) d(h(u),n(()))
(6)
where d is vector metric space Z and Oret is the parameters in relational metric module. Since graph neural network can represent structural information on graphs as vectors, by comparing h(u) and h()). we can measure the relational similarity of an object-node pair. However, training the graph neural networks is very challenging. Because when ground truth object-node mapping is accessible, we can not simply minimize the distance mapped object-node pairs. Since graph structure can not be simply classified to a set of discrete categories, we can not apply similar strategy as in learning object category metric.
Inspired by Wasserstein Distance (Rüschendorf) 1985). defined as the cost expect of the optimal transport on two distributions, we propose a distance that applicable to two discrete sets. Specifically, apply the cost sum of the optimal linear sum assignment the node sets and object sets as image-graph level distance, denoted as D(G,1)
D(G,I)= = Σ c(u,u)=
Σ (a.u)EM
Acot u) + (1 (u, v)
(u.))EM-
where the definition of M* is
M arg min
Σ
clu, v)
(8)
(G.(G.D) (M.D)M
where I) is the set of all possible match between the scene graph node set and visual object set. We then propose train the model by minimizing D I) for all mapped image-graph pairs.
Precisely optimizing above distance via gradient descent is intractable, because the process of searching the minimum match M* is non-differential. An alternative solution that in each it- eration, we first calculate the minimum match M based on current GNN parameters, then run gradient descent step that minimizes the distance of matched node-object pairs However, in the initial training stage, the relational embeddings randomly spread in the space. Thus, the match found
The justification of this alternative solution be found in Appendix
5

---

Under review as a conference paper at ICLR 2021
under current parameter may contain noise and mislead the training. To address this challenge, approximate M as M', which is defined as:
M'
arg min MEF(G..) (a.o)EM
Σ (u, v).
Since searching the minimum match on bipartite graph can be solved by Hungarian Algorithm (Kuhn, 1955) in polynomial time, such approximation can be computed as long as the multiple instance classifier trained advance. Our loss function on graph neural network branch can be written as:
Lynn (G,1=
Σ (L.))MM
ACobs(u, v) + (1 A) C C (1 X)
Σ ACret u) (10) (u,0)EM'
where C is constant. Note that if apply simple Euclidean distance or inner-product distance as d, above loss function will have a trivial solution where Cyet (u, v) = 0 for all (u, To avoid it, we their normalized version. Given distance d like Euclidean distance, its normalized version is defined as:
exp (-p(, v))
anorm (u, v) - ΣKEr (xp(-d(p (-d(u, uu k))
(11)
4.3 TRAINING AND INFERENCE
The training algorithm our model is illustrated in Alg. In general, the training can be divided as two phases. In the first phase, train the object-class based metric via pool based multiple instance learning. In the second phase, we train the Graph Neural Networks based on the well-trained object- class based metric. In testing stage, since Ooby and Orel have been well-trained, can calculate M with Hungarian Algorithm simply and output M as mapping result.
Algorithm Training Algorithm on Weakly Supervised Dataset D)
Require: Dataset D
Ensure: The well-trained parameters Bobs of the object category based metric and 0ret of the rela- tion based metric.
Randomly initialize foty and Orel
2: while doby not converged do
3: Sample batch of samples GI from D and forward the samples into the neural networks
in Sec. 4.1.
4: Calculate the loss function LMIL in Eq.
5: Update Bobj according to the loss function LMIL-
6: end while
7: Calculate minimum match M with Hungarian algorithm
8: while Orrt not converged do
9: Sample batch of samples GI, from D and forward the samples into the graph neural networks in Sec. 4.2.
Calculate the loss function Lynn in Eq. TO
11: Update ferj according to the loss function Lynn.
end while
5 EXPERIMENTS ON WEAKLY SUPERVISED SCENE GRAPH GROUNDING
5.1 EXPERIMENT SETTING
Datasets: We evaluate our grounding approach on two prevailing datasets: Visual Genome (ri)nKr et al. (017) and Visual Relation Detection (VRD) (Lu al. 2016). For VG dataset, we apply the split and preprocess protocol Xu et al. (2017). For VRD dataset, we apply the
original split and protocal in the original paper (Lu et al. 2016).
Baselines: We compare our method with following baselines. which support weakly supervised learning images and their ungrounded scene graphs: (1) VSPNNr (Zareian et al. 2020a) is

---

Under review as a conference paper at ICLR 2021
Visual Genome
VRD
Methods
Ground Truth
RPN Proposal
Ground Truth
RPN Proposal
WS-VRD 36.0
10.7
27.8
9.6
VSPNet-vl 38.6
14.7
52.6
9.7
VSPNet-v2
41.9
15.2
57.7
13.1
Our Method 50.8
15.8
68.9
16.5
Table 1: Results of Weakly Supervised Scene Graph Grounding Methods with VG and VRD: Our proposed model surpass all baseline models by noticeable margins.
framework jointly learning scene graph parsing and scene graph grounding under weak supervision. Its graph alignment module can alternatively select starting from mapping edges or mapping nodes. We denote the first one VSPNet-v/ and the latter as VSPNet-r2; (2)WS-VRD (Baldassarre) et al. 2020) is an explanation- based weakly supervised scene graph parser. It first aggregate the relation feature of all visual object pairs and predict all possible relation types that may exist in the image. Then it assign each predicted relationship label to the object pair that make the greatest contribution to the prediction.
Among models above, WS-VRD does not have an explicit grounding module. Therefore, to evaluate its performance on grounding task, we borrow the idea of VSPNet and first use it to transform input images to scene graphs where each node is bounding box. Then we use graph alignment algorithm to map each scene graph node bounding box based on the output graph.
Visual Object Extraction: To extract the visual object features from raw image, following VSPNet (Zareian et al. 2020a) we apply an off-the-shelf RCNN (Ren et al. 2015) object detector pre- trained on OpenImage (Kuznetsova et al. (020) dataset. The location of each object is represented as bounding box. Since scene graph grounding is highly affected by the quality of object proposals, report grounding accuracy under two settings: one is that the object detector is given ground truth bounding box as proposal, denoted as Ground Truth, and the other is that the object detector use the proposals from its own RPN (Region Proposal Network) module, denoted RPN Proposal. To be fair, we apply this extraction protocol on all methods.
We apply Accuracy evaluate the models on grounding task following (Sadu et al. (2019). In Ground Truth setting, a node is correctly mapped if it is exactly mapped to the corresponding ground truth bounding box. In RPN Proposal setting, a node is correctly mapped the IoU score between the mapped box and ground truth box is larger or equal to 0.5. The final accuracy averaged over all scene graph nodes.
Evaluation Metries:
5.2 RESULTS ON SCENE GRAPH GROUNDING
Quantitative results shown in Tab. 1. illustrates our advantages on both datasets, regardless of which type of proposals are given. On Visual Genome, our model improves the recall by nearly 9% with ground truth bounding boxes, while gets slightly higher recall with RPN proposals. Besides, our model achieves 68.9% recall, more than 10% higher than any other baselines, on VRD with ground truth bounding boxes. Additionally, with proposals generated by RPN, our model shows a 3% boost noticeable that the improvement made by our proposed method much larger when ground truth bounding boxes are given. Such difference indicates that model can better learn the mapping between objects and scene graph nodes with desirable object proposals.
on the recall.
5.3 ABLATION STUDY
To quantify the contribution of different components of model, conduct ablation study by comparing our model with two variants: (1) Without Minimum Match: after calculating the cost function matrix, instead of conducting minimum match via Hungarian Algorithm, simply map each node to the box with minimum cost function value with it; (2) Without Relational Metric: we set X in Eq. to 1 so that the model only makes use of the object category metric.
The results are shown in Tab. Compared with the model without minimum matching, our model has noticeable performance boost, indicating that matching node-box through Hungarian algo-

---

Under review as a conference paper at ICLR 2021
Visual Genome
VRD
Methods
Ground Truth RPN Proposal
Ground Truth
RPN Proposal
Ours w/o minimum matching
40.6
12.1
57.2
10.5
Ours w/o relational metric
49.1
15.4
66.7
16.6
Ours (full)
50.8
15.8
68.9
16.5
Table 2: Ablation Study on Our Grounding Model: The results show that our model benefits from the matching process and the relational metric
Node Type
Node Numbers
Acc w/o rel-metric
Ace with rel-metric
Ambi-Nodes with relations Ambi-Nodes without relations 155k
106k
28.3
33.3
91k
26.0
26.9
Unique Nodes
77.2
77.3
Table 3: Detailed Ablation Study on Our Grounding Model on VG dataset: The results show that model benefits from the matching process and the relational metric
rithm much better than greedy methods. It is also shown that our model is able to learn feasible information from data, compared with the model without relational metrics.
Furthermore, to verify that our relational based metric successfully learn to use relational infor- mation to distinguish the nodes with same object class (like the case of "person riding horse" and "person riding bike"), we report the grounding accuracy on different fractions of the nodes on test set of Visual Genome Dataset. We apply the ground truth bounding boxes as candidate boxes. We divide the nodes to three types: ambiguous nodes with relations, ambiguous nodes without relations and unique nodes. A node is unique node if there is no other nodes belonging to the same object class of in the same graph. Otherwise, the node is an ambiguous node. Obviously, for unique nodes, the object class information is adequate for grounding. For ambiguous nodes with relations, relational information is important for distinguishing them from the nodes with same object class.
The results shown in Tab. 3 demostrates that, for the ambiguous nodes with relations, the relational metric significantly improve the grounding accuracy. Also, even for the ambiguous nodes that do not have relations, relational metrie also helps the grounding. This is because our model applies minimum matching. When matching the ambiguous nodes with relations correctly, it also help the ambiguous nodes without relations exclude wrong options. Meanwhile, our relational metric does not interrupt the grounding of the unique nodes, which can be well handled by the object-based metric.
APPLICATION: IMPROVING WEAKLY SUPERVISED SCENE GRAPH PARSING
Apart from scene graph grounding shown in Sec. 5, we further apply our grounding results to a downstream task: weakly supervised scene graph parsing. More specifically, given a weakly supervised scene graph parsing dataset, we first apply our grounding model to map each node in the scene graph to bounding box in the image. In this way we generate grounded scene graph parsing annotations. We use above annotations train scene graph parsers in fully supervised manner. In the evaluation stage, we compare them with the parsers directly trained on the original dataset under weak supervision.
Experiment Setting: We conduct this experiment on Visual Genome with three weakly supervised methods: (1) VSPNet (Zareian et al., 2020a),, the current state-of-the-art; tron (MLP) which accepts the concatenation of two object features as input and predicts their visual relations; (3) A graph neural network (GNN) over fully connected graphs where each node repre- sents visual object. All three models can be trained under full supervision or weak supervision. To train the MLP and the GNN under weak supervision, we adopt the multi-instance instance- -etrannnee-eainggggg.. (MIL) based loss function.
(2) multi-layer percep-
8

---

Under review as a conference paper at ICLR 2021
SGCIs
PredCls
Methods
Supervision
Recall@50
Recall @ 100
Recall @50
Recall 100
MLP FS
Full
22.8
25.9
40.7
49.3
GNN FS
Full
23.5
26.6
41.5
50.0
VSPNet FS
Full
31.5
34.1
67.4
73.7
MLP-MIL
Weak
19.7
22.5
38.2
47.0
GNN-MIL
Weak
18.2
20.7
37.4
45.8
VSPNet WS
Weak
30.5
32.7
57.7
62.4
MLP (Ours)
Weak
22.4
25.1
41.1
49.8
GNN (Ours)
Weak
21.7
24.8
38.8
47.8
VSPNct (Ours)
Wcak
30.6
33.3
59.9
65.2
Table 4: Results of Scene Graph Classification and Predicate Classification VG: Our grounding results can act as a reinforcement to other scene graph parsing models and further improve their performances
SGGen
Methods
Recall @50
Recall @ 100
MLP-MIL
1.9
2.2
GNN-MIL
1.5
1.8
VSPNet WS
4.7
5,4
MLP (Ours)
2.4
3.0
GNN (Ours)
2.4
3.0
VSPNet (Ours)
5.1
5,8
Table Results of Scene Graph Generation on VG: Our grounding results can bring boosts to other scene graph parsing models in the scenario without ground-truth bounding boxes
Evaluation Metrics: Following (Herzig al. 2018) and (Zareian et al. 2020a). we evaluate the models above with three setups: scene graph generation (SGGen), scene graph classification (SG- Cls) and predicate classification (PredCls). SGGen requires a model to generate triplets from an input image without ground truth bounding box. A generated triplet is considered for both subject and object, the detected bounding boxes have an IoU of at least 0.5 with ground truth and (2) the categories of subject, object and relation all correctly predicted. SGCIs requires a model to predict object categories along with their mutual relations given ground truth bound- ing boxes. And PredCls asks for predicting relations for all visual object pairs given ground truth bounding boxes. We report Recall @K score, the ratio of ground truth triplets correctly detected by the model when top K triplet predictions taken into account, with K=50, 100. For the fully supervised models, we only report their performance of SGCIs and PredCls because they require denser proposals (300) SGGen setup compared to the weakly supervised models (20 proposals required), which is unfair for the comparison.
correct if (1)
Results: Quantitative results are shown in Tab. and 5, Such results demonstrate that our ground- ing model can act as an reinforcement for other weakly supervised approaches. In particular, our grounding improves the performance of the ML. model for nearly 3% over all metrics with ground truth boxes. The recalls of the GNN model also raise for more than I % with the help of our ground- ing results. Additionally, the correspondences generated from our model brings performance boost to VSPNet, the state-of-the-as on all metrics of all set up.
7 CONCLUSION
In this paper, we propose the task of weakly supervised scene graph grounding, establishing corre- spondences between visual objects and scene graph nodes, and provide an algorithm solving it. Our method surpasses the current state-of-the-an on the task. Furthermore, the outputs of our model can be applied to downstream scene graph parsing algorithm and effectively improve the performance.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Federico Baldassarre, Kevin Smith, Josephine Sullivan, and Hossein Azizpour. Explanation- based -upeevs-e learning of visual relations with graph networks. arXiv preprint 2eXiv:2006600562.. 2020.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Gonzazz- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. arXiv preprint arXiv:1312.6203 2013.
Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing network for scene graph generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6163-6171, 2019.
Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference compuler vision and Pultern recognition, pp. 3076-3366, 2017.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Péree Solving the multiple instance problem with axis parallel rectangles. Artificial intelligence, 89(1-2):31-71, 1997.
Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. Generating natural language
planations for visual question answering using scene graphs and visual attention. arXiv preprint 1902v:1902.0557 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai. and Mingyang Ling, Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1969-1978, 2019.
Chuanzi He, Haidong Zhu, Jiyang Gao, Kan Chen, and Ram Nevatia. Cparr: Category- pro- posal analysis for referring relationships. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Roci Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping im- ages to scene graphs with Information Processing Systems, pp. 7211-7221, 2018
structured prediction. In Advances in Neural
Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. Weakly-supervised semantic segmentation network with deep seeded region growing. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7014-7023, 2018.
Sho Inayoshi, Keita Otani, Antonio Tejero-de Pablos, and Tatsuya Harada. Bounding box channels for visual relationship detection. In European Conference on Computer Vision, 2020.
Justin Johnson, Agrim Gupta, and Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 160900060000207., 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International journal of computer vision, 123(())32-73, 2017.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuser Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJCV, 2020.
Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao' ou Tang. Vip-enn: Visual phrase guided convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1347-1356, 2017.
Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3746-3753, 2020.
Guoqing Liu, Jianxin Wu, and Zhi-Hur Zhou. Key instance detection in multi-instance learning. 2012.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference Computer Vision, 2016.
Oded Maron and Tomás Lozano-Pérez. framework for multiple-instance learning. In Advances in neural information processing systems, pp. 570-576, 1998.
Li Mi and Zhenzhong Chen. Hierarchical graph attention network for visual relationship detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13886-13895, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Julia Peyre, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Weakly- Weakly-supervised learning of visual relations. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5179- 5188, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Ludger Rüschendorf. The wasserstein distance and approximation theorems. Probability Theory and Related Fields, 70(1):117-129 1985.
Arka Sadhu, Kan Chen, and Ram Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4694 4703, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008
Botian Shi, Lei Ji, Pan Lu, Zhendong Niu, and Nan Duan. Knowledge aware semantic con- cept expansion for image-text matching. In Proceedings of the Twenty-Eighti International Artificial Intelligence. IJCAI-19, pp. 5182-5189. International Joint Con-
Joint Conference ferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/iccai22117722 URL
https://doi wwwp:lorrr..... wttgg//....99..... 24963/ 20119/997700
11

---

Under review as a conference paper at ICLR 2021
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning compose dynamic tree structures for visual contexts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6619-66288 2019.
Wenbin Wang. Ruiping Wang. Shiguang Shan, and Xilin Chen. Exploring context and visual pattern of relationship for scene graph generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. $188-8197 2019.
Xinggang Wang. Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15-24, 2018a.
Yu-Siang Wang, Chenxi Liu, Xiaohui Zeng, and Alan Yuille. Scene graph parsing dependency parsing. arXiv preprint 1803.09 2018b.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-5419, 2017.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pp. 670-685, 2018.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 10685-10694, 2019.
Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang. Jing Shao, and Chen Change Loy. net Mining deep feature interactions for visual relationship recogmition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 322-338, 2018
Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In Proceedings of the IEEE international confer- ence on computer vision, pp. 1974-1982, 2017.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Weakly supervised visual semantic parsing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020a.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. arXiv preprint 2rXiv:2001.02314 2020b.
Alireza Zareian, Haoxuan You, Zhecan Wang, and Shih-Fu Chang. Learning visual commonsense for robust scene graph generation. European Conference on Computer Vision, 2020c.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 5831-5840, 2018.
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embedding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5532-5540, 2017a.
Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang. Ppr-fen: Weakly supervised visual relation detection via parallel pairwise r-fen. In Proceedings the IEEE International Conference on Computer Vision pp. 4233-4241, 201 7b.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A POOLING-BASED MIL
Label Space (L1, L2, L3, L4, L6, L6}
object feature
object
11
12
13
14
16
O1
0.1
01
0.1
011
Neural Network Classifier with Solumax
object
O2
0.5
01
O3
abject
Pooling along
with column
Corresponding Label (L1, L3, L4)
025
0u2b
016
62
0.or
0:66
Maximize
Minimize
Figure 3: The pipeline of pooling-based MII..
Here we describe the pooling-bart based MIL with a pipeline Figure In each updating iteration, a neural network classifier takes the object features in and output the probabilities of each label in the label space. The probability vectors form a matrix where each raw corresponds to an object. After that, pooling operator like aggregates each column in the matrix scalar. Then gradient descent step is applied to maximize the aggregated probability of labels existing in the image and minimize the others.
B JUSTIFICATION OF THE ALTERNATIVE OPTIMIZATION METHOD
Here we give the justification of the alternative optimization method mentioned in Sec 4.2. Let c(u, v; Bobjs 0yet)
represent the cost function given current model parameters Ooojs O Let us define M (Oobj. Oret) as:
(0 Oret) arg min
Σ
c(u, v; thelasbj Oret)
(12)
Me=(G,I) (a.0)EM
Suppose we run an optimization step and obtained new parameters objz O If the new parameters satisfy:
Σ
c(u, ui Bobj. Ore)) 2
Σ
clu, ug Oobj. 0ret)
(13)
(W.))EM-(Ba)) Brec)
(B.a))
Based on the definition M (B's) P'e)) we have:
Σ
cuu, Oobgg 0re))
Σ
c(u, O' Oret)
(14)
(u. (Bobs Arat)
(u.t)EM (ha
And thus have:
Σ
c(u, 21 Bobj. Orei)
Σ
cuu, 21 8 Oongs 0rrt)
(15)
(u. M (Boes Arat)
C DATASETS
As mentioned in Sec. 5, we conduct our experiment with two datasets Visual Genome (VG) KKiishaa al. 2017) and Visual Relation Detection( VRD) (Lu et al. 2016).
13

---

Under review as a conference paper at ICLR 2021
VG comes with 108k images, along with object, relation and scene graph annotations. Following Zareian et al. (2020a), we keep 150 most frequent entities categories and 50 most frequent predicate classes. We also follow the split adopted by Zareian et al. (2020a), which contains 75k images for training and 32k for testing.
VRD contains 5k images with 100 object categories and 70 predicate types. More specifi- cally, the training set consists of 4k images while the test set has lk.
D IMPLEMENTATION DETAILS
D.I OVERALL SETTING OF GROUNDING MODEL
We set À 0.5 for VG dataset and À 0.9 for VRD datasset. As for the word embedding, in implementation, for VG dataset, we apply the same pre-trained GloVe word embedding (Pennington et al. 2014) as VSPNet for both object category names and relation type names. For VRD dataset, we apply the word embedding in spacy library. The word embedding dimensions of all datasets and methods are set as 300, following VSPNet. We apply Adam (Kingma & Ba 2014) with learning rate of le - 4 to optimize our model
D.2 OBJECT CLASS BASED METRIC
Here, let use denote the object category label set of the dataset as L 12,,,,..... We apply two- layer perceptron to estimate P(yelu) given the feature extracted by the Faster-RCNN, In the case where ground-truth bounding boxes are given, we directly use the perceptron to predict the category of the object. More specifically, we set the output dimension of the percepetron as [Z], which is the object category number of the dataset (150 in VG and 100 in VRD). Then we apply Softmax function to normalize the output vector to be distribution. In the case where we are given RPN proposals, we first use the perceptron to project an object feature to a 300-dimension vector Iu- Then calculate the inner dot product between the 300-dimension vector and word embedding of each object category. Then the object category distribution P(y (z,)) of an object v is predicted as:
exp F. w(4)
P(y = 11(x)) = Σj<11 exp w(l))
(16)
where w(l)) is the word embedding of category 4. For the pooling operator in LaSLL apply mean for the RPN proposal setting of VG and max for others.
D.2.1 RELATIONSHIP BASED METRIC
Visual Graph Neural Network:
We apply mean as aggregating operation. We set the MLP, and MLP₂ two-layer perceptrons, with hidden dimension of 512 and output dimension of 512.
Scene Graph Neural Network: We apply sum as aggregating operator in line and of Alg. As for the aggregating operator in line 8, we apply following definition:
h h2
AGGREGATE h:) = [S(x) [O())
(17)
which is the same as Johnson et al. For the MLP in line we apply a two-layer perceptron, with hidden dimension of 512 and output dimension of 512.
D.2.2 SETTING OF SCENE GRAPH PARSING MODEL.
Here we introduce the implementation of MLP and GNN parsing models in detail. As for the VSPNet, directly apply all of the original setting in the original implementation of the authors.
The loss function MLP FS and GNN FS is cross entropy applied on object category and relation type respectively. As for MIP-MIL and GNN-MIL, the loss function of object category is the same as the loss function of object class based metric. And the loss function of relation type on one sample (I, = (U, R)) is defined as:
Loss function:
L
loga y..) P(ye tw P(U(e))) =vr)lu, EEE
(18)
14

---

Under review as a conference paper at ICLR 2021
where is max-pooling operator among all possible object pairs. P(y y-) is the possi- bility that the relation from to o' belongs to type yr predicted by the model. When applying our grounding results on MIP,, GNN and VSPNet, we just treat our grounding result as ground-truth and run the model in fully supervisd setting.
Optimization: We apply Adam (Kingma & Ba. 2014) with learning rate of le 4 to optimize our model. We use validation set of IK images for early stopping. To be fair, for all three settings (fully supervision, weakly supervision and the version with our grounding), we provide ground- truth scene graph the validation sets for GNN and MLP. As for VSPNet, directly copy their hyperparameter setup. In VSPNet + our Grounding setting, we borrow the hyperparameters of their fully supervised version.
Neural Network Architecture: For MLP, we apply two-layer perceptrons whose hidden dimension number equal to their input dimension number. For GNN, directly use the architecture and hyperparameter of the Visual Graph Neural Network in grounding model. We concatenate the output of the VGNN and the initial feature from Faster-RCNN get the final feature of each visual object. Then we feed them into MLP to predict their categories and feed the concatenation of all object pairs into another MLP predict their relation categories.
E VISUALIZATION OF OUR GROUNDING MODEL
We visualize our scene graph mapping results on several samples given ground-truth bounding boxes. The visualization results are shown in Fig. We Further visualizes some failure cases Fig. 5 In the first failure case, the model gets confused between "book" and "paper", while in the second case, the model fails to distinguish different plates. In Fig 6 we show that when there are multiple objects with the same category the image, our model is able to distinguish them through object-wise relations.
15

---

Under review as a conference paper at ICLR 2021
Man
On
Ski
Snow
ntain
Box
Above
Table
Sitting On
Pizza
Logo
Cup
Hair
Has
W om
an
frout
Has
Buildi
ng
Coat
Lamp
Lapto p
Sitting
Desk
Leg
Figure 4: The visualization the mapping found by our model given ground bounding boxes.
16

---

Under review as a conference paper at ICLR 2021
Book
On
NNer
Has
Table
People
Face
In Front
Paper
of
Box
Sitting On
Plate
Table
egeta
ble
Sitting On
Food
On
Plate
Engine
Seat
Has
or-
Bike
On
W hhel
Above
On
Handle
Window
Table
ln
Room
In
Bed
On
Book
Figure 5: Visualization on Failure Cases of Our Model
17

---

Under review as a conference paper at ICLR 2021
Gassses
Wearing
Man
Has
Hair
W Woman
Has
Hair
Bottle
Holding
Guy
Shirt
In
Guy
Nose
Dog
Of
Eye
Man
of
Eye
W Wearing-
J Jacket
Wave
Over
Man
On
Board
On
Man
Figure 6: Visualizations on Ambiguous Objects with Relations
18


Reviews:
[{"review_id": "lb5LUZd4Rq-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###############################\nSummary:\nThis paper focuses on a new scene understanding task: Weakly-Supervised Scene Graph Grounding. Given the image and a paired scene graph, this task needs to link each scene graph node to an object in the image. Specifically, it regards this problem as a bipartite graph matching problem, and designs a cost function considering both object-class similarity and relation-based similarity. Experiments on two datasets (Visual Genome and Visual Relation Detection) have demonstrated the effectiveness.\n\n################################\nAdvantages:\n+ The paper is the first work to explore the task: Weakly-Supervised Scene Graph Grounding. This task can serve as a crucial preprocessing step for weakly-supervised scene graph generation: After solving this grounding task, the weakly-supervised scene graph generation task can be easily transferred to fully-supervised scene graph generation by regarding the grounding results as ground-truth annotations.\n\n+ The whole paper is well-written and all notations are quite clear.\n\n#################################\nWeaknesses:\n\n- The novelty of the idea for weakly-supervised scene graph grounding is limited. It regards this grounding task as a bipartite graph matching problem, which is very similar to the existing work for weakly-supervised scene graph parsing (ie, VSPNet). \n\n- Although this paper is the first work to explore weakly-supervised scene graph grounding task, the task is very similar to the existing weakly-supervised phrase grounding task.\n\n- The comparisons between the proposed method and baselines are not fair. For example, in Table 1, for the baseline WS-VRD [Baldassarre et al., 2020] model, it only utilizes predicates as input for weakly-supervised learning, while the proposed model uses the whole scene graph as input.\n\n- The details of some experiments are not clear. For example, for the compared baseline VSPNet [Zareian et al., 2020a], the authors use two variants of models (VSPNet-v1 and VSPNet-v2), but the details of these two variants are not clear, ie, how to change an scene graph parsing model (VSP) for the scene graph grounding task.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HCEFU9T2JQ7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a method on the task of Scene Graph Grounding, where the object and (object, object) relationship ground-truth labels are only at the image-level but not localized, i.e., instance-level. The core of the proposed method is the Wasserstein distance minimizer (aka, Earth Mover's Distance) for two object pair sets: one is from visual detection, and the other is from the textual nouns. The initial cost for each set is initialized by MIL probabilities. Experiments demonstrate a considerable performance boost over a SOTA (Zareian et al., 2020b)\n\nStrength:\n1. Strong performance in Table 1 (I do not mean that the experiments are comprehensive, see weakness below)\n\n\nWeakness:\n1. The motivation is not clearly justified in the Introduction. For example, for readers not familiar with this task, the authors may want to provide a clear illustrative example of this task, to show why scene graph grounding is challenging and significant. Moreover, compared to (Zareian et al., 2020b), whom the authors are targeting, in fact, I see no obvious weakness in their method and how you resolve it. \n\n2. I am not objecting to use the simple EMD to solve weakly supervised matching problems, as it is straightforward. However, I feel that the authors are just using the right tool to do the right job, without convincing the readers why this tool is sufficient and necessary, especially compared to many other possible tools, such as using GNN, (Zareian et al., 2020b), MIL (Zhang et al. 2017b), and graphical models (Justin et al, Image Retrieval with Scene Graphs, CVPR'15). \n\n3. The overall pipeline is not clearly illustrated. For example, after the EDM matching, how to obtain the final scene graph alignment?\n\n4. Experiments are not comprehensive. First, I'd like to see visual qualitative examples of the success and failures of grounding, especially for improving weakly supervised SGG. Second, ablations such as using different metrics for the matrices are missing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "H6N0GOpP2z-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new problem called weakly supervised scene graph grounding, which is potentially very useful but barely studied in the literature. Given a dataset of images labeled with scene graphs, but without bounding box annotation, the task is to learn to align each node of a given scene graph to the corresponding object in the image. This is useful for weakly supervised training of scene graph generators, as well as other applications such as multimodal grounding. The authors clearly define the problem and propose an evaluation setting for it. They also propose a new method for this task and compare to existing baselines. Their findings are interesting, novel, and potentially influential for future work.\n\nNevertheless, there are some confusions/weaknesses that can be improved in the rebuttal:\n\n1. There is insufficient insight about the proposed method and whether/why it works. Particularly, the authors first train a weakly supervised object recognition model using a multiple instance learning framework (Eq. 3), then use that to align ground truth nodes to objects (Eq. 9), and then use this alignment as a ground truth for training the graph neural networks (Eq. 10). However, the ground truth alignment produced by the object recognition model only takes into account object classes and not their relations, while the goal of the graph neural networks is to incorporate relations to disambiguate between instance of the same class. Hence, the ground truth is inadequate to learn the task and the model may just learn to ignore relations and use objects alone to do the alignment. Experiment results (table 2) confirm this concern to some extent, as the ablation without relational metric (no graph neural net) achieves very close performance to the proposed model. Therefore, it is essential for the authors to justify their proposed model e.g. using extra experiments/analyses/visualizations, to prove that the graph neural networks indeed does learn to take relations into account for grounding objects.\n\n2. The writing of the paper can be improved. It is not very clear, except from Algorithm 2, that the training framework has 2 phases: training the object recognition model, and then training the graph comparison model based on that. The authors may add an overview to explain this. There are also notation issues and confusions in the equations. For instance, the definition of U, N, and Y below Eq 3 is confusing. Also it is not clear what is the input set to \\sigma (pooling). The definition of \\pi in Eq. 8 is also not precise enough. And in Eq. 10, the second equation is confusing, maybe the \\lambda inside the sum should be removed. Furthermore, Section 6 should be elaborated more. For instance, it should be made clear that this makes the training a 3-step process: learning object recognition, then learning the graph metric, and then using that to learn the scene graph generation model. \n\n3. Experiments provide new findings and useful insights, but they can be improved. For instance, it is not clear why, even though the proposed alignment method is far better than the VSPNet baseline in table 1, its effect on the scene graph generation performance is small (table 3). Moreover, the SGCls and PredCls evaluation metrics are both unrealistic metrics that assume ground truth bounding boxes are available during inference. The authors are encouraged to report SGDet, to show how effective the proposed method is in real-world settings.\n\nDespite the suggested improvements in experimentation, writing, and justification of the method, this paper carries a significant value for the community, as it calls attentions to a new task, that is barely studied, but has potential applications. The empirical findings of the paper provide new insights that do not exist in the literature, and can directly guide future work. Hence, my recommendation is to accept this paper. \n\nThe authors are strongly encouraged to address the concerns mentioned above. I also recommend resolving grammar mistakes throughout the paper, although that should not affect the final decision. Here are some more suggestions to improve the paper:\n\n1. The authors are encouraged to more extensively justify why this task is important and difficult. They did provide an example of a person riding a horse while another person is riding a bike, but that single example may not be convincing enough for others to pursue further research in this direction. The authors may provide some statistical analysis of how many difficult grounding cases exist in the dataset, what makes these cases difficult, and how improving the grounding can improve downstream tasks such as scene graph generation.\n\n2. There are some recent papers on visual grounding by incorporating relationships between objects, such as [1]. I recommend including a comprehensive review of such papers and discussing their connections and differences with this work.\n\n[1] He, Chuanzi, Haidong Zhu, Jiyang Gao, Kan Chen, and Ram Nevatia. \"CPARR: Category-based Proposal Analysis for Referring Relationships.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 948-949. 2020.\n\n\n######## Post-Rebuttal Updates:\n\nThe authors have addressed most of my concerns, and I appreciate their effort. I am not convinced that early-stopping alone can help the model learn to utilize relation information, when the target ground truth does not require the model to do so. However, I recommend accepting this paper as it introduces a new task and presents strong results. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4wPdxiv2KV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\nThis paper proposes a weakly supervised scene graph grounding method. The setting of this paper is new and interesting, and I think this task would be beneficial to downstream tasks. The main contribution of this method is the mapping between visual objects and nodes. To bridge the gap between them, the authors propose two metrics and learn their model on a weakly-supervised dataset. The ablation study shows the effectiveness of each contribution, and they further apply the method to weakly supervised scene graph parsing.\n\n**Pros**:\n- This paper proposes the weakly supervised scene graph grounding task and formulates the mapping between scene graph and objects as a minimum match problem on a bipartite graph instead of graph alignment. \n- They train their model with two metrics (object-class based and relation-based) and search the minimum match in an iterative manner.\n- The visualization shows that nodes and objects can be well aligned.\n\n**Cons**:\n- What's the design of pool-based multiple instance learning? Can you provide more details?\n- The updating algorithm of the scene graph is not new; it is close to the graph encoding process; it updates the representations according to its neighbor. \n- In fig. 3, the authors mentioned that the boxes are ground-truth, while they also mentioned that the visual objects are extracted by off-the-shelf Faster-RCNN.\n- For VG and VRD, in my opinion, the objects have been semantically aligned with scene graph nodes since the objects are detected by the pre-trained detector (OpenImage). Why not use a scene graph generated from captions as the input.\n- Why the $\\lambda$ for VG and VRD is different, how to get those values?\n- As for visualization, the authors only show the scene graphs with the same objects. They should also show the images with objects, not in the scene graph.\n- The writing needs to be improved; for example, Eq. 11 should be exp(-d(u,v)).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on scene graph grounding under weak supervision.  The reviewers appreciated the consideration of this task and formulation of a solution for it.  However, concerns were raised over the importance of this weakly-supervised grounding task, how it addresses challenges in previous methods, the empirical evaluation, insights obtained, motivation, and clarity of exposition.  After reading the authors' response, the subsequent discussion and reconsideration resulted in a sense that while the task is new, the overall contribution and remaining questions over empirical evaluation mean the paper is not yet ready for publication at ICLR.
Decision: Reject