Joint Modeling of Visual Objects and Relations for Scene Graph Generation
Authors: Minghao Xu
Meng Qu
Bingbing Ni
Jian Tang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Scene Graph Generation
Conditional Random Field
Mean-field Variational Inference

Abstract: An in-depth scene understanding usually requires recognizing all the objects and their relations in an image, encoded as a scene graph. Most existing approaches for scene graph generation first independently recognize each object and then predict their relations independently. Though these approaches are very efficient, they ignore the dependency between different objects as well as between their relations. In this paper, we propose a principled approach to jointly predict the entire scene graph by fully capturing the dependency between different objects and between their relations. Specifically, we establish a unified conditional random field (CRF) to model the joint distribution of all the objects and their relations in a scene graph. We carefully design the potential functions to enable relational reasoning among different objects according to knowledge graph embedding methods. We further propose an efficient and effective algorithm for inference based on mean-field variational inference, in which we first provide a warm initialization by independently predicting the objects and their relations according to the current model, followed by a few iterations of relational reasoning. Experimental results on both the relationship retrieval and zero-shot relationship retrieval tasks prove the efficiency and efficacy of our proposed approach.

Joint Modeling of Visual Objects and Relations for Scene Graph Generation
Minghao Xu'
Meng Qu
Bingbing Ni
Jian Tang
Shanghai Jiao Tong University, Shanghai 200240, China
Mila Québec Al Institute
University of Montréal "HEC Montréal
SCIFAR Al Research Chair
(xuminghaol 18, nibingbing) @sjtu.edu.cn
meng menggquu@umonteallcc jian.tan@@hec.om hec.ca
Abstract
An in-depth scene understanding usually requires recognizing all the objects and their relations in an image, encoded as a scene graph. Most existing approaches for scene graph generation first independently recognize each object and then predict their relations independently. Though these approaches are very efficient, they ignore the dependency between different objects as well as between their relations. In this paper, propose principled approach jointly predict the entire scene graph by fully capturing the dependency between different objects and between their relations. Specifically, we establish unified conditional random field (CRF) to model the joint distribution of all the objects and their relations a scene graph. We carefully design the potential functions enable relational reasoning among different objects according to knowledge graph embedding methods. We further propose efficient and effective algorithm for inference based mean- field variational inference, in which we first provide a warm initialization by independently predicting the objects and their relations according the current model, followed by a few iterations of relational reasoning. Experimental results both the relationship retrieval and zero-shot relationship retrieval tasks prove the efficiency and efficacy of our proposed approach.
Introduction
Modern object recognition 32 of visual objects, which has significantly boosted many industrial applications such as intelligent surveillance [88 49] and autonomous driving (3] 38]. To have deeper understanding a visual scene, detecting and recognizing the objects in the scene is however insufficient. Instead, compre- hensive cognition of visual objects and their relationships is more desirable. Scene Graph Generation (SGG) [33] is natural way to achieve this goal, in which graph incorporating all objects and their relations within a scene image is derived to represent its semantic structure.
10 35
and detection [28 27 57 systems excel at the perception
Most previous works for SGG 48 objects in scene and then predict their relations independently. practice, though such methods very efficient, they ignore the dependency between different objects and between the relations of different object pairs. For example, a car could frequently co-occur with a street, and the relation eating could always appear along with the relation sitting on. Modeling such dependency could be very important for accurate scene graph prediction, especially for rare objects and relations. There indeed some recent works [6] [5] along this direction. For example, Dai et al. [6] explored the triplct-level label dependency among a head object, a tail object and their relation. These methods have shown very promising results, while they only explored the limited dependency within triplet. How to capture the full dependency between different objects and between their relations within whole scene graph remains very challenging and unexplored.
55
53
36
37 usually first independently predict different
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To attain such a goal, in this paper, we propose a principled approach called Joint Modeling for Scene Graph Generation (JM-SGG) to predict the whole scene graph by jointly capturing all the label dependency within it, ie. the dependency between different objects and their relations and also the interdependency between them. Specifically, we model the joint distribution of all objects and relations in a scene graph with the conditional random field (CRF) framework [17]. To flexibly model the joint distribution, the key is to define effective potential functions on both nodes (i.e. objects) and edges (i.e. relations between objects). We define the potential functions on objects according to the object representations extracted by existing neural network based object detector. It however nontrivial to design effective potential functions capture the relation between two objects in an edge and meanwhile allow relational reasoning among different edges, which models the dependency among the relations on various edges. Inspired by the existing work of knowledge graph embedding [20), which represents entities and relations in the same embedding space and performs relational reasoning in that space, define our potential functions according to the knowledge graph embedding method and hence allow efficient relational reasoning between different object pairs in a scene graph.
edges, since these potential functions have to
Such a fully expressive model also brings challenges to both learning and inference due to the complicated structures between different random variables in the CRF, i.c. objects and their relations. We therefore further propose an efficient and effective inference algorithm based on mean- mean-field variational inference, which is able to assist the gradient estimation for learning and derive the most likely scene graph for test. Traditional mean-field methods usually suffer from the problem of slow convergence. Instead of starting from a randomly initialized variational distribution as in traditional mean-field methods, we propose to initialize the variational distribution, iee. the marginal distribution of each object and each relation, with factorized tweak of JM-SGG model, and then perform few iterations of message passing induced by the fixed-point optimality condition of mean field to refine the variational distribution, which allows our approach to enjoy both good precision and efficiency.
To summarize, in this paper, we make the following contributions:
We propose Joint Modeling for Scene Graph Generation (JM-SGG) which is fully expressive model that can capture all the label dependency in whole scene graph.
We propose principled mean-field variational inference algorithm to enable the efficient learning and inference of JM-SGG model.
We verify the superior performance of our method on both relationship retrieval and zero-shot relationship retrieval tasks under various settings and metrics. Also, we illustrate the efficiency and efficacy of the proposed inference algorithm by thorough analytical experiments.
2 Related Work
Scene Graph Generation (SGG). This task aims to extract structured representations from scene images [13.. including the category of objects and their relationships. Previous works performed SGG
by propagating the information from different local regions [48, 53 50 36), introducing external
knowledge 52), employing well-designed loss functions [56] 14 34 and performing unbiased
scene graph prediction [4. 19 37]. Most of these methods predict each object and relation label independently based an informative representation, which fails to capture the rich label dependency within a scene graph and is thus less expressive. Several former works [ attempted to model such label dependency within single relational triplet but not on the whole scene graph.
Improvements over existing methods. The proposed JM-SGG model is, to our best knowledge, the first approach that jointly models all the label dependency within a scene graph, including the one within object or relation labels and the one between these two kinds of labels. To attain this goal, a unified CRF is constructed for graphical modeling, and mean-field variational inference algorithm is designed for efficient learning and inference, which show technical contributions.
Conditional Random Fields (CRFs). CRFs are class of probabilistic graphical modeling methods which perform structured prediction upon the observed data. CRF-based approaches have been broadly studied on various computer vision problems, including segmentation [17
43 51 26). super- (29 and scene graph generation [6] (5) These former works
resolution 39 46), image denoising
utilizing CRF for SGG 6 5 aimed to model the conditional distribution of single triplet upon
visual representations. By comparison, our approach models the conditional distribution of a whole
scene graph upon the observed scene image, which is more expressive.
2

---

3 Problem Definition and Preliminary
3.1 Problem Definition
This work focuses extracting scene graph, i.e. a structured representation of visual scene [13.. from an image. Formally, we define a scene graph as G (yo, R). yo denotes the category labels of all objects O in the image, and it holds that yo E C for each object E O, where C' stands for the set of all object categories, including the "background" category. R = {(o,, r, ot) is the set of relational triplets/edges with E T the relation type from head object on to tail object or (on, Or E O), where T represents all relation types, including the type of "no relation". In this work, aim at jointly modeling visual objects and visual relations defined below:
Joint Scene Graph Modeling. Given an image we aim to jointly predict object categories yo and the relationships R among all objects, which models the joint distribution of scene graphs, i.e. ((G(1) = p(yo, R()), with comprehensively considering the dependency within yo and R and also the interdependency between them.
3.2 Conditional Random Fields
Conditional Random Field (CRF) is a discriminative undirected graphical model. Given a set of observed variables x, it models the joint distribution of labels y based Markov network that specifies the dependency among all variables:
p(ylx)
1 (xx
II (c(x).)c). Z(x) = EII (x(xx))c).
(1)
where &c denotes the nonnegative potential function defined over the variables in clique C' (a clique a fonnecco local subgraph), and Z (xx a normalization constant called partition function.
Model
In this section, we introduce Joint Modeling for Scene Graph Generation (JM-SGG). Current methods solve the problem by independently predicting each object and relation label upon an informative representation, and thus the prediction of different labels cannot fully ben- efit each other. JM-SGG tackles the limitation by jointly modeling all the objects and relationships a visual scene with unified conditional random field, which en- ables the prediction various object and relation labels to sufficiently interact with each other. Nevertheless, learning and inferring this complex CRF is nontrivial, and we thus propose to maximum likelihood esti- mation combined with mean-field variational inference, yielding an efficient algorithm for learning and inference. Next, we elucidate the details of our approach.
CRF overview
G
You
Yu
(a) object components
(b) relation components
observed variable
latent variable
Figure 1: The CRF model for scene graph. (a) An object component models the de- pendency of object category Yo on image I. (b) A relation component models the de- pendency of relation type r on image and also the interdependency among object and relation labels (i.e. You. yo, and r))
4.1 Representation
In the JM-SGG model, we organize the observed scene image I and all object and relation labels in the latent unified
scene graph (i.e. No and R) as the nodes conditional random field. Since the interactions of these nodes are either for a single object or for the relationship between an object pair, we decompose the graphical structure of whole network into two sets of components. (1) Object components: For an object o E O, we consider the dependency of its category label on its visual representation and thus connect Yo with J, as shown in Fig. aa). (2) Relation components: for relational triplet (on, r, 0t) E R, we consider the dependency of relation type on the visual cues in image and also model the interdependency among the object and relation labels in this triplet (i.e. You . Yo:
3

---

and r), which forms a relation component as Fig. Tb) shows. By combining all object and relation components, the CRF can capture the comprehensive label dependency within a scene graph. We now define the joint distribution of scene graphs upon the observed scene image as below:
pe(G)]) Zo(1) fe(G.1),
(2)
Φ(yo 1) (e 1) II oEO
II
Vrr Vou Yous I);
(3)
where & summarizes the parameters of whole model, fe is an unnormalized likelihood function, Ze denotes the partition function, and é and & are the potential functions defined on object and relation components, respectively. Next, we define these potential functions based on the extracted visual representations and the correlation among different labels.
Visual representation extraction. Given a scene image I, we first utilize a standard object detector (e.g. Faster R-CNN (28) in our implementation) to obtain set of bounding boxes which potentially contain the objects in the image, and object representations zo = (zolo E O} (Eo E DR are then derived by RolAlign [I]. We regard the union bounding box over pair of objects as their context region and again use RolAlign to get all context representations žπ = {zhxl (on, r, 0e) E R] (#hr TR. D). Here, D denotes the latent dimension of objects and contexts. By denoting the whole object detector as ge. this feature extraction process can be represented as: (zo. =R) ge()).
Potential function definition. The potential function @ I) for object component models the dependency of object category Yo on object representation 2. by measuring their affinity. To conduct such measure, represent each object category with a prototype [33 (i.e. a learnable embedding vector) in the continuous space, which forms a prototype set C = {C, E R categories (D denotes the dimension of object space). On such basis, we define Φ(yo: T) by computing the distance between object representation 2, and the prototype of object category yo:
li E C} for all object
á(yo, f) exp (-d(C,,: yo: zo)),
(4)
where d is a distance measure (e.g. Euclidean distance in our practice).
The potential function (r: Mo., ,o., for relation component models the dependency of relation type r on the relevant visual representations in image I, and it also models the interdependency among the object and relation labels of triplet (i.e. No,, Mo, and r). Therefore, we can factorize vò(r, Mon Mo, I) into a term (rvsuaalr, (r, I) for modeling visual influence and another term Veriplet (r, Mop a No.) for modeling the label consistency within triplet:
(r. Vou yo., I) = (riisual I) Mrrippee Nor Vo.).
(5)
Similarly, for measuring in the continuous space, prototype set T = (T, E RK E T) is constructed for all relation types (K denotes the dimension of relation space).
We consider two kinds of visual representations that affect the prediction of relation type i.e. the context representation Zha and the head and tail object representations 2o, and zo.- The influence of context representation can be easily measured by projecting context representation The to the relation space and computing its distance to the prototype of relation type r. However, measuring the influence of head and tail object representations and evaluating the label consistency within a triplet are nontrivial, which require to model the ternary correlation among head object, tail object and their relationship. Inspired by the idea of Transk (20). an effective knowledge graph embedding technique, we model such ternary correlation by treating each relation as a translation vector from head object embedding to tail object embedding in the same embedding space. Specifically, we first apply the translation vector T, specified by relation r to head object embedding, and then compute the distance between the translated embedding and tail object embedding. Based on these thoughts, we define vissnal (r, and Veriplet (r, You: Mo,) as follows:
Viisuai 1) exp (-(d(T,,, Moths) d(Moños -T,, (o..)))))))
(6)
Piriplor (r. Bo, yo.) = exp (- dMM + TT. (T,.M,C,...)
(7)
where M. E RR denotes the projection matrix mapping from context space to relation space, and M, E R is the projection matrix mapping from object space to relation space. Next, we state how to learn the parameters in JM-SGG model.

---

4.2 Learning
In the learning phase, we seek to learn the parameters C, T, M, and M, of potential function and the parameters of object detector by maximum likelihood estimation, where θ summarizes all these parameters. Specifically, we aim to maximize the expectation of oog-likelhhod function log pe (G]T) with respect to the data distribution pa. i.e. [()) = EGapa log pe (G[I)], by performing gradient ascent. The gradient of the objective function ((O) with respect to O can be computed as below:
(e) = Ea- V log fe (G, - EG-pe Ve log fG)
where pe, is the model distribution that approximates pa (i.e. the conditional distribution pe (GIT) defined by JM-SGG model). This formula has been broadly adopted in the literature [|3 77, and
we provide the proof in supplementary material. In practice, we estimate the first expectation in Eq. (8) with the ground-truth scene graphs in mini-batch. The estimation of the second expectation in Eq. (8 requires to sample scene graphs from the model distribution, which is nontrivial due to the intractable partition function (I) that sums over all possible scene graphs. One solution is to run the Markov Chain Monte Carlo (MCMC) sampler, but its computational cost is high, and we therefore use mean-field variational inference for more efficient sampling (the detailed scheme is stated in Sec. 4.3)
Instead of fixing the parameters of a pre- trained object detector during learning as in former works [53] 36|37] 37 34.. we fine-tune the parameters of object detector during maximum likelihood learning. In this way, the detector can extract more precise object and context representations by learning the likelihoods of whole scene graphs. Also, we apply a traditional bounding box regression constraint freg (+) 28 to the detector for preserving its localization capability, and these two learning objectíves share the same weight. Next, we introduce the inference scheme for JM-SGG model.
4.3 Inference
The inference phase aims to compute the conditional distribution pe (G)T) defined by JM-SGG model and also sample from Exact inference is always infeasible due to the complex structures among the latent variables yo and R of the scene graph well as the intractable partition function. Therefore, we approximate (G]z)) with a variational distribution (o(G)) via the mean-field approximation 24):
qe (G) = II ge
II go(r),
(9)
DEO
(oN,T:o.)ER
where each factor ge (Mo) and (r) defines a categorical distribution, i.e. Ey,ec ge (v.) = 1 and Eret 9e (r) 1. In this variational distribution, all object and relation labels are assumed to be independent, and it shares the same set of parameters θ with (e 7), which greatly reduces the number of parameters needed for variational inference. For brevity, we will omit θ in the following distribution notations, e.g. simplifying ge geGG) as (G)).
In general, we are seeking for variational distribution that satisfies the factorization in Eq. ( and also maximizes the variational lower bound L(q) = (C) [log plo - log (G)) (i.e. equivalent to minimizing the KI. divergence between (G) and p [)). Typically, this is achieved by optimizing the variational distribution with fixed-point iterations 144 45). which can however be inefficient, especially for the images with many objects. We thus design an inference algorithm that appropriately initializes each factor in (G') and iteratively updates all factors. Intuitively, factor initialization is similar to existing SGG methods, where object and relation labels are predicted independently; factor update can be viewed as a refinement procedure, which makes the predictions from the initialization step more consistent. With factor initialization and factor update, the proposed inference method combines the advantages of both existing methods and CRFs, f.e. efficiency and consistency.
Factor initialization. For initialization, we neglect the interdependency among different object and relation labels, iee. omitting the potential function thriptet (r, Yo, a Vb,) ) in p(G) T), yielding simplified model distribution p(G))). In this way, we can easily derive the following factors for initialization which makes g(G) = q(G));
D(yo, ((U0) Emec 1.
Vo O,
(10)
Vvisual(r;
q(r) EVET (T'
V(o,,T, R.
(11)
5

---

See supplementary material for the proof. Intuitively, we initialize each factor by only considering its dependency on visual representations, and, on such basis, label interdependency will then be taken into account to refine each factor. In such an initialization approach, the computation of different factors independent with each other and thus can be done efficiently in a parallel manner. In Sec. 6. we empirically illustrate the better convergence performance of this initialization scheme compared to the random initialization which is commonly employed in previous works [45 [2]..
Factor update. Based on these initialized factors, we perform update by taking into account the interdependency among the object and relation labels in scene graph, f.e. using the full expression of P[G[I) with potential function surippet (r, you Vo.). ). In the mean-field formulation of Eq. (9) if we are to update one factor ((3) (or q(r)) with all other factors fixed, its optimum q" (yo) (or 9" (r)) which maximizes the variational lower bound L(g) can be specified by the following expression:
log q² (yo) log o(yo: 1) +
Σ (0,r,o) No, FET
Σ
Σ
ayy, )((r) log Veriplet (T', No: Mo.)
(12)
Σ
E
Σ
A(yon (q(r) log Preiplee (r, Mon yo) + const Yo E O,
(o). Yog ECT
log 8² (r) = log Vivisual I)
Σ Σ
Q(yon)9(Ua log (rrripter Yor, Vo.) + const V(o,, E R. (13)
Bog EC No,
The proof is provided in supplementary material. During computation, we omit the additive constants above, since they can be naturally eliminated when computing normalized q" (v,) and q q((r), i.e. taking the exponential of both sides and normalizing q (vo) over C and g (r) over TT Taking close look at Eqs. (12) and (3). we can find that each factor is updated by aggregating the information from its neighboring factors (e.g. from the factors g(vo,) and g(Yo,) of head and tail objects to the factor ((r) of their relation), which can be efficiently implemented by matrix multiplication as in message passing neural networks [8]. In practice, we simultaneously update all factors in a single iteration based on the states of factors in last iteration, i.e. performing asynchronous message passing in mean field [41 (4), which forms an efficient iterative update scheme. We analyze the efficiency and efficacy of this update scheme in Secs. 6.1 and 6.2
Inference algorithm. The whole infer- ence algorithm is summarized in Alg. Upon on the input scene image I, we first initialize each factor in (G)) by Eqs. (10) and [II. After that, we per-
Algorithm 1 Inference algorithm of JM-SGG.
Input: Scene image 7, iteration number Nr-
Output: Factors {g(y))) }. [4(r)) of (G)).
Initialize (4(y))) {((r)) by Eqs. (T)).
form factor update for Nr iterations. each iteration, the log-optimum of each factor is computed based on the factors of last iteration by Eqs. (12 and (33) and the normalized factors are then de- rived by softmax for update.
for t Nr do
Derive {log 9* (vo)}, {log (r)) by Eqs. (12
13
Update all factors:
{q(V))} + pooitmax(log q" (yo))}, {q(r)} t (softmax(log 9" (r))).
end for
Sampling strategy. After such an iterative inference, we obtain a factorized variational distribution ((G) which well approximates the conditional distribution p(G]7) defined by JM-SGG model. Now, instead of sampling from the intractable model distribution p(G)T), we can easily sample scene graphs from q(G) by independently drawing each object/relation label from the corresponding factor (i.e. ((y,) or q())), where each factor is a categorical distribution. In practice, we sample Ns scene graphs from (G) for each image in mini-batch, yiclding totally N5 samples for estimating the second expectation term in VeC(e) (Eq. (8)) where NB denotes batch size.
Prediction strategy. At the test time, we need to infer the scene graph with the highest probability in PGG[T), and it can also be efficiently done using the variational distribution (G). In specific, based on the factorized definition of g(G). we can easily select the object category (or relation type) with the highest probability each factor ((y.) (or q(r)). and the selected object and relation labels together form a scene graph that well approximates the most likely scene graph with respect to the model distribution TT) Similar prediction strategies have been widely used in previous works that employed field methods [15 40).

---

5 Experiments
5.1 Experimental Setup
Dataset. We use the Visual Genome (VG) dataset [6 (CC BY 4.0 License), a large-scale database with structured image concepts, for evaluation. We use the pre-processed VG from Xu et al. 48 (MIT License) which contains 108k images with 150 object categories and 50 relation types. Following previous works (53 36 37, we employ the original split with 70% images for training and 30% images for test, and 5k images randomly sampled from the training split are held out for validation.
Evaluation tasks. We evaluate the proposed method on two widely studied tasks:
Relationship Retrieval (RR). This task examines model's comprehensive capability of localizing and classifying objects and their relationships. It is further divided into three sub-tasks from easy to hard: (1) Predicate Classification (PredCls): predict the predicate/relation of all object pairs using the ground-trufe bounding boxes and object labels; (2) Scene Graph Classification (SGCIs): predict all object categories and relation types given the ground-trutt bounding boxes; (3) Scene Graph Generation (SGGen)) localize the objects in an image and simultaneously predict their categories and all relations, where an object is regarded as correctly detected if it has at least 0.5 IoU overlap with the ground-truth box. Since two evaluation protocols were typically used in the literature, we adopt two metrics in our experiments, i.e. computing the recall for each relation type and reporting the mean (mR@# k) 21 48 (53 and computing a single recall for all relation types (R@ k) 4. 37 (44,, where we use both 50 and 100 for k as in previous works 21 48] 4
Following Xu et al. [88] we apply the graph constraint that only one relation is obtained for each ordered object pair. Totally, we report model's performance on 12 configurations.
Zero-Shot Relationship Retrieval (ZSRR). This task was first introduced by Lu et al. 24 to evaluate model's ability of identifying the
triplets that have not been observed during training. For this task, we employ the metric Zero-Sho Recall & (ZSR@ k) and conduct evaluation under three settings, i.e. PredCls, SGCIs and SGGen. Also, the configurations where &equals to 50 and 100 are both evaluated.
Performance comparisons. We compare the proposed method with existing scene graph generation algorithms,, including IMP+ 48 (a re-implemeentaiion of IMP by Zellers al. (33), VTransE (55), Motifs [53] KERN 4 VCTree (36]. VCTree-TID TTre (37), VCTree-FBM 34 and GB-
FREQ 53
Net-B
52
We adapt the results on the metric mR @ k from original papers, and the results the
metric R @ and ZSR @ k are evaluated by the released source code for some methods, i.e. VTransE, VCTree-TDE and VCTree-EBM on R @ k. and KERN and GB-Net-3 on ZSR@ k.
5.2
Implementation Details
52 we adopt the Faster R-CNN (8 with a VGG-16 (32 backbone as object detector, and the VGG-16 backbone is initialized with the weights of the model pre-trained on ImageNet [30]. We use the same detector confi configuration as Zellers et al. [53] for fair comparison. The dimension D of object and context space and the dimension K of relation space are both set as 4096, iee. the output dimension of the fc7 layer of VGG- Our method is implemented under PyTorch (25). and the source code will be released for reproducibility.
Model details. Following previous works 48 55 53
Training details. our experiments, the object detector is first pre-trdined by an SGD optimizer (batch size: 4, initial learning rate: 0.001, momentum: 0.9, weight decay: 5 x 10-1) for 20 epochs, and the learning rate multiplied by 0.1 after the 10th epoch. During maximum likelihood learning, we train the potential functions and fine-tune the object detector with another SGD optimizer (batch size: 4. potential function learning rate: 0.001, detector learning rate: 0.0001, momentum: 0.9, weight decay: 5 x 10 +) for 10 epochs, and the learning rate is multiplied by 0.1 after the 5th epoch. Without otherwise specified, the iteration number Nr is set as for training and for test, and the per image sampling size N3 is set as 3. These hyperparameters are selected by the grid search on validation set, and their sensitivities are analyzed in Sec. 6.2
An NVIDIA Tesla V100 GPU is used for training.
Evaluation details. As stated in Sec. 4.3 we independently predict each object category and relation type by selecting the most likely one in the corresponding factor of variational distribution. The objects predicted as "background" are discarded along with the relations linking to them, and the relations predicted as "no relation" are also removed. To derive a ranked triplet list for RR and ZSRR tasks, we save the probability of each object and relation and compute the probability product
7

---

Table I: Relationship Retrieval performance of various methods in terms of R @@ k: and mR@k
Task
PredCIs
SGCIs
SGGGen
Mothod
mR m-00
mR 100
R@
R@
mR 50
mR
100
R
R 100
mR 50
mR
100
R@500
R 100
VTranSE
17.1
18.6
60.7
62.1
8.7
35.0
35.7
22.3
25.9
FREQ 58
13.3
15.8
59.9
64.1
6.8
7.8
32.4
34,0
4.3
5.6
23.5
27.6
Motifs
13.3
14.4
65.2
67.1
7.1
7.6
35.8
36.5
5.3
6.1
27.2
30.3
KERN 4
17.7
19.2
65.8
67.6
9.4
10.0
36.7
37.4
6,4
7.3
27.1
29.8
VCTice 50
17.9
19.4
10.8
38.1
8.0
27.9
31.3
VCTree TDE
VCTree-EBM
18.2
66.8
68.3
12.5
38.7
39.2
9.1
28.0
31.3
GB-Net-B [2
22.1
24.0
66.6
68.2
12.7
13.4
37.3
38.0
7.1
8.5
26.3
29.9
JM-SG (triplet)
23.0
25.6
68.5
69.3
11.6
13.2
41.9
42.5
8.4
10.6
28.0
31.2
JM-SGU (w/o FU)
23.7
26.5
69.1
70.2
12.1
13.6
42.7
43.2
8.7
10.5
28.2
31.3
JM-SGG
24.9
28.0
70.8
71.7
13.1
14.7
43.4
44.2
11.8
29.3
32.2
within each head-relation-tai triplet, and all triplets are then ranked according to the values of their probability products descending order. We report model's performance the last epoch.
5.3 Experimental Results
Relationship Retrieval (RR). In Tab. we compare our method with existing approaches under 12 settings of the RR task. It can be observed that the proposed JM-SGG model achieves the best performance on 10 of 12 settings. In particular, compared the state-of-ther -tetthh-ttt VCTree- (37], previous work dedicated to addressing unbiased scene graph prediction, JM-SGG performs better on 4 of 6 settings for unbiased prediction (i.e. the settings using metric mR k). We think these superior results are mainly ascribed to the proposed joint scene graph modeling, in which the class imbalance among different relation types mitigated by emphasizing the role of these sample-scarce relation types under the context of whole scene graphs.
Zero-Shot Relationship Retrieval (ZSRR). Table 2: Zero-Shot Relationship Retrieval perfor-
Tab. reports the performance of various ap-
mance of various methods in terms of ZSR @ k.
proaches on 6 settings of the ZSRR task. The
Task
PredCls
SGCIs
SGGen
comparison with FREQ (53 not included on this task, since this baseline method can only predict the relational triplets appearing in the training set. We can observe that the JM-SGG model outperforms cxisting methods on all settings, and, especially, 34% performance gain on ZSR 50 is achieved on the SGCIs sub-task. These results illustrate the effective- ness JM-SGG on discovering the novel re- lational triplets that have been observed during learning.
Method
@50
@100
@100
@
@100
IMP+
[48]
12.6
15.2
3.3
3.8
0.5
1.2
VTransF: 55
11.3
14.7
2.5
3.3
0.8
1.5
Motifs 53
10.9
14.5
2.2
3.0
0.1
0.2
KERN 4
12.3
15.4
3.8
1.4
VCTree 80
10.8
14.3
2.6
0.7
VCTreeTDD
37
VCTree EBM
34
11.6
15.7
5.1
2.5
3.4
GB-Net-3 52
12.8
16.0
3.4
4.1
2.0
2.6
JM-SGG (triplet)
12.7
16.3
3.7
4.8
2.0
3.1
JM-SGG (wlo FU)
14.2
18.0
4.2
5.5
2.3
3.2
JM-SGG
14.9
18.6
6.2
2.9
3.6
6
Analysis
6.1
Ablation Study
Ablation study for joint scene graph modeling. To better verify the effectiveness of joint scene graph modeling, we study variant of JM-SGG which models the joint distribution of an individual relational triplet instead of the whole scene graph, denoted as JM-SGG (triplet) (see supplementary material for more details). In Tabs. and JM-SGG clearly outperforms JM-SGG (triplet) all metrics including the metric mR @k for unbiased prediction, which demonstrates the benefit of joint scene graph modeling on mitigating the class imbalance among different relation types.
Ablation study for factor initialization. In this experiment, we compare the proposed initialization method (Eqs. (10) and (II) with the random initialization which randomly initializes the categorical distribution for each factor g(y.) and ((r) variational distribution gG)). Under these two initializa- tion schemes, we respectively plot model's performance after different iterations of factor update in Fig. a).. After four iterations, two schemes converge to the solutions with comparable performance, while initialization approach shows faster convergence (i.e. converge after two iterations).

---

13.6
18.5
10
mR355
13
Fandum Durn
11.6
Tax 11.5
0
3
10
Iteration
Ng
(a)
(b)
(c)
Figure 2: (a) Comparison between random initialization and the proposed initialization scheme. (b) Sensitivity analysis of iteration number Nr during training and test. (c) Sensitivity analysis of per image sampling size Ns. (All results are reported on SGCIs task under the metric mR (5.)
Table 3:
Ablation study on modeling head-relation-taii triplets.
Task
PredCis
SGCIs
SUGin
Method
m0
mR 100
R@-0
R@:100
mR@30
mR@
100
R@:0
R
100
mR
mR 100
R@
R 100
(TransE)
22.7
68.8
JM-SGG
24.9
28.0
70,8
71.7
13.1
14.7
43.4
32.2
Ablation study for factor update. In this part, we study another configuration where the initialized factors are directly used for scene graph prediction without factor update, denoted as JM-SGG (w/o the superior performance of JM-SGG over JM-SGG (w/o FU) verifies the necessity of performing factor update to refine the initial label predictions.
FU). In Tabs. [ and [
Ablation study modeling head-relation-tal triplets. Previous works 55 used TransE 2 to model the relation between two objects, while our method employs TransR 20 to model head- relation-tail triplets. To investigate the effectiveness such a model design, we substitute TransR with TransE in our model, named as JM-SGG (TransE). Specifically, this model variant regards object and relation embeddings lie in the same space, and thus the projection matrix M, is removed from two relation potential terms visuull and Vrriplet. In Tab. it can be observed that TransR clearly outperforms TransE in the JM-SGG model, which demonstrates the importance of modeling objects and relations in two distinct embedding spaces.
Sensitivity Analysis
Sensitivity of iteration number Ny. In Fig. (b), plot the performance of JM-SGG model under different iteration numbers. It can be observed that, for training, one iteration of factor update is enough derive a decent variational distribution for the sampling purpose: for test, two iterations are required to converge to the optimal approximation of the model distribution.
Sensitivity of per image sampling size Ns. We vary the value of per image sampling size N₂ for learning and plot the corresponding model performance in Fig. We can observe that through sampling least three scene graphs from the variational distribution for each image, the second expectation term in Eq. 8 can be well estimated, which stably enhances model performance.
6.3 Visualization
In Fig. visualize the typical scene graphs generated by JM-SGG model, in which the results with and without applying factor update are respectively shown. these two examples, factor update succeeds in correcting
wrong relation labels (e.g. person has jean - person wearing jean) by considering the dependency among different object and relation labels. More visualization results provided in the supplementary material.
7 Conclusions and Future Work
In this work, we propose the Joint Modeling for Scene Graph Generation (JM-SGG) model. This model is able to jointly capture the dependency among all object and relation labels the scene graph, and its learning and inference can be efficiently performed using the mean-field variational inference

---

fde
Jaan sabbar
Socke updcr
BM Senor aslar
Soow updce
Figure 3: The scene graphs generated by JM-SGG model. (Red labels in scene graphs are wrong.)
algorithm. The extensive experiments on both relationship retrieval and zero-shot relationship retrieval tasks demonstrate the superiority of JM-SGG model.
The current JM-SGG model cannot be directly used for visual reasoning, and its inference method makes a strong assumption of fully factorized variational distribution. Therefore, our future work will include exploring downstream visual reasoning tasks (e.g. visual question answering [ and visual commonsense reasoning [54)) based JM-SGG model and further improving our approximate inference algorithm (e.g. by defining more expressive variational distribution).
Broader Impacts
This research project focuses predicting objects and their relations in visual scene by fully capturing the dependency among all objects and relations, and the predicted object and relation labels further organized as scene graph. Compared the conventional visual recognition systems that only predict objects, our approach is able to simultaneously provide object and relationship prediction. This merit enables applications, like intelligent surveillance and autonomous driving.
in-depth scene understanding and can potentially benefit many real-world
However, it cannot be denied that the annotation process for scene graph generation model is labor-intensive. For example, 11.5 objects and 6.2 relations, on average, are required to be annotated for each image in the Visual Genome dataset, and the dataset contains 108k images in total. Therefore, how to train a scene graph generation model in a more efficient way by using less labeled data remains to be further explored.
Acknowledgments and Disclosure of Funding
This project was supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ldt., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bine Gift Fund and a NRC Collaborative R&D Project FA44D-COREAA This project was also partially funded by IVADO Fundamental Research Project grant PRF2019-33 3133333137722 Binghing Ni is supported by National Science Foundation of China (U20B2072, 61976137).
The authors would like to thank Zhaocheng Zhu, Louis-Pascal Xhonneux and Zuobai Zhang for providing constructive advices during this project, and also appreciate the Student Innovation Center of SJTU for providing GPUs.
10

---

References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: visual question answering. In International Conference Computer Vision, 2015.
[2] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, 2013.
[3] Miguel Á. Perreira-Perpiñän and Geoffrey E. Hinton. On contrastive divergence learning. In Robert G. Cowell and Zoubin Ghahramani, editors, International Workshop on Artificial Intelligence and Statistics, 2005.
[4] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing network for scene graph generation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[5] Weilin Cong, William Wang, and Wang-Chiee Lee. Scene graph generation via conditional random fields. CoRR, abs/1811 1000075, 2018.
[6] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In IEEE Conference on Computer Vision and Puttern Recognition, 2017.
[7] Asja Fischer and Christian lgel. Training restricted boltzmann machines: An introduction. Pattern Recognition, 47(1):25-33) 2014.
[8] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, 2017.
[9] Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[10] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In Interna- lional Conference on Computer Vision, 2017.
[12] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1711-1000, 2002.
[13] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[14] Boris Knyazev, Harm de Vries, Catalina Cangea, Graham W. Taylor, Aaron C. Courville, and Eugene Belilovsky. Graph density-aware losses for novel compositions in scene graph generation. In British Machine Vision Conference, 2020.
[15] Patrice Kochl and Mare Delarue. Application of a self-consistent mean field theory to pre- dict protein side-chains conformation and estimate their conformational entropy. Journal of Molecular Biology, 239(2):249-277 1994.
[16] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, (23()):32-73,201)
[17] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. International Conference on Machine Learning, 2001.

---

[18] Wei Li, Xiatian Zhu, and Shaogang Gong. Harmonious attention network for person re- identification. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[19] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In IEEE Conference on Computer Vision and Putlern Recognition, 2020.
[20] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI Conference on Artificial Intelligence. 2015.
[21] Cewu Lu, Ranjay Krishna, Michael S. Bernstein, and Fei-Fei Li. Visual relationship detection with language priors. In European Conference on Computer Vision, 2016.
[22] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: Dimension-fieee bounds and kernel limit. In Conference on Learning Theory, 2019.
[23] Amir Mukhtar, Likun Xia, and Tong Boon Tang. Vehicle detection techniques for collision avoidance systems: A review. IEEE Transactions on Intelligent Transportation Systems, 16(5): 2318-2338, 2015.
[24] Manfred Opper and Ole Winther. Gaussian processes for classification: Mean-field algorithms. Neural Computation, 12(11)225552268, 2000.
[25] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NeurlPS Workshop, 2017.
[26] Quang- Haingg-Heuuua Pham, Thanh Nguyen, Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. Jsis3d: Joint cemantic-instant segmentation of 3d point clouds with multi-task pointwise networks and multi-value conditional random fields. In JEEE Conference on Computer Vision and Pattern Recognition, 2019.
[27] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[28] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2015.
[29] Stefan Roth and Michael Black. Fields of experts: framework for learning image priors. In IEEE Conference Computer Vision and Pattern Recognition, 2005.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, (1533)2211-2222 2015.
[31] Lawrence Saul, Tommi S. Jaakkola, and Michael I. Jordan. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4:61-76, 1996.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[33] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[34] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gérard G. Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. CoRR, abs/2103.022211 2021.
[35] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, 2019.
12

---

[36] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[37] Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[38] Tianyu Tang, Shilin Zhou, Zhipeng Deng, Huanxin Zou, and Lin Lei. Vehicle detection in aerial images based region convolutional neural networks and hard negative example mining. Sensors, 17(2))336, 2017.
[39] Marshall F Tappen, Bryan C Russell, and William Freeman. Efficient graphical models for processing images. In IEEE Conference on Computer Vision and Pattern Recognition, 2004,
[40] KF Tiampo. JB Rundle, S McGinnis, SJ Gross, and W Klein. Mean-field threshold systems and phase dynamics: An application to earthquake fault systems. Europhysics Letlers, 60(3):481, 2002.
[41] Alessandro Treves. Mean-field analysis of neuronal spike dynamics. Network: Computation in Neural Systems, 4(3):259 284, 1993.
[42] Raviteja Vemulapalli, Oncel Tuzel, and Ming- Yu Liu. Deep gaussian conditional random field network: A model-basee deep network for discriminative denoising. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
[43] Raviteja Vemulapalli, Oncel Tuzel, Ming- Yu Liu, and Rama Chellapa. Gaussian conditional random field network for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[44] Vibhav Vineet, Jonathan Warrell, Paul Sturgess, and Philip H. S. Torr. Improved initialization and gaussian mixture pairwise terms for dense random fields with mean-field inference. In British Muchine Vision Conference, 2012.
[45] Vibhav Vineet, Jonathan Warrell, and Philip Torr. Filter-based mean-field inference for random fields with higher-order terms and product label-spaces. International Journal of Computer Vision 110(3)290-337, 2014.
[46] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[47] Eric P. Xing, Michael I. Jordan, and Stuart Russell. A generalized mean field algorithm for variational inference in exponential families. In Conference in Uncertainty in Artificial Intelligence, 2003.
[48] Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[49] Yichao Yan, Qiang Zhang, Binghing Ni, Wendong Zhang, Minghao Xu, and Xiaokang Yang. Learning context graph for person search. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[50] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruy Batra, and Devi Parikh, Graph R-CNN for scene graph generation. In European Conference on Computer Vision, 2018.
[51] Ze- Huan Yuan, Tong Lu, Yirui Wu, et al. Deep-dense conditional random fields for object co-segmentation. In International Joint Conference on Artificial Intelligence, 2017.
[52] Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In European Conference on Computer Vision, 2020.
[53] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
13

---

[54] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference Computer Vision and Pattern Recognition. 2019.
[55] Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embed- ding network for visual relation detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[56] Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. In IEEE Conference on Computer Vision and Puttern Recognition, 2019.
[57] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects points. CoRR, abs/ Abs/1904 2019.
14


Reviews:
[{"review_id": "jABZ-Rekwu6", "reviewer": "Reviewer_PwTs", "summary": "* The paper proposes a method for generating scene graphs of images by modeling the objects and relationships with a Conditional Random Field. The paper claims that other existing methods do not model the dependency between all the objects and relations in an image. The proposed method is meant for modeling all the relations jointly which could lead to better performance in generating scene graphs.\n* The potential function for object components is defined using the representation obtained from an object detector network. The potential function for relationship components is defined using the knowledge graph embedding method which represents both the objects and relationships in the same embedding space.\n* A mean-field approximation of the probability distribution of the CRF is obtained by treating the labels for objects and relations as independent of each other. A message passing algorithm is used to iteratively update the factors and perform inference with the mean-field approximation. This is also helpful for sampling the scene graphs during the learning phase.\n* The experiments are performed on the VisualGenome dataset and results are shown for a few different tasks like Relationship Predicate Classification, Scene Graph Classification and Generation. The paper also includes results on the task of zero shot relationship retrieval that evaluates the model’s ability to identify head-relation-tail combinations not observed during training. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have a few concerns regarding the Originality, Quality and Clarity. The concerns are marked with the [-ve] prefix.\n\nOriginality\n* [-ve] The claim that other works don’t model all the relationships in an image is not true. The idea of using factors for relationships and aggregating information over all the relationships using an iterative message passing algorithm has been used by W.Cong et al. [4]. The Algorithm 1 in [4] has a message passing step that gathers information from the neighbors of a node in a graph, thus aggregating information from the entire graph over the iterations. The aggregation method is not just special in [4], it is also a pretty standard way to do inference over graphs.\n* [-ve] The paper claims to propose a novel inference algorithm. However, it is not clear what aspect of the inference algorithm is novel when compared to that of existing mean-field approximation for CRF models. This inference algorithm has been used for CRFs in many prior computer vision works. Ex: Krahenbuhl and Kolton, NeurIPS 2011.\n\nQuality\n* [-ve] The results are better than some of the state of the art methods. While the paper claims that this is due to joint scene graph modeling (line 337), I think a more thorough analysis would be required to justify the claim. The effect of the proposed relationship potential function that is different from other works is not fully measured.\n* The definitions of the potential functions seem to be the main distinguishing contribution when compared to works like Cong et al. [4]. The work in [4] uses TransE (Bordes et al. 2013) whereas this paper uses TransR [19]. Could this be the one of the reasons for the improved results? This could be verified with an ablation experiment where TransR is substituted with TransE. I am suggesting this experiment because the results from the ablation settings which don’t perform full scene graph modeling (triplet-only and without factor update) are already better in some settings when compared to the other methods.\n* The results from the model without factor update is better than the triplet only model which does one step of factor update. This is an interesting result which needs some explanation. Does this imply that the results can be noisy if we don’t perform full scene graph modeling? It will be good to mention this explicitly which adds to the motivation of doing full scene graph modeling.\n* The experiments section provides a good sensitivity analysis showing how the results converge with increasing number of factor update iterations.\n* The code is provided for reproducing the results\n\nClarity\n* [-ve] The paper is written well to provide a clear explanation of the method along with the proofs of the mathematical lemmas. However, the related work section is lacking since it doesn’t mention how the proposed method is different and/or better compared to existing works.\n\nSignificance\n* The results are better than the state of the art methods in many settings.\n* The work is useful for the community as an example for an alternative method to perform scene generation. While the state of the art methods like Dhingra et al., CVPR 2021 show much better performance on the same tasks using Transformers, the paper proposed a viable method that uses CRFs. It is beneficial to the community since it answers the question “How would CRF models perform on the task of scene generation?”\n\nOther suggestions\n* The experiments use VGG16 as the backbone for comparison with other methods. However, it will be good to include additional results with Resnet as the backbone since the latest methods have shown improvements when switching from VGG16 to Resnet. Ex: Dhingra et al., CVPR 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have provided a short discussion on the limitations and any societal impact the work might have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "hHWS9tW4OZq", "reviewer": "Reviewer_TbVc", "summary": "The paper proposes a new method for supervised scene graph generation from images. The new method models dependencies between objects and their relations as a conditional random field (CRF), conditioned on the output of an object detector. To facilitate inference in the GRF, the deterministic prediction by the object detector is used to initialize a mean field approximation to the posterior. This is then updated iteratively via message passing. The method is shown to yield improved performance in relationship retrieval on the Visual Genome dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Method\nThe intuition behind the proposed method is clear, and the design choices made in its implementation are plausible. The theoretical analysis is correct as far as I can tell. The use of the mean field approximation is a limitation, but one that is reasonable and also discussed in the paper.\n\n### Originality\nThe introduction and related work section claims that the proposed method is the first to model dependencies across the entire scene graph. I don't believe that this is the case: At the least, Suhail et al. [33] propose an energy based model that uses a graph neural network to assign energy values to scene graphs, which will be able to model arbitrary dependencies between the nodes. They also use a similar inference procedure, by first obtaining an initial scene graph via a deterministic predictor, and then iteratively updating it to maximize the likelihood (energy) under the probabilistic model. While the proposed method may be the first to model dependencies via a traditional graphical model like a CRF, and there may be some benefits to that, such a claim would need to be discussed and supported by evidence.\n\n### Empirical Evaluation\nThe model is evaluated in the standard setting of relationship retrieval on the Visual Genome dataset, with Table 1 indicating that the proposed methods performs almost universally better than previous methods. However, given that the authors report baseline numbers for VCTree, VCTree-TDE, and VCTree with the EBM loss from [33], I am very confused that they chose to not also report the results for VCTree-TDE with EBM loss [33]. This appears to be the current state of the art on this benchmark, and also beats the proposed method on most of the reported metrics. Am I missing any reason why this wouldn't be a fair comparison?\n\nMore generally, the robustness of the evaluation could be improved by including another dataset, such as GQA.\n\n\n### Clarity\nOverall, the proposed method is explained in a clear and reproducible way.\nThe paper could benefit from another editing pass, e.g., the caption of Fig. 2(c) is broken.\nThe readability of some of the math, especially formulas (12) and (13) could be improved, e.g., by avoiding nested subscripts such as $y_{o_h}$, and\nmerging summation signs.\n\n### Summary\nOverall, the paper proposes a plausible method, but the discussion of related work and the empirical evaluation raise questions. I am open to increasing my score if these can be addressed during the response phase.\n\n### Update\nIn their response, the authors have agreed to remove the claim that they are the first to jointly model scene graphs. They have also demonstrated that their contribution is orthogonal to the unbiased prediction techniques employed in VCTree-TDE, and that the two can be combined to achieve even better performance. This addresses my two main concerns. I have therefore raised my score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "_ux_mx0fKc0", "reviewer": "Reviewer_szaq", "summary": "This paper presents a conditional random field (CRF) based joint modeling of objects and relations for the task of scene graph generation. Unlike previous relation discovery methods, this paper employs embedding-based relation feature representation, and thus enables an unified modeling of the unitary and clique potential functions. Using MCMC sampler and mean-field variational inference, the proposed JM-SGG can effectively update the relation triplets utilizing the contextual label dependency. The experiments show that JM-SGG has good performances on the relationship retrieval task, even under the zero-shot setting.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this study is helpful to researchers in the fields of visual understanding, especially scene graph generation. It is happy to see how the probabilistic graph model is successfully integrated into scene graph generation. This paper is well written, clearly illustrated, and appropriately structured. The experimental results also validate the claims.\n\nBut in my opinion, conditioned random field and mean-field variational inference are not at the first time applied into scene graph generation. For example, Cong et al. 2018, Scene Graph Generation via Conditional Random Fields has introduced some important techniques used in this paper. Even though this paper was just `published' in arxiv, it would be appreciated to discuss their technical connections, and how the proposed method outperforms.\n\nAnother concern lies in the experiments. At first, why not in addition report the results by non-graph constraint? Each ordered object pair should contain more than one relation if the relation label cannot convincingly exclude each other. For example, <person, has, jean> is not wrong, even though <person, wearing, jean> may be a more precise description. It would be nice if the proposed JM-SGG can capture the most possible relations for a particular object pair.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed the limitations and potential negative societal impact of this work in the section of the conclusion, and in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Tx5fLOU0pFg", "reviewer": "Reviewer_Bh7i", "summary": "This paper proposes a joint prediction of the entire scene graph that fully captures the dependency among different objects and relations using a unified conditional random field.  The proposed model can be summarized as starting from joint modeling of the object and relation component label and visual features. Joint modeling uses CRF to model comprehensive dependency of the object using unnormalized likelihood and partition function. In addition, the potential function used for object and relation components computed affinity through a distance-based learnable prototype. Finally, the knowledge graph embedding techniques for projection of different embedding to the same embedding space (e.g. context space to relation space, object space to relation space) have been used.\n\nAll the learnable parameters are trained with maximum log-likelihood function and the partition function uses a mean-field variational inference for efficient sampling. Mean-field inference initialized with independent relation labels for triplet component factor update.\n\nAuthors have conducted extensive experiments on the Visual Genome dataset and a brief ablation study that provide a good insight into the method. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper for joint modeling of object and relation using CRF and mean-field variance algorithm is novel and significant in the SGG task. The paper is well written, although the introduction and description of the long list of components hindrance its readability. The combination of CRF and knowledge graph technique will lead to exploration of commonality between scene and knowledge graph, thus making it a relevant work in this area.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are a few limitations of the paper : \n1. Authors should conduct more experiments on relatable datasets like GQA or Open Image.\n2. Ablation focuses on the joint modeling, factor update, initialization, etc, where the paper assumes  CRM and prototypical embedding helped the network. Ablation should provide the impact of these components.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for submitting your work to NeurIPS. The paper introduces a supervised approach to scene graph generation from images: objects and relations among them are captured using a CRF conditioned on a (deep) object detector and using e.g. distance-based learnable prototype. Overall, the rolling discussion helped to clarify many of issues raised by the reviewers. This is a solid but also somewhat incremental paper that, and this is a big advantage of the paper, shows the value of hybrid methods for challenging AI tasks. In any case, please incorporate your feedback from the rolling discussion into the final version. 
Decision: Accept (Poster)