Learning to Recognize the Unseen Visual Predicates
Authors: Defa Zhu
Si Liu
Wentao Jiang
Guanbin Li
Tianyi Wu
Guodong Guo
Venue:  (2020)
Keywords: Visual Relationship Detection
Scene Graph Generation
Knowledge
Zero-shot Learning

Abstract: Visual relationship recognition models are limited in the ability to generalize from finite seen predicates to unseen ones. We propose a new problem setting named predicate zero-shot learning (PZSL): learning to recognize the predicates without training data. It is unlike the previous zero-shot learning problem on visual relationship recognition which learns to recognize the unseen relationship triplets (<subject, predicate, object>) but requires all components (subject, predicate, and object) to be seen in the training set. For the PZSL problem, however, the models are expected to recognize the diverse even unseen predicates, which is meaningful for many downstream high-level tasks, like visual question answering, to handle complex scenes and open questions. The PZSL is a very challenging task since the predicates are very abstract and follow an extreme long-tail distribution. To address the PZSL problem, we present a model that performs compatibility learning leveraging the linguistic priors from the corpus and knowledge base. An unbalanced sampled-softmax is further developed to tackle the extreme long-tail distribution of predicates. Finally, the experiments are conducted to analyze the problem and verify the effectiveness of our methods. The dataset and source code will be released for further study. 

Under review as a conference paper at ICLR 2020
LEARNING TO RECOGNIZE THE UNSEEN VISUAL PREDICATES
Anonymous authors Paper under double-blind review
ABSTRACT
Visual relationship recognition models are limited in the ability to generalize from finite seen predicates unseen ones. We propose a new problem setting named predicate zero-shot learning (PZSL): learning to recognize the predicates without training data. It is unlike the previous zero-shot learning problem on visual relationship recognition which learns to recognize the unseen relationship triplets (ssubject, predicate, object>) but requires all components (subject, predicate, and object) to be seen in the training set. For the PZSL problem, however, the models are expected to recognize the diverse even unseen predicates, which is meaningful for many downstream high-level tasks, like visual question answering, to handle complex scenes and open questions. The PZSL is very challenging task since the predicates are very abstract and follow an extreme long-tail distribution. To address the PZSI. problem, we present a model that performs compatibility learning leveraging the linguistic priors from the corpus and knowledge base. An unbalanced sampled-softma is further developed to tackle the extreme long-tail distribution of predicates. Finally, the experiments are conducted to analyze the problem and verify the effectiveness of our methods. The dataset and source code will be released for further study.
INTRODUCTION
Visual relationship recognition (Johnson et al.) 2015 Lu al., 2016 Xu et al., (017) aims to estimate the relationships between pairs of localized entities, i.e., performing the recognition of triplets <subject, predi cate, object It structurally describes images, which provides rich semantic information of an image to many applications including visual question answering (VQA) (Li et al. 2018), image captioning (Yang et al., 2019) and image retrieval (Johnson et al. 2015). The relationship recognition methods are mainly supervised to recognize the entities and then combine various entities in pairs to identify predicates between them. There is an increasing interest in relationship zero-shot learning (ZSL) that learns recognize the unseen relationship triplets, where the studies (Lu et al. 2016 Yu et al. 2017) on this ZSL problem setting assume the components (subjject pred cate e, and ob jectt) of the relationship triplet are seen. However, almost all of them only focus on dozens of frequent predicates and do not study the generalization of the seen predicates to the unseen ones.
In this work, we propose the predicate zero-shot learning (PZSL) problem setting focusing on rec- ognizing the unseen predicates (no manual annotations image samples). For example, given no instance of chew in the training data, the model is expected to recognize it during testing. Recog- nizing diverse even unseen predicates significant for providing very rich relationship information, describing the complex scenes, and analogizing the known abstract concepts to the novel ones. The solution of the PZSL problem will greatly promote many downstream tasks, such as generating image caption with vivid predicates which are even unseen in the description corpus (image captioning) and answering the open questions (with novel predicates) on the complex scene (VQA).
Although zero-shot learning in image classification has received increasing attention (Larochelle et al. 2008), PZSL is not explored. Furthermore, the PZSL problem more challenging in the following aspects. a) Recognizing predicates is difficult since predicates specific as objects. Analogizing the seen abstract predicates to the unseen ones further escalates the difficulty. Many object 7SL. methods (Lampert al. 2014 2009) adopt the pre-defined attributes of
often abstract not as

---

Under review as a conference paper at ICLR 2020
Common space
destroy.v.02
destroy.v002
destroy.v.02
bite. v.
bite.v.01
bite.v.01
vat.v.01
v.0.
eatv:04
kill.v.01
killv.01
kill.v.01
chew.v.01
chew.v.01
Aligning
Matching
Training set
Knowledge graph
Test set
Figure 1: A basic model for recognizing unseen visual predicates. The visual data and knowledge graph's nodes are mapped into a common space by a visual and knowledge module respectively, where the sub-spaces from visual and knowledge module embedding space correspondingly. Note that the visual predicate feature contains features of the subject, object and the union of them. The basic model contains two stages. First, the visual feature and semantic embedding space are aligned by taking the seen predicates in the training set as anchors, the so-called compatibility learning. Second, the samples in the test set mapped into visual feature space and matched with the nearest predicate neighbor from the semantic embedding space.
named with visual feature and semantic
objects to recognize the unseen object. However, it is difficult to define the attributes of predicates. b) Predicates of existing datasets follow an extreme long-tail distribution (92.26) 92.26% predicates with the number of instances lower than 10 in Visual Genome (Krishna et al.) (017)). The statistics of the dataset (VG-Zero) in this work are shown Fig. Under this distribution, the model tends to collapse to output few frequent predicates. Note that if the infrequent predicates are not recognized, unseen predicates are more unlikely to be recognized.
To address the P7SL problem, we introduce a basic model to perform compatibility learning (Frome jct al., [2013] Akata et al., 2016 2015) (Fig. [Tp, leveraging the linguistic priors from the corpus and knowledge base (Wang et al. 2018 Kampilmeyer et al. 2018). To represent the abstract predicates, we adopt the pre-trained word (sentence) vectors to initialize the predicates, connect them with linguistic relations defined in knowledge bases, and map them into semantic embedding space (middle of Fig D. A visual module is then applied to map paired image regions (left of Fig. I into visual feature space. The visual feature and semantic embedding spaces fall in the common space (top of Fig. During training, the visual feature and semantic embedding space are aligned with the seen predicates as anchors, i.c., visual feature and semantic embedding labeled with the same predicate fall onto the same point/area in the common space. During testing, the samples in the test set are mapped into the visual feature space and matched with the nearest neighbor semantic embeddings of predicates (like chew). Furthermore, to tackle the long-tail distribution, an unbalanced sampled-softman developed adjust the gradient penalty of the infrequent predicates.
The main contributions include: a) We define the predicate zero-shot learning (PZSL) problem and introduce the corresponding dataset (based on Visual Genome). b) We propose a basic model to address the PZSL problem by compatibility learning leveraging the linguistic priors from the corpus and knowledge base. c) We develop an unbalanced sampled-softme for handling the extreme long-tail distribution of predicates. d) The dataset and source code of this work will be released.
2 RELATED WORK
Visual relationships have been studied from various aspects including statistical motifs (Zellers) ct al., 2018), antity-relationsiig dependencies (Xu et al. 2017), spatial priors (Dai et al. (017). language statistics (Li et al. 2017). Almost all of them focus recognizing dozens of the most frequent predicates. By contrast, our work explores to train a model with about 1000 predicates and test it with about 100 unseen predicates. The two most relevant problem settings are relationship zero-shot learning setting (Lu et al. 2016) and open vocabulary setting (Zhang et al.] 2018). Lu et al. (2016) try recognize the unseen relationships (e.g., <elephant, stand on, strrett> by
2

---

Under review as a conference paper at ICLR 2020
transferring knowledge from similar relationships (e.g., <dog, stand on, street>). Note that all the test predicates and entities are seen in the training set. By contrast, the main difficulty in problem setting is that all test predicates are unseen in the training set. Zhang al. (2018) perform visual relationship recognition with an open vocabulary setting focusing on large-scale recognition problem without study on ZSL.. The most relevant approach in visual relationship recognition is Graph R-CNN (Yang et al. 2018) which adopts graph convolutional network (GCN) (Kipf & Welling 2016) to capture contextual information between visual objects and relations. By contrast, we utilize GCNs to leverage the relations of categories to generate meaningful semantic embeddings.
Zero-shot learning (ZSL). To recognize unseen objects, compatibility learning frameworks (Frome) it alll,,201 2013; Fu et al. 2015; Fu & Sigal 2016) map visual and semantic features into the common space and align the visual and semantic manifolds with the seen categories. During testing, these methods recognize the given visual feature by performing a nearest neighbor search on the semantic embeddings of the categories. Recent works (Wang et al. 2018 Kampffmeyer et al. 2018) utilize the linguistic relations between seen and unseen categories in knowledge graph (KG) for ZSL. Wang ct al. (2018) propose to train a GCN supervised by the classifier's s weights of neural network. Our method can be considered as hybrid of compatibility learning frameworks and knowledge graph based methods. The node embedding of KG is mapped into the same space with the visual feature.
External knowledge bases (KB), such Wikipedia and ConceptNet (Speer & Havasi 2013). has
been introduced in visual relationship recognition to provide linguistic and commonsense priors. Yu et al.] (2017) extract the <sub ject, predi cate, object: triplets from Wikipedia and leverage the statistics P(pred|sub obj) to help recognizing the unseen relationship triplets. Baier al. (2017) train a semantic model from KB improve the performance. [uu et al. (2019) 9) takes the detected objects to retrieve on the ConceptNet to obtain a set of triplets to enhance the visual features. Unlike the prior works essentially using the statistics of <sub ject, predi cate, ob jectt> from the external KB, our work leverages the lingnistic relations of predicates defined in WordNet (Miller 1992) to explicitly connect the predicates, such as <attack, is a hyponym of, fight>, for recognizing the unseen predicates.
3 PROBLEM SETUP
Setting: Let the full predicate vocabulary as Vpred = pred U U pred and entity vocabulary as Ven. where Dtr prea and Pirea are the training and test predicate vocabulary respectively, and "entity" refers to "subject" and "object". The training and test predicates are disjoint, i.e., Vired n pred = Φ. The dataset is denoted as D = {Li, (bg Sagi Pty: b,,, o;g))} where 8ij, Ozg E Ven denote subject and object labels of the j-th relationship in i-th image I, (the green box in Fig. 2 (A)), bīy, o2, the corresponding boxes of the subject and object, and Ptj E Virea UVee pred is the corresponding predicate label. Any image that contains a test predicate p E pred is assigned to the test set D.e. and only the regions with test predicate p E Vie pred are used for evaluation. The rest data is split into the training set Der and validation set D During testing, given an image and pair of subject and object boxes from the test set Dre, the model (trained on D(r)) recognizes triplet <sub ject, predicate, ob objcc,>, where the accuracy of predicates is in concern.
Assumption: We assume that, for any test predicate Pie Vrrea: there exists training predicate
Ptr E Vireed having semantic association with Pte. For example, chew is a test predicate, the predicates meaning an action using teeth (like bite) or intaking food (like eat) is expected to be included the training set. Let the visual features and semantic embedding of bite are aligned, so does that of eat. As result, the visual feature of chew, visual similar to that of bite and eat, is able to match to the predicates likes bi te and eat in the semantic embedding space. To satisfy this assumption, the training predicates should be in large-scale to cover as much semantics as possible.
4 APPROACH
In this section, first present the pipeline of our basic model, then the Simplified PinSage (a fast graph convolutional network) for propagating on the large-scale knowledge graph, and finally unbalanced campled-softmat to handle the extreme long-tail distribution.
3

---

Under review as a conference paper at ICLR 2020
ML
Leentrr
Simplified PinSage
Object Predicate
Seen
Unseen
Vx
MLP,
PCp
Entity
ppredicate
Simplified PinSage
MLP
Predicate
Faster-RCNN Vo
Ep
Word Vector
Image
Visual module
Loss
Knowledge module
WordNet
(A) Pipeline
Target
First-order
Secund-arder
Unsampled
* Sampled sub-griph
exp(V, p Ep)
layer I:NN (w)
layer 2: (u)
Frequency
8, & 8. &y Or
ExeR, exp Ex.)
Sampling
#Neighbor -
#Neigghor
Predicate ID
402
Simplified PinSage
(C) Unbulanced sampled
Figure 2: A) The pipeline of our basic model. The visual features (from the visual module) and corresponding semantic embeddings (from the knowledge module) are constrained to close. B) Simplified PinSage. We sample the graph nodes in two aspects during training. Take 2-layer GCN as an example, for on-demand sampling, to get the final embedding of the target node u, the neighbors of u are needed. Backtracking in this way, For the neighbor-limit sampling, the number of each node's neighbors is limited by a constant. C) Unbalanced sampled-softmax. To tackle the long-tail distribution problem, negative predicate batch B, is sampled from the predicate distribution to calculate the loss function.
only need first and second-orde
neighbors of
4.1 PIPELINE
The pipeline our method consists of visual and knowledge modules, refer to Fig They are modeled by Faster R-CNN (Ren al. 2017) and two GCNs (Kipf & Welling 2016 respectively.
Visual module aims to extract visual features of entity and predicate. Given an image as input, the corresponding features of the subject, object and context region are cropped out, where context region refers to the union of subject and object regions. All these feature regions are ROI aligned (He et al. 2017) as ROI features with fixed size (7 x 7). The ROI features of subjects (blue) and objects (yellow) are then mapped into visual entity features V, and Vo by the same multilayer perceptron MLPen- Furthermore, the ROI feature of context region (green box) is fed to MTPP, and the output which is fused with V, and V. to generate visual predicate feature Vp. All these features V,, V. and V will be aligned with the corresponding semantic embeddings E., E, and Ep.
Knowledge module aims to generate the meaningful semantic embeddings of the categories (including predicates and entities). To introduce the language prior and implicit association the categories, embeddings of the categories are initialized with word (sentence) vectors pre-trained on large-scale corpus (like Glo Ve (Pennington et al. 2014)). The categories are then connected by the linguistic relations (defined by KB, WordNet) to build knowledge graph. Note that many categories not directly connected, but can be indirectly connected through categories within the KB outside the dataset. Thus the knowledge graph contains huge number of nodes and provides rich linguistic information. Taking the knowledge graph as input, the semantic embeddings of predicate E, and entity E.n. are generated by graph convolutional networks GCN, GCN respectively. For this part, the Simplified PinSage is introduced for fast processing on the large-scale graph, refer to 4.2
Loss function defined as a summation of the entity and predicate terms as follows:
-= =-entiity + Lpredicntes
(1)
where Epredicate is designed with the proposed unbalanced sampled-sote softmaxx for tackling the long-tail distribution of predicates, refer to 4.3 while Lentity is negative log likelihood with softmax:
exp Ec))
Centity Evv.. ΣkEV.. exp(Ve Ex)
(2)
where Ven is the label of the visual entity feature V..
4

---

Under review as a conference paper at ICLR 2020
4.2 SIMPLIFIED PINSAGE
Simplified PinSage is adopted to map the knowledge graph into an embedding space for PZSL, see Fig2 Inspired by PinSage (Ying et al. 2018), the propagation algorithm divided into three steps: message passing, skip shortcut and normalization. We denote FCw.B o Wa + B, and the process of graph propagation is formulated as follows:
k-1 Zu
1
Σ PEN(")
(FC,, ,ig o h
(3)
[N(u)
A = ReLU o 1am h 'D).
(4)
1 /2/1//ll
(5)
where Eq. (3) and (5) indicate message passing, skip shorteut and normalization respectively, (u) in Eq. (5) denotes the neighbor set of u (u also falls into Nu)),, [.--] in Eq. (4) means "concatenate", ni is the initial embedding, and h is output of k-th graph propagation layer. For n-layer GCN, Beu. takes hü input to get the final embedding E lying in the same space with the visual feature V. Note that the main computation
forward propagation related to the number of edges (Eq. (3). It is too computation and space consuming perform propagation on the whole knowledge graph with about 2.2 billion edges. The graph sampling technology in web-scale
recommender system (Ying et al. 2018 Eksombatchai et al. 2018) is introduced as solution.
On-demand sampling. For generating embeddings of mini-batch categories, we only need to sample necessary sub-graph as input GCN (GCN, GC.Ne... avoiding propagating the whole knowledge graph. Take 2-layer GCN as an example. Only first and second-order neighbors are needed compute the final embedding of the target nodes. Refer Fig. (B), to get the embedding of u (deep blue), the second layer of GCN needs the embeddings (outputs of the first layer) of neighbors (blue) of for message passing, i.e., (N)).. To get the embeddings of N((x) in the first layer, the neighbors of the node set N ((u) are needed (light blue), i.c., (z).. Hence, we only sample a sub-graph containing the node set (u)) (u). general, to get final embeddings of nodes batch L/ for n-layer GCN, only 1st to n-th order neighbors are needed, i.e., LE
Neighbor-limit sampling. Many nodes of the knowledge graph contain a large number of neighbors, which makes computation and space consumption uncontrollable. To further reduce the consumption, we limit the number neighbors per node to a threshold т. i.e., randomly sampling т neighbors. Refer to Fig. 2 (B), the unsampled neighbors (gray) do not contribute to the propagation. For testing, all neighbors are sampled to calculate the final embeddings, and we only need to propagate once to obtain the final semantic embeddings. The experiments in 5 show that neighbor-limit sampling can be considered as a dropout-like operation for greatly avoiding overfitting.
Thus, obtain embedding of k predicates from n-layer GCN, the number of edges not greater than a relaxed upper bound krr", where k T I 100 and n < 3. Thus we have kr S 10 < 2.2 x 10
4.3 UNBALANCED SAMPLED-SOFTMEN
A variant softmax function is proposed to measure the similarity between visual features and semantic embeddings, inspired by sampled softmax (Jean et al. 2014) in machine translation. For training of predicate recognition, if all elements of training vocabulary are sampled as negative embeddings, such as standard softmax, the visual features may always match the few most frequent predicate embeddings since the loig-tail distribution of the predicates in the dataset. The impact of this distribution on zero-shot learning is devastating. To tackle this problem, we propose an unbalanced sampled-softmax (USS):
exp(Vp E.)
S ΣLEB, cxp(V, Fx))
Bn, P Ppred:
where V, is the visual feature whose predicate category is p. E, is the corresponding semantic embedding of predicate i. Unlike sampled softmax that adopts the pre-divided sub-vocabulary as negative categories, the negative predicates B, C Vir is sampled from the predicate distribution
5

---

Under review as a conference paper at ICLR 2020
Ppred. It is possible that B, includes the GT predicate p. Finally, the loss function of predicate recognition is in the form of
Spredicate Eov,,))--og(((S))
The sampling method is vital for Eq. 77. The uniform sampling, degrading into an estimated version of softmax, does not help with long-tail distribution. We design a sampling method to ensure the recognition of predicates with fewer samples so that the model can be further generalized to recognize unseen predicate categories. The idea is that the fewer categories appear as positive categories, the less they are sampled as negative categories. We adopt the frequency of predicates as the probability Ppred to sample the negative predicates, see Fig. 2 distribution problem by adjusting the gradient of the infrequent predicates.
(C). This sampling method handles the long-tail
Let h4 = V E,, the gradient of Sp w.r.t h: is discussed as follows:
a
[ (1pp1 i) -.) if E B, dh, = 11 - i)s,
(8)
if & B,.-
Frequent predicates often fall into the first case in Eq. X which is the same as standard softmax. To the opposite, infrequent predicates always fall into the second case that the reward is increased when is GT (p- = i) and that there no punishment when it a negative predicate (p A i).
EXPERIMENTS
In this section, we start by discussing the datasets, knowledge graph, and implementation details. We then perform the ablation studies to verify the components of our model and visualize our results.
VG-zero dataset. We introduce a new dataset based on the latest released Visual Genome dataset (VG vl.4) (Krishna et al., (2017) which contains 108,077 images with 21 relationships (triplets) on average per image. We manually cleaned up the box annotations in the same way with Xu et al. (2017). Since the original annotation is noisy, 1 1155 synsets in WordNet are used to replace the original predicate categories as regularization, where the correspondence between the original categories and synsets provided in the VG dataset. About 10% of predicates (105 predicates) are selected as test vocabulary. The frequency of the selected predicates falls in a range from 10 to 300, where the lower bound 10 set to guarantee the quality of test set for the infrequent labels are noisy, and the upper bound 300 follows the rule that categories in test set should be least populated or rare (Xian et al. 2017) in zero-shot learning. Images annotated with predicates test vocabulary are selected as the test set (containing 4350 images). We then randomly select 5000 images as the validation set with the rest as the training set. Similar to predicates, the entity categories are also replaced by 7k+ synsets. In addition, we use hypernym relationships cluster entity categories into 96 categories since the entity synsets are still so specific that include numerous names and object recognition not our focus. For example, categories like woman, father are clustered into the person category.
Knowledge graph. The knowledge graph G(V, E) is built from WordNet Miilerr 1992), where V and E are nodes set and edges set respectively. Synsets (synonym set) in WordNet nodes in G. Edge (u, v) is added into & if u and u have one of the following relationships: hypernym, hyponym, part meronym, part holonym, substance meronym, substance holonym, entailment, substance holonym and sharing lemmas. Notice that the self-loop will be included by the "sharing lemmas" relationship. Finally, the resulted knowledge graph contains 101,260 nodes and about 2.2 billion edges.
Implementation details. For all experiments, the model is trained for 150k iterations with batch size set to 4. We set the learning rate as 2e-3 and is reduced by 0.1 times at the 100k and 130k. respectively. We adopt the warmup strategy (Goyal et al. 2017) at the beginning. ResNet50 (He et al. 2016) is used as backbone network with weights pre-trained on COCO (Lin et al. 2014), which is fixed during training. Images are resized such that their short edge is 800 pixels. For the knowledge graph, we use the definitions of synsets as the input of off-the-shelf language models to generate the initial embeddings. More specifically, we use the word (Glo and sentence (InferSent (Conneau et al. 2017)) embedding methods to get the initial embeddings. For the word embedding method, we take every word of the definition as a token to GloVe and average all the word embeddings to get the 300-D embedding. For the sentence embedding method, the whole definition is used as the input of InferSent get the 4096-D embedding. The common space and hidden features in GCN are 512-D.

---

Under review as a conference paper at ICLR 2020
5.1
ABLATION STUDY
Table 1: Accuracy of unseen predicate recognition.
Hits @k (%)
NO.
Propagation Initial embedding
Loss
Generalized setting
Traditional setting
5
10
20
2
5
10
20
W/O KG
GloVe
B, 10
0.0
0.0
0.0
2.5
5.2
13.0
23.1
37.9
2
I-layer GCN
GloVe
B, = 10
1.9
4.7
10.1
5.8
10.7
20.2
32.3
48.5
2-layer GCN
GloVe
B, 10
4.3 7.0
11.3
7.5
12.2
22.2
33.2
48.7
4
3-layer GCN
GloVe
Bn 10
1.9
3.9
8.4
5.3
9.3
19.9
33.3
49.1
5
2-layer GCN
GloVe
Softmax
0.0
0.0
0.0
2.2
4.2
10.8
19.0
33.0
2-layer GCN
GloVe
B, 5
3.2
6.5
10.6
6.5
10.9
20.1
32.5
48.1
7
2-layer GCN
GloVe
B, = 20
4.1
7.4
11.8
8.9
13.0
21.3
32.0
49.5
2-layer GCN
GloVe
B3, 50
2.0
4.7
9.9
5.5
10.4
21.7
32.9
48.6
9
2-layer GCN
GloVe
Bm 100
1.3
3.8
8.5
4.6
8.8
18.3
29.5
46.2
10
T = 5
GloVe
B, = 10
1.9
4.7
10.1
4.2
8.9
20.2
31.7
48.6
11
20
GloVe
13, 10
1.2
4.1
8.7
3.9
8.3
19.8
32.6
50.5
12
T 50
GloVe
B. = 10
2.5
5.3
9.2
4.5
9.1
18.1
30.8
48.8
13
100
Glo
Bn 10
1.9
4.0
7.8
4.4
8.1
17.5
31.4
50.3
14
2-layer GCN
Normal
Bn 10
0.0
0.0
0.0
1.4
2.4
5.0
10.2
19.6
2-layer GCN
InferSent
B, 10
7.0
11.4
7.3
12.4
23.1
35.8
53.5
16
Random guess
0.1
0.8
1.7
0.9
1.9
4.7
9.5
19.0
Table 2: Accuracy of recognition of triplets with unseen predicates.
Hits @k (%)
Methods
Generalized setting
Traditional setting
10
20
2
10
20
W/O KG
0.0
0.0
0.0
2.3
12.5
20.1
Softmax
0.0
0.0
0.0
0.9
1.9
5.0
9.9
17.5
Ours
1.3
2.8
5.8
3.2
6.0
11.6
18.0
26.1
Random guess
4.7e-7
9.4e
1.9e-6
1.0e
2.1e
5.2e
1.0e
2.1e
We evaluate methods with Hit@k on generalized and traditional setting (Tab. [and2): Given any test region pair, we check whether the GT label falls within predicted categories with top k score. If yes, the sample is counted as Hil. The @it@k is #Test region parr The methods are only required to Hit the unseen predicates ignoring the subjects and objects in Tab while they are required to Hit the full relationship triplets in Tab For the generalized setting, the methods search on the seen and unseen predicates vocabulary Ver Vie for inferrence. For the traditional setring, the methods only search on the unseen predicates vocabulary Vie- Note that the generalized setting more challenging since the search space of the generalized setting is significantly larger than that of the traditional one. Observing Tab. and 2, we can draw the following conclusions.
#itt
Knowledge graph prior is critical our algorithm. As shown in the first row of the Tab. I simple 2-layer MLP for semantic embedding, which neglects the relationship of predicates, cannot identify the unseen predicates in the generalized setting accuracies) and can only obtain lower recognition rate on the traditional setting. An l-layer GCN that simply considers node relationships can already deliver significant performance gains (row Tab. []. with large margin (10.6 (%)) on Hit@20 20 traditional setting. The results show that by modeling the explicit connection between predicates, the knowledge can be effectively transferred from seen categories to unseen ones, which results in performance boost on the unseen predicate. The same conclusion can be obtained from Tab,
The unbalanced sampled-softin
effectively tackles the long-tail distribution problem. Refer to row 5 in Tab. [ adopting the softmax loss function results in the worst performance since the outputs collapse into few frequent predicates and could not be generalized to novel predicates. By contrast, adopting the frequency of predicates sampling probability has obvious advantages with 13.7 (%) increase on Hit@ traditional setting (comparing row and in Tab. TT. The same conclusion can be made by comparing row 2 and 3 in Tab.
7

---

Under review as a conference paper at ICLR 2020
a
b
persona.0) at-.a.rwwrrkk. morkaak ww..l---nstrreeeniii nJId
person Ill sow leg indtr untentallttyn. nul3 person.a. Iillag --l--iittrmmaatalii a.63 person.a.0 www.an.1-slgg.... X81 instrmenttaliiss...
ataimal. in Ol-con www.sulllatttra....m 83
person.n www.ww.ll--fedd..oo
wwwnna.l-mark.. 1l--ark...0 www.rr...--inntammae cnstramenttalyy.uu. person.a.l.markk LocoL-inn instramntnn auserlintttllyy....
aininal au www.ool-eo...--oooe.o -tw.tti--ttttraa...... sinimala..01 aillfirrr.a
person.a. Bwwchllw...D-----. St-oo....---
person.n.91 ma-m.orrr-mm...
The images with paired located entities are on the top, where the blue and yellow boxes represent subject and object respectively. The results of the generalized and traditional setting are at the first two rows (correct: blue, wrong: yellow), while the ground truth triplets are at the last row.
Figure 3:
The number of negative categories should be moderate (row and 6~-9). Note that the larger [B, the higher probability that the infrequent predicates fall into B,, e-g., if [Bm]] = Derr U Viel. the USS degenerates to softmax. When [B,| < 100, the performance of models with different [B,,] become similar, while setting with B,, 20 approach the best accuracy (49.5 (%) on Hit 20 traditional setting). However, the performance of the version with [B,1 100 drops significantly (46.2 (%)).
Embedding initialization is necessary, but the impact of different embedding methods perfor- mance is minor. Refer to row 14 in Tab[] we adopt noise following normal distribution to initialize node embeddings, whose results are almost the same as random guessing (row 16). The initial embedding method clusters the semantically similar categories, which implicitly connects seen and unseen categories. These implicit connections are helpful for zero-shot learning. While using different embedding methods results in little difference: the InferSent method enjoys advantages over the GloVe method on the traditional setting with Hit@5~20 but with no advantage in other evaluations.
The numbers of neighbors and layers result in negligible performance. For traditional setting, observing row 10~113 13 in Tab. [ the impact of limited. Similarly, GCNs with different layers achieve almost equal accuracies, refer to row 2-4. For the generalized setting, row 4 (3-layer GCN) and 13 (with 100 neighbors) achieve relatively low accuracy, reducing 2.9 (%) on Hit 20 generalized setting. It can be interpreted that large number of training parameters and complete neighborhood information in the knowledge module make the model overfit the training vocabulary.
5.2 QUALITATIVE RESULTS
Fig 3 shows the results our method, where output is in the form of <sub ject, predicate, object>. The recognition of predicates includes generalized and traditional settings shown row I and 2 (blue/yellow font represent correct/wrong). while the ground truth triplets are displayed in the last row. The case a completely correct in both settings. In case b. our method makes a mistake in the generalized setting while the result of the recognition (swing. v. 01) is close to the ground truth (s (slug. v. 01). This case shows that predicate zero-shot learning the generalized setting is hard for semantically similar categories across training and test vocabularies. The case is so confusing that even humans can make misjudgment. In case our method determines the predicate as chew. v. 01 and outputs more appropriate answer than the ground truth on the recognition of the object. In conclusion, the predicate zero-shot learning is challenging, but our method effective.
6 CONCLUSIONS & FUTURE WORK
In this work, we define a predicate zero-shot learning problem and propose a solution by mapping visual features and semantic embeddings (in knowledge graph) into the common space. Furthermore, graph sampling strategies are introduced for accelerating graph propagation, and an unbalanced sampled-softtax is proposed for tackling the long-tail distribution. Finally, we plan to explore the following future work for this problem. a) Consistency of differences of visual feature and semantic embedding space can be considered to model the cross- cross-modal analogy. b) A semantic-awan negative sampling of predicate categories is solution worth exploring. c) Learning the visual predicate translation vector (Bordes et al. 2013) Wang et al. 2014 Lin et al. 2013) from subject to object such VtransF (Zhang al.] 2017 deserve attempted.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. 7PAMI, 2016.
Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using semantic modeling of scene descriptions. In ISWC, 2017.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, 2013.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loie Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, 2017.
Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In CVPR, 2017.
Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, Charles Sugnet, Mark Ulrich, and Jure Leskovec. Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. In WWW, 2018.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013.
Yanwei Fu and Leonid Sigal. Semi-supervised vocabulary-informedd learning. In CVPR, 2016.
Zhen- Yong Fu, Tao A. Xiang, Elyor Kodirov, and Shaogang Gong. Zero-shot object recognition by semantic manifold distance. In CVPR, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1706.02677, 2017.
Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. In CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In NIPS, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint 1:X1v:1412.2007. 2014.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. arXiv preprint arXiv: 1724, 2018.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.00 029070 2016.
Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object
classes by between-class attribute transfer. In CVPR, 2009.

---

Under review as a conference paper at ICLR 2020
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. TPAMI, 2014.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, 2008.
Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao ou Tang. Vip-cnn: Visual phrase guided convolutional neural network. In CVPR, 2017.
Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou.
Visual question generation as dual task of visual question answering. In CVPR, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, 2015.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, 2016.
George A. Miller. Wordnet: lexical database for english. Commun. ACM, 1992.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: towards real-time object detection with region proposal networks. TPAMI, 2017.
Robyn Speer and Catherine Havasi. Conceptnet 5: A large semantic network for relational knowledge. In The People's 's Web Meets NLP. Springer, 2013.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, 2014.
Yongqin Xian, Bernt Schiele, and Zeynep Akata, Zero-shot learning- the good, the bad and the ugly. In CVPR, 2017.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017.
Jianwei Yang, Jiasen Lu,, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV, 2018.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. CVPR, 2019.
Rex Ying. Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD, 2018.
Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In ICCV, 2017.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embedding network for visual relation detection. In CVPR, 2017.
Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed Elgammal, and Mohamed Elhoseiny. Large-scale visual relationship understanding. arXiv preprint arXiv: 10660, 2018.
10

---

Under review as a conference paper at ICLR 2020
A
ADDITIONAL DETAILS OF VG-ZERO DATASET
Distribution of Predicates
Distribution of Objects
Distribution of Triplets
Frequency (log scale)
Foo fo. for for fx fir
body_parrt.n.01
Frequency (log scala)
have.v03
Frequency (log
fabric.n.00
h.. 4c. tor
gripuuD1
sheet.n.06 -in.01 land.n.04
facilityn.011
repaicu.01
2o.
200
600
ovo
500
1006
40
5000 10000 150te s000 2590e 2000 359ce 4000
Prodicates (sorted by frequency)
Objects (sorted by frequency)
Triplets (sorted by frequency)
Figure 4: Distribution of predicates, objects and relationship triplets.
The process of relabeling. The relationship instances in Visual Genome are not only labeled with the "predicate" (in natural language, e-g., has), but also with the synset (node of WordNet, e.g.. have. v. 01) corresponding to the "predicate". We filter out all relationship instances without labeling synset and use the synset replace the original predicate, c.g. "have.v.01" replace "has". The category of the entity is treated in the same way.
The statisties of predicates, objects, and triplets are shown in Fig. 4 Predicates follow an extreme long-tail distribution. The instances' number of the top 5 predicates count for 80% of the total instances (1155 predicates).
B VISUALIZATION OF THE EMBEDDINGS
Initial embeddings
Final embeddings
Initial embeddings
Final embeddings
edeendaadd3 provide
anchor
NAR 00
happaarax.02
volivOO PR5-VOL
PI
pay.v.01
adheee.v.d
spendd-.2
carofulyy..00
change
Mitxx.1
cradiao.02
evercharge..ot
dismiss..01
consentrativo.oo OZ
SINABLALE
grip.v00
pay.v.01
growt
approsch.
smilexx00
chears:03
sprent EE
comsufllggs.
stedkou0
undrem.vo1
revelx.02
smecko waterooo
plenvvi
veriiouus OL
smifla
smallv81 181
nisex.57
undress.ve v1
sniff.v.01
Figure 5: Comparing the initial embeddings and the final embeddings. The unseen predicates are marked in blue, while the seen predicates are marked in green.
We use SNE to visualize the initial and final embeddings of predicates, where the initial embeddings from GloVe and the final embeddings generated by knowledge module. Refer to Fig. 5 the blue font means the unseen predicates while the green font means the seen ones. We focus on the neighborhood of unseen predicates. It can be concluded that our knowledge module leverages explicit connections to "pull together" the semantically similar predicates.
11


Reviews:
[{"review_id": "rkxZmVqCYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Title: Good work, requires some edits.\n\n1. Summarize:\n\nThis paper proposes a new problem setting in visual relation detection which is called “Predicate Zero-shot Learning (PZSL)”. They provide a clear motivation and description of this setting. They propose a solution to this problem which leverages linguistic priors and knowledge bases. Furthermore they propose an unbalanced sampled-softmax to tackle the long tail distribution of predicates.\n\n2. Clearly state your decision. One or two key reasons for this choice.\n\nI will go for a weak accept for the paper at this stage. (+) I think the proposed problem setting is well-motivated and useful. Also, (+) the proposed initial solution to this problem is interesting. However, (-) they propose a “fast graph convolution network” which seems to be precisely equivalent to a PinSage.  Also, (-) the paper requires to be polished as it lacks clarity.\n\n3. Main discussion\n\nMy first argument is: I’m not sure why the authors have changed the name of PinSage and just mentioned that “their” “Fast Graph Convolution Network” is “inspired” from PinSage. To me it looks exactly the same. If there are any differences, it should be stated clearly. In fact, I would not be against using PinSage as a part of their approach. However, trying to rename it without clear reasons is not a good idea.\n\nMy second argument is that the paper lacks clarity in writing (for detailed suggestions please refer to comments and feedbacks). Specially the evaluation section lacks details and clarity: a) In the beginning of this section (page 7), the authors talk about “generalized” and “traditional” settings without properly defining them. b) The descriptions for Table 1 and Table 2 fail to provide enough details to help understand the difference between the results in these two tables (one of them states “Accuracy of unseen predicate recognition” and the other one “Accuracy of recognition of triplets with unseen predicates”). \n\n4. Comments and feedback.\n\nIntroduction: \n\nParagraph one in the: \n1. The relationship recognition methods are mainly supervised “that” → “to”.\n2. last line: …. and do not study “on generalizing” → “the generalization of”.\n\nParagraph two:\n1. no manual annotations or “real samples” → “image samples”. (a real sample is ill-defined)\n2. For example, no instance of chew → For example “given” no instance of chew.\n\nParagraph three:\n1. … is difficult since predicates are often abstract not as specific →   is difficult since predicates are often abstract “and” not as specific.\n2. Furthermore, unlike many object ZSL methods … → This line to the end is very complicated and hard to understand.\n\nRelated Works:\n\n1. Visual Relationships: I would cite “Graph R-CNN for scene graph generation” since it is the most relevant work regarding the similarity of pipeline (using GCNs).\n2. External Knowledge bases  (KB): I would cite “Improving Visual Relationship Detection using\nSemantic Modeling of Scene Descriptions” since it is one of the most relevant works using knowledge graph modelings to improve visual relation detection.\n\nProblem Setup:\n\nDo you plan to provide the proposed dataset splits so others can work on this setting? I consider this very important given your paper’s contribution. Maybe it is better if it is also mentioned in the paper.\n\nPipeline:\n\n1. Paragraph 2: … the output of which is fused with …: Given Figure 2, it does not seem like $V_p$ is being created by fusing $V_s$ and $V_o$. It looks more like it is extracted directly from the image (union of bounding boxes).\n\n2. In Figure 2: In the representation of Pipeline (A), the graph is colored by dark blue for objects and light blue for predicates. The represented graphs show Object to Object and Predicate to Predicate connections which I’m not sure if it is correct. Shouldn’t we always have a light blue between every pair of dark blue connections?\n\nEvaluation:\n\n1. Please consider the mentioned points in the Main Discussions.\n2. In Table 1, I suggest re-naming “embedding” to “initial embedding”.\n3. In Table 1, Hit@k should be Hits@k.\n4. Please define the metrics clearly (Hits@k).\n\nExtra: I have a question regarding the ablation studies with GloVe, Normal and InferSent initialization. The question is whether this initialization is necessary? It seems like in the setting “W/O KG”, even though the embeddings are initialized with GloVe, there is no gain (all of the Hits@k values are 0.0). So GloVe embedding without KG bring no external semantic knowledge? Then why use them? Regarding that, I can see that a GCN, initialized with normally distributed embeddings (row 14 in Table 1) has given 0.0 accuracies, but I find this very counter intuitive, as graph convolution layers already have trainable weights capable of compensating for the lack of ‘proper’ initialized embedding and getting 0.0 does not make sense to me.\n\nConclusions and Future Work:\n\n1. two lines before the last: please use “\\citep”.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeeZWtaFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the task of predicting visual predicates (e.g., eat, bite, take) between pairs of entities. In particular, the paper focuses on the zero-shot setting where the test predicates are unseen during training. The model uses linguistic prior from a knowledge graph (WordNet):\nwith graph embedding (fast GCN), unseen predicates are embedded based on the information propagated from seen predicates. The model is trained so that the visual feature vector and the correct predicate embedding are nearby in the joint embedding space. The method was evaluated on a zero-shot split of the Visual Genome dataset.\n\nOverall, as a task and dataset paper, the paper should have sold the task more by highlighting its special properties. While the task of predicting unseen predicates is interesting, the setting and the technique are similar to previous work on predicting other types of unseen labels (e.g., unseen objects, as referenced in the paper). The new task could still be interesting if it presents different challenges (e.g., maybe predicates are more ambiguous than objects, or visual predicates are harder to embed). But from the description in the paper, most of the challenges seem to also exist in other zero-shot settings (long-tail distribution; large space of labels). The paper could benefit from providing examples or statistics that demonstrate the challenges of the task.\n\nThe proposed method looks correct but is a rather direct application of existing methods. The experiment setup looks OK. Based on the error analysis, the labels look very noisy and subjective, but this seems to be a common problem in the visual predicate prediction task (hence the recall-based evaluation).\n\nAdditional questions:\n\n- The provided examples in the error analysis look pretty tricky; e.g., \"swing\" and \"slug\" are judged as different. How well would a human do on this task?\n\n- How much would Hit@k be if the test label is seen during training (not zero-shot)?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eKItmTFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper creates a new task for zero-shot learning of predicates (specifically in cases where the individual predicate components have never been seen in the training, rather than the more traditional setting where the full s-v-o relationship is unseen but each component is).  They create a new subset of the Visual Genome dataset specifically targeted towards predicting unseen predicates.  They also provide results using a knowledge graph (in this case WordNet), to integrate linguistic and visual features for prediction.  Interestingly, their pipeline introduces a new softmax variant, the unbalanced sampled softmax, which addresses the problem of over-predicting common predicates.\n\nI generally tend towards accepting this paper.  The reasons being that this paper has a few strong contributions: (1) they design a new task set-up with data they selected and cleaned from VG, (2) new modelling pipeline with empirical analysis backing the modeling choices, and (3) new softmax variant.  \n\nA few comments about things that could be strengthened or addressed further: \n- There could be more meaningful comparison to other zero-shot learning algorithms.  Even if they are not fully comparable because they were originally meant for a slightly different zsl set-up, it would be nice to have more baselines from external work.\n- Why was the unbalanced sampled softmax was being used for only predicate prediction and not entity prediction? \n- It wasn’t totally clear to me whether all of the verbs/entities were in WordNet and/or Glove.  If not, can the authors clarify what the overlap was and how this might be affecting performance?\n- As noted by the authors, there are some cases in Figure 3 where humans would consider the answer to be confusing or might actually prefer the machine response.  Can human performance on this task could be measured?  Perhaps humans could possibly evaluate a subset of the machine vs. gold answers?\n\nMinor Edits:\n- Related work: WordNet is misspelled in the last line\n- Figure 3: please consider using colors other than red/green, this is not readable for color-blind readers\n- Section 5.2: “The case c is confusing that even” →  “The case c is so confusing that even”\n- Section 5.2: “and output a more appropriate” --> “and outputs a more appropriate”\n- In the references: the Devise paper by [Frome et al 2013] is listed twice\n- In the references: the first author of ConceptNet5 should be Robyn Speer", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new problem setting of predicate zero-shot learning for visual relation recognition for the setting when some of the predicates are missing, and a model that is able to address it.

All reviewers agreed that the problem setting is interesting and important, but had reservations about the proposed model. In particular, the reviewers were concerned that it is too simple of a step from existing methods. One reviewer also pointed towards potential comparisons with other zero-shot methods.

Following that discussion, I recommend rejection at this time but highly encourage the authors to take the feedback into account and resubmit to another venue.
Decision: Reject