Title: LEARNING TO RECOGNIZE THE UNSEEN VISUAL PREDICATES

Abstract: Visual relationship recognition models are limited in the ability to generalize from finite seen predicates to unseen ones. We propose a new problem setting named predicate zero-shot learning (PZSL): learning to recognize the predicates without training data. It is unlike the previous zero-shot learning problem on visual relationship recognition which learns to recognize the unseen relationship triplets (<subject, predicate, object>) but requires all components (subject, predicate, and object) to be seen in the training set. For the PZSL problem, however, the models are expected to recognize the diverse even unseen predicates, which is meaningful for many downstream high-level tasks, like visual question answering, to handle complex scenes and open questions. The PZSL is a very challenging task since the predicates are very abstract and follow an extreme long-tail distribution. To address the PZSL problem, we present a model that performs compatibility learning leveraging the linguistic priors from the corpus and knowledge base. An unbalanced sampled-softmax is further developed to tackle the extreme long-tail distribution of predicates. Finally, the experiments are conducted to analyze the problem and verify the effectiveness of our methods. The dataset and source code will be released for further study.

1. INTRODUCTION:

Visual relationship recognition (Johnson et al., 2015; Lu et al., 2016; Xu et al., 2017) aims to estimate the relationships between pairs of localized entities, i.e., performing the recognition of triplets <subject, predicate, object>. It structurally describes images, which provides rich semantic information of an image to many applications including visual question answering (VQA) (Li et al., 2018) , image captioning (Yang et al., 2019) and image retrieval (Johnson et al., 2015) . The relationship recognition methods are mainly supervised to recognize the entities and then combine various entities in pairs to identify predicates between them. There is an increasing interest in relationship zero-shot learning (ZSL) that learns to recognize the unseen relationship triplets, where the studies (Lu et al., 2016; Yu et al., 2017) on this ZSL problem setting assume the components (subject, predicate, and object) of the relationship triplet are seen. However, almost all of them only focus on dozens of frequent predicates and do not study the generalization of the seen predicates to the unseen ones.

In this work, we propose the predicate zero-shot learning (PZSL) problem setting focusing on recognizing the unseen predicates (no manual annotations or image samples). For example, given no instance of chew in the training data, the model is expected to recognize it during testing. Recognizing diverse even unseen predicates is significant for providing very rich relationship information, describing the complex scenes, and analogizing the known abstract concepts to the novel ones. The solution of the PZSL problem will greatly promote many downstream tasks, such as generating image caption with vivid predicates which are even unseen in the description corpus (image captioning) and answering the open questions (with novel predicates) on the complex scene (VQA).

Although zero-shot learning in image classification has received increasing attention (Larochelle et al., 2008) , PZSL is not explored. Furthermore, the PZSL problem is more challenging in the following aspects. a) Recognizing predicates is difficult since predicates are often abstract not as specific as objects. Analogizing the seen abstract predicates to the unseen ones further escalates the difficulty. Many object ZSL methods (Lampert et al., 2014; 2009) adopt the pre-defined attributes of objects to recognize the unseen object. However, it is difficult to define the attributes of predicates. b) Predicates of existing datasets follow an extreme long-tail distribution (92.26% predicates with the number of instances lower than 10 in Visual Genome (Krishna et al., 2017) ). The statistics of the dataset (VG-Zero) in this work are shown in Fig. 4 . Under this distribution, the model tends to collapse to output few frequent predicates. Note that if the infrequent predicates are not recognized, the unseen predicates are more unlikely to be recognized.

To address the PZSL problem, we introduce a basic model to perform compatibility learning (Frome et al., 2013; Akata et al., 2016; 2015) (Fig. 1 ), leveraging the linguistic priors from the corpus and knowledge base (Wang et al., 2018; Kampffmeyer et al., 2018) . To represent the abstract predicates, we adopt the pre-trained word (sentence) vectors to initialize the predicates, connect them with linguistic relations defined in knowledge bases, and map them into a semantic embedding space (middle of Fig. 1 ). A visual module is then applied to map paired image regions (left of Fig. 1 ) into a visual feature space. The visual feature and semantic embedding spaces fall in the common space (top of Fig. 1 ). During training, the visual feature and semantic embedding space are aligned with the seen predicates as anchors, i.e., a visual feature and semantic embedding labeled with the same predicate fall onto the same point/area in the common space. During testing, the samples in the test set are mapped into the visual feature space and matched with the nearest neighbor semantic embeddings of predicates (like chew). Furthermore, to tackle the long-tail distribution, an unbalanced sampled-softmax is developed to adjust the gradient penalty of the infrequent predicates.

The main contributions include: a) We define the predicate zero-shot learning (PZSL) problem and introduce the corresponding dataset (based on Visual Genome). b) We propose a basic model to address the PZSL problem by compatibility learning leveraging the linguistic priors from the corpus and knowledge base. c) We develop an unbalanced sampled-softmax for handling the extreme long-tail distribution of predicates. d) The dataset and source code of this work will be released.

2. RELATED WORK:

Visual relationships have been studied from various aspects including statistical motifs (Zellers et al., 2018) , entity-relationship dependencies (Xu et al., 2017) , spatial priors (Dai et al., 2017) , language statistics (Li et al., 2017) . Almost all of them focus on recognizing dozens of the most frequent predicates. By contrast, our work explores to train a model with about 1000 predicates and test it with about 100 unseen predicates. The two most relevant problem settings are relationship zero-shot learning setting (Lu et al., 2016) and open vocabulary setting (Zhang et al., 2018) . Lu et al. (2016) Welling, 2016) to capture contextual information between visual objects and relations. By contrast, we utilize GCNs to leverage the relations of categories to generate meaningful semantic embeddings.

Zero-shot learning (ZSL). To recognize unseen objects, compatibility learning frameworks (Frome et al., 2013; Fu et al., 2015; Fu & Sigal, 2016) map visual and semantic features into the common space and align the visual and semantic manifolds with the seen categories. During testing, these methods recognize the given visual feature by performing a nearest neighbor search on the semantic embeddings of the categories. Recent works (Wang et al., 2018; Kampffmeyer et al., 2018) utilize the linguistic relations between seen and unseen categories in a knowledge graph (KG) for ZSL. Wang et al. (2018) propose to train a GCN supervised by the classifier's weights of neural network. Our method can be considered as a hybrid of compatibility learning frameworks and knowledge graph based methods. The node embedding of KG is mapped into the same space with the visual feature.

External knowledge bases (KB), such as Wikipedia and ConceptNet (Speer & Havasi, 2013) , has been introduced in visual relationship recognition to provide linguistic and commonsense priors. Yu et al. (2017) extract the <subject, predicate, object> triplets from Wikipedia and leverage the statistics P(pred|sub, obj) to help recognizing the unseen relationship triplets. Baier et al. ( 2017) train a semantic model from a KB to improve the performance. Gu et al. (2019) takes the detected objects to retrieve on the ConceptNet to obtain a set of triplets to enhance the visual features. Unlike the prior works essentially using the statistics of <subject, predicate, object> from the external KB, our work leverages the linguistic relations of predicates defined in WordNet (Miller, 1992) to explicitly connect the predicates, such as <attack, is a hyponym of, fight>, for recognizing the unseen predicates.

3. PROBLEM SETUP:

Setting: Let the full predicate vocabulary as V pred = V tr pred ∪ V te pred and entity vocabulary as V en , where V tr pred and V te pred are the training and test predicate vocabulary respectively, and "entity" refers to "subject" and "object". The training and test predicates are disjoint, i.e., V tr pred ∩ V te pred = φ. The dataset is denoted as

D = {(I i , b s ij , s ij ; p ij ; b o ij , o ij )}

, where s ij , o ij ∈ V en denote subject and object labels of the j-th relationship in i-th image I i (the green box in Fig. 2 

(A)), b s ij , b o ij

are the corresponding boxes of the subject and object, and p ij ∈ V tr pred ∪ V te pred is the corresponding predicate label. Any image that contains a test predicate p ∈ V te pred is assigned to the test set D te , and only the regions with test predicate p ∈ V te pred are used for evaluation. The rest data is split into the training set D tr and validation set D val . During testing, given an image and pair of subject and object boxes from the test set D te , the model (trained on D tr ) recognizes a triplet <subject, predicate, object>, where the accuracy of predicates is in concern.

Assumption: We assume that, for any test predicate p te ∈ V te pred , there exists training predicate p tr ∈ V tr pred having semantic association with p te . For example, if chew is a test predicate, the predicates meaning an action using teeth (like bite) or intaking food (like eat) is expected to be included in the training set. Let the visual features and semantic embedding of bite are aligned, so does that of eat. As a result, the visual feature of chew, visual similar to that of bite and eat, is able to match to the predicates likes bite and eat in the semantic embedding space. To satisfy this assumption, the training predicates should be in large-scale to cover as much semantics as possible.

4. APPROACH:

In this section, we first present the pipeline of our basic model, then the Simplified PinSage (a fast graph convolutional network) for propagating on the large-scale knowledge graph, and finally unbalanced sampled-softmax to handle the extreme long-tail distribution. Visual module aims to extract visual features of entity and predicate. Given an image as input, the corresponding features of the subject, object and context region are cropped out, where context region refers to the union of subject and object regions. All these feature regions are ROI aligned (He et al., 2017) as ROI features with fixed size (7 × 7). The ROI features of subjects (blue) and objects (yellow) are then mapped into visual entity features V s and V o by the same multilayer perceptron M LP en . Furthermore, the ROI feature of context region (green box) is fed to M LP p and the output of which is fused with V s and V o to generate visual predicate feature V p . All these features V s , V o and V p will be aligned with the corresponding semantic embeddings E s , E o and E p .

Knowledge module aims to generate the meaningful semantic embeddings of the categories (including predicates and entities). To introduce the language prior and implicit association of the categories, the embeddings of the categories are initialized with word (sentence) vectors pre-trained on a large-scale corpus (like GloVe (Pennington et al., 2014) ). The categories are then connected by the linguistic relations (defined by a KB, WordNet) to build a knowledge graph. Note that many categories are not directly connected, but can be indirectly connected through categories within the KB outside the dataset. Thus the knowledge graph contains a huge number of nodes and provides rich linguistic information. Taking the knowledge graph as input, the semantic embeddings of predicate E p and entity E en are generated by graph convolutional networks GCN p , GCN en respectively. For this part, the Simplified PinSage is introduced for fast processing on the large-scale graph, refer to § 4.2.

Loss function is defined as a summation of the entity and predicate terms as follows:

EQUATION

where L predicate is designed with the proposed unbalanced sampled-softmax for tackling the long-tail distribution of predicates, refer to § 4.3, while L entity is a negative log likelihood with softmax:

EQUATION

where c ∈ V en is the label of the visual entity feature V c .

4.2. SIMPLIFIED PINSAGE:

Simplified PinSage is adopted to map the knowledge graph into an embedding space for PZSL, see Fig. 2 (B ). Inspired by PinSage (Ying et al., 2018) , the propagation algorithm is divided into three steps: message passing, skip shortcut and normalization. We denote F C W,B • x = W x + B, and the process of graph propagation is formulated as follows:

EQUATION

EQUATION

EQUATION

where Eq. ( 3), ( 4) and ( 5) indicate message passing, skip shortcut and normalization respectively, N (u) in Eq. (3) denotes the neighbor set of u (u also falls into N (u)), [•, •] in Eq. ( 4) means "concatenate", h 0 u is the initial embedding, and h k u is output of k-th graph propagation layer. For n-layer GCN, F Wout,Bout takes h n u as input to get the final embedding E lying in the same space with the visual feature V . Note that the main computation in forward propagation is related to the number of edges (Eq. ( 3)). It is too computation and space consuming to perform propagation on the whole knowledge graph with about 2.2 billion edges. The graph sampling technology in web-scale recommender system (Ying et al., 2018; Eksombatchai et al., 2018) is introduced as an solution.

On-demand sampling. For generating embeddings of mini-batch categories, we only need to sample a necessary sub-graph as input to GCN (GCN p or GCN en ), avoiding propagating on the whole knowledge graph. Take 2-layer GCN as an example. Only first and second-order neighbors are needed to compute the final embedding of the target nodes. Refer to Fig. 2 (B), to get the embedding of u (deep blue), the second layer of GCN needs the embeddings (outputs of the first layer) of neighbors (blue) of u for message passing, i.e., N (u). To get the embeddings of N (u) in the first layer, the neighbors of the node set N (u) are needed (light blue), i.e., N 2 (u). Hence, we only sample a sub-graph containing the node set N (u) ∪ N 2 (u). In general, to get final embeddings of nodes batch U for n-layer GCN, only 1st to n-th order neighbors are needed, i.e., ∪ n i=1 N i (U). Neighbor-limit sampling. Many nodes of the knowledge graph contain a large number of neighbors, which makes computation and space consumption uncontrollable. To further reduce the consumption, we limit the number of neighbors per node to a threshold τ , i.e., randomly sampling τ neighbors. Refer to Fig. 2 (B), the unsampled neighbors (gray) do not contribute to the propagation. For testing, all neighbors are sampled to calculate the final embeddings, and we only need to propagate once to obtain the final semantic embeddings. The experiments in § 5 show that neighbor-limit sampling can be considered as a dropout-like operation for greatly avoiding overfitting. Thus, to obtain embedding of k predicates from n-layer GCN, the number of edges are not greater than a relaxed upper bound kτ n , where k, τ ≤ 100 and n ≤ 3. Thus we have kτ n ≤ 10 8 2.2 × 10 10 .

4.3. UNBALANCED SAMPLED-SOFTMAX:

A variant softmax function is proposed to measure the similarity between visual features and semantic embeddings, inspired by sampled softmax (Jean et al., 2014) in machine translation. For training of predicate recognition, if all elements of training vocabulary are sampled as negative embeddings, such as standard softmax, the visual features may always match the few most frequent predicate embeddings since the long-tail distribution of the predicates in the dataset. The impact of this distribution on zero-shot learning is devastating. To tackle this problem, we propose an unbalanced sampled-softmax (USS):

EQUATION

where V p is the visual feature whose predicate category is p, E i is the corresponding semantic embedding of predicate i. Unlike sampled softmax that adopts the pre-divided sub-vocabulary as negative categories, the negative predicates B n ⊂ V tr is sampled from the predicate distribution P pred . It is possible that B n includes the GT predicate p. Finally, the loss function of predicate recognition is in the form of

EQUATION

The sampling method is vital for Eq. ( 7). The uniform sampling, degrading into an estimated version of softmax, does not help with long-tail distribution. We design a sampling method to ensure the recognition of predicates with fewer samples so that the model can be further generalized to recognize unseen predicate categories. The idea is that the fewer categories appear as positive categories, the less they are sampled as negative categories. We adopt the frequency of predicates as the probability P pred to sample the negative predicates, see Fig. 2 (C ). This sampling method handles the long-tail distribution problem by adjusting the gradient of the infrequent predicates.

Let h i = V • E i , the gradient of S p w.r.t h i is discussed as follows:

EQUATION

)

Frequent predicates often fall into the first case in Eq. ( 8), which is the same as standard softmax. To the opposite, infrequent predicates always fall into the second case that the reward is increased when it is GT (p = i) and that there is no punishment when it is a negative predicate (p = i).

5. EXPERIMENTS:

In this section, we start by discussing the datasets, knowledge graph, and implementation details. We then perform the ablation studies to verify the components of our model and visualize our results.

VG-zero dataset. We introduce a new dataset based on the latest released Visual Genome dataset (VG v1.4) (Krishna et al., 2017) which contains 108,077 images with 21 relationships (triplets) on average per image. We manually cleaned up the box annotations in the same way with Xu et al. (2017) . Since the original annotation is noisy, 1155 synsets in WordNet are used to replace the original predicate categories as regularization, where the correspondence between the original categories and synsets is provided in the VG dataset. About 10% of predicates (105 predicates) are selected as test vocabulary. The frequency of the selected predicates falls in a range from 10 to 300, where the lower bound 10 is set to guarantee the quality of test set for the infrequent labels are noisy, and the upper bound 300 follows the rule that categories in test set should be least populated or rare (Xian et al., 2017) in zero-shot learning. Images annotated with predicates in test vocabulary are selected as the test set (containing 4350 images). We then randomly select 5000 images as the validation set with the rest as the training set. Similar to predicates, the entity categories are also replaced by 7k+ synsets. In addition, we use hypernym relationships to cluster entity categories into 96 categories since the entity synsets are still so specific that include numerous names and object recognition is not our focus. For example, categories like woman, father are clustered into the person category.

Knowledge graph. The knowledge graph G(V, E) is built from WordNet (Miller, 1992) , where V and E are nodes set and edges set respectively. Synsets (synonym set) in WordNet are nodes in G. Edge u, v is added into E if u and v have one of the following relationships: hypernym, hyponym, part meronym, part holonym, substance meronym, substance holonym, entailment, substance holonym and sharing lemmas. Notice that the self-loop will be included by the "sharing lemmas" relationship. Finally, the resulted knowledge graph contains 101,260 nodes and about 2.2 billion edges.

Implementation details. For all experiments, the model is trained for 150k iterations with batch size set to 4. We set the learning rate as 2e -3 and is reduced by 0.1 times at the 100k and 130k, respectively. We adopt the warmup strategy (Goyal et al., 2017) at the beginning. ResNet50 (He et al., 2016) is used as a backbone network with weights pre-trained on COCO (Lin et al., 2014) , which is fixed during training. Images are resized such that their short edge is 800 pixels. For the knowledge graph, we use the definitions of synsets as the input of off-the-shelf language models to generate the initial embeddings. More specifically, we use the word (GloVe) and sentence (InferSent (Conneau et al., 2017) ) embedding methods to get the initial embeddings. For the word embedding method, we take every word of the definition as a token to GloVe and average all the word embeddings to get the 300-D embedding. For the sentence embedding method, the whole definition is used as the input of InferSent to get the 4096-D embedding. The common space and hidden features in GCN are 512-D. 18.0 26.1 Random guess 4.7e -7 9.4e -7 1.9e -6 1.0e -6 2.1e -6 5.2e -6 1.0e -5 2.1e -5

5.1. ABLATION STUDY:

We evaluate methods with Hit@k on generalized and traditional setting (Tab. 1 and 2 ): Given any test region pair, we check whether the GT label falls within predicted categories with top k score. If yes, the sample is counted as a Hit. The Hit@k is #Hit #T est region pair . The methods are only required to Hit the unseen predicates ignoring the subjects and objects in Tab 1, while they are required to Hit the full relationship triplets in Tab 2. For the generalized setting, the methods search on the seen and unseen predicates vocabulary V tr ∪ V te for inferrence. For the traditional setting, the methods only search on the unseen predicates vocabulary V te . Note that the generalized setting is more challenging since the search space of the generalized setting is significantly larger than that of the traditional one. Observing Tab. 1 and 2, we can draw the following conclusions.

Knowledge graph prior is critical to our algorithm. As shown in the first row of the Tab. 1, a simple 2-layer MLP for semantic embedding, which neglects the relationship of predicates, cannot identify the unseen predicates in the generalized setting (0 accuracies) and can only obtain a lower recognition rate on the traditional setting. An 1-layer GCN that simply considers node relationships can already deliver significant performance gains (row 2 of Tab. 1), with a large margin (10.6 (%)) on Hit@20 traditional setting. The results show that by modeling the explicit connection between predicates, the knowledge can be effectively transferred from seen categories to unseen ones, which results in a performance boost on the unseen predicate. The same conclusion can be obtained from Tab. 2.

The unbalanced sampled-softmax effectively tackles the long-tail distribution problem. Refer to row 5 in Tab. 1, adopting the softmax loss function results in the worst performance since the outputs collapse into few frequent predicates and could not be generalized to novel predicates. By contrast, adopting the frequency of predicates as sampling probability has obvious advantages with a 15.7 (%) increase on Hit@20 traditional setting (comparing row 3 and 5 in Tab. 1). The same conclusion can be made by comparing row 2 and 3 in Tab. 2. Embedding initialization is necessary, but the impact of different embedding methods on performance is minor. Refer to row 14 in Tab.1, we adopt noise following normal distribution to initialize node embeddings, whose results are almost the same as random guessing (row 16). The initial embedding method clusters the semantically similar categories, which implicitly connects seen and unseen categories. These implicit connections are helpful for zero-shot learning. While using different embedding methods results in little difference: the InferSent method enjoys advantages over the GloVe method on the traditional setting with Hit@5∼20 but with no advantage in other evaluations.

The numbers of neighbors and layers result in negligible performance. For traditional setting, observing row 10∼13 in Tab. 1, the impact of τ is limited. Similarly, GCNs with different layers achieve almost equal accuracies, refer to row 2∼4. For the generalized setting, row 4 (3-layer GCN) and 13 (with 100 neighbors) achieve relatively low accuracy, reducing 2.9 (%) on Hit@20 generalized setting. It can be interpreted that a large number of training parameters and complete neighborhood information in the knowledge module make the model overfit the training vocabulary.

5.2. QUALITATIVE RESULTS:

Fig. 3 shows the results of our method, where output is in the form of <subject, predicate, object>. The recognition of predicates includes generalized and traditional settings shown at row 1 and 2 (blue/yellow font represent correct/wrong), while the ground truth triplets are displayed in the last row. The case a is completely correct in both settings. In case b, our method makes a mistake in the generalized setting while the result of the recognition (swing.v.01) is close to the ground truth (slug.v.01). This case shows that predicate zero-shot learning in the generalized setting is hard for semantically similar categories across training and test vocabularies. The case c is so confusing that even humans can make misjudgment. In case d, our method determines the predicate as chew.v.01 and outputs a more appropriate answer than the ground truth on the recognition of the object. In conclusion, the predicate zero-shot learning is challenging, but our method is effective.

6. CONCLUSIONS & FUTURE WORK:

In this work, we define a predicate zero-shot learning problem and propose a solution by mapping visual features and semantic embeddings (in knowledge graph) into the common space. Furthermore, graph sampling strategies are introduced for accelerating graph propagation, and an unbalanced sampled-softmax is proposed for tackling the long-tail distribution. Finally, we plan to explore the following future work for this problem. a) Consistency of differences of visual feature and semantic embedding space can be considered to model the cross-modal analogy. b) A semantic-aware negative sampling of predicate categories is a solution worth exploring. c) Learning the visual predicate as a translation vector (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015) from subject to object such as VtransE (Zhang et al., 2017) deserve attempted.

:



Figures and Tables:

Figure1: A basic model for recognizing unseen visual predicates. The visual data and knowledge graph's nodes are mapped into a common space by a visual and knowledge module respectively, where the sub-spaces from visual and knowledge module are named with visual feature and semantic embedding space correspondingly. Note that the visual predicate feature contains features of the subject, object and the union of them. The basic model contains two stages. First, the visual feature and semantic embedding space are aligned by taking the seen predicates in the training set as anchors, the so-called compatibility learning. Second, the samples in the test set are mapped into visual feature space and matched with the nearest predicate neighbor from the semantic embedding space.

Figure2:A) The pipeline of our basic model. The visual features (from the visual module) and corresponding semantic embeddings (from the knowledge module) are constrained to be close. B) Simplified PinSage. We sample the graph nodes in two aspects during training. Take 2-layer GCN as an example, for on-demand sampling, to get the final embedding of the target node u, the neighbors of u are needed. Backtracking in this way, we only need first and second-order neighbors of u. For the neighbor-limit sampling, the number of each node's neighbors is limited by a constant. C) Unbalanced sampled-softmax. To tackle the long-tail distribution problem, a negative predicate batch B n is sampled from the predicate distribution to calculate the loss function.

Figure 3: The images with paired located entities are on the top, where the blue and yellow boxes represent subject and object respectively. The results of the generalized and traditional setting are at the first two rows (correct: blue, wrong: yellow), while the ground truth triplets are at the last row.

try to recognize the unseen relationships (e.g., <elephant, stand on, street>) by transferring knowledge from similar relationships (e.g., <dog, stand on, street>). Note that all the test predicates and entities are seen in the training set. By contrast, the main difficulty in our problem setting is that all test predicates are unseen in the training set.

Table content: <table><tr><td>Zhang et al. (2018) perform</td></tr><tr><td>visual relationship recognition with an open vocabulary setting focusing on large-scale recognition</td></tr><tr><td>problem without study on ZSL. The most relevant approach in visual relationship recognition is</td></tr><tr><td>Graph R-CNN (Yang et al., 2018) which adopts a graph convolutional network (GCN) (Kipf &amp;</td></tr></table>

Accuracy of unseen predicate recognition.

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="2">Hits@k (%)</td></tr><tr><td colspan="3">NO. Propagation Initial embedding</td><td>Loss</td><td colspan="2">Generalized setting</td><td>Traditional setting</td></tr><tr><td/><td/><td/><td/><td>5 10</td><td>20</td><td>1</td><td>2</td><td>5</td><td>10 20</td></tr><tr><td>1</td><td>W/O KG</td><td>GloVe</td><td colspan="2">|B n | = 10 0.0 0.0</td><td>0.0</td><td>2.5 5.2 13.0 23.1 37.9</td></tr><tr><td>2</td><td>1-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 10 1.9 4.7</td><td>10.1</td><td>5.8 10.7 20.2 32.3 48.5</td></tr><tr><td>3</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 10 4.3 7.0</td><td>11.3</td><td>7.5 12.2 22.2 33.2 48.7</td></tr><tr><td>4</td><td>3-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 10 1.9 3.9</td><td>8.4</td><td>5.3 9.3 19.9 33.3 49.1</td></tr><tr><td>5</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">Softmax 0.0 0.0</td><td>0.0</td><td>2.2 4.2 10.8 19.0 33.0</td></tr><tr><td>6</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 5 3.2 6.5</td><td>10.6</td><td>6.5 10.9 20.1 32.5 48.1</td></tr><tr><td>7</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 20 4.1 7.4</td><td>11.8</td><td>8.9 13.0 21.3 32.0 49.5</td></tr><tr><td>8</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 50 4.7</td><td>9.9</td><td>5.5 10.4 21.7 32.9 48.6</td></tr><tr><td>9</td><td>2-layer GCN</td><td>GloVe</td><td colspan="2">|B n | = 100 1.3 3.8</td><td>8.5</td><td>4.6 8.8 18.3 29.5 46.2</td></tr><tr><td>10</td><td>τ = 5</td><td>GloVe</td><td colspan="2">|B n | = 10 1.9 4.7</td><td>10.1</td><td>4.2 8.9 20.2 31.7 48.6</td></tr><tr><td>11</td><td>τ = 20</td><td>GloVe</td><td colspan="2">|B n | = 10 1.2 4.1</td><td>8.7</td><td>3.9 8.3 19.8 32.6 50.5</td></tr><tr><td>12</td><td>τ = 50</td><td>GloVe</td><td colspan="2">|B n | = 10 2.5 5.3</td><td>9.2</td><td>4.5 9.1 18.1 30.8 48.8</td></tr><tr><td>13</td><td>τ = 100</td><td>GloVe</td><td colspan="2">|B n | = 10 1.9 4.0</td><td>7.8</td><td>4.4 8.1 17.5 31.4 50.3</td></tr><tr><td>14</td><td>2-layer GCN</td><td>Normal</td><td colspan="2">|B n | = 10 0.0 0.0</td><td>0.0</td><td>1.4 2.4 5.0 10.2 19.6</td></tr><tr><td>15</td><td>2-layer GCN</td><td>InferSent</td><td colspan="2">|B n | = 10 4.1 7.0</td><td>11.4</td><td>7.3 12.4 23.1 35.8 53.5</td></tr><tr><td>16</td><td/><td>Random guess</td><td/><td>0.1 0.8</td><td>1.7</td><td>0.9 1.9 4.7 9.5 19.0</td></tr></table>

Accuracy of recognition of triplets with unseen predicates.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">Hits@k (%)</td><td/><td/><td/></tr><tr><td>Methods</td><td colspan="3">Generalized setting</td><td/><td colspan="3">Traditional setting</td><td/></tr><tr><td/><td>5</td><td>10</td><td>20</td><td>1</td><td>2</td><td>5</td><td>10</td><td>20</td></tr><tr><td>W/O KG</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.1</td><td>2.3</td><td>7.8</td><td>12.5</td><td>20.1</td></tr><tr><td>Softmax</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.9</td><td>1.9</td><td>5.0</td><td>9.9</td><td>17.5</td></tr><tr><td>Ours</td><td>1.3</td><td>2.8</td><td>5.8</td><td>3.2</td><td>6.0</td><td>11.6</td><td/><td/></tr></table>

Peer review (iclr2020):
Title: Good work, requires some edits. 1. Summarize: This paper proposes a new problem setting in visual relation detection which is called “Predicate Zero-shot Learning (PZSL)”. They provide a clear motivation and description of this setting. They propose a solution to this problem which leverages linguistic priors and knowledge bases. Furthermore they propose an unbalanced sampled-softmax to tackle the long tail distribution of predicates. 2. Clearly state your decision. One or two key reasons for this choice. I will go for a weak accept for the paper at this stage. (+) I think the proposed problem setting is well-motivated and useful. Also, (+) the proposed initial solution to this problem is interesting. However, (-) they propose a “fast graph convolution network” which seems to be precisely equivalent to a PinSage. Also, (-) the paper requires to be polished as it lacks clarity. 3. Main discussion My first argument is: I’m not sure why the authors have changed the name of PinSage and just mentioned that “their” “Fast Graph Convolution Network” is “inspired” from PinSage. To me it looks exactly the same. If there are any differences, it should be stated clearly. In fact, I would not be against using PinSage as a part of their approach. However, trying to rename it without clear reasons is not a good idea. My second argument is that the paper lacks clarity in writing (for detailed suggestions please refer to comments and feedbacks). Specially the evaluation section lacks details and clarity: a) In the beginning of this section (page 7), the authors talk about “generalized” and “traditional” settings without properly defining them. b) The descriptions for Table 1 and Table 2 fail to provide enough details to help understand the difference between the results in these two tables (one of them states “Accuracy of unseen predicate recognition” and the other one “Accuracy of recognition of triplets with unseen predicates”). 4. Comments and feedback. Introduction: Paragraph one in the: 1. The relationship recognition methods are mainly supervised “that” ? “to”. 2. last line: …. and do not study “on generalizing” ? “the generalization of”. Paragraph two: 1. no manual annotations or “real samples” ? “image samples”. (a real sample is ill-defined) 2. For example, no instance of chew ? For example “given” no instance of chew. Paragraph three: 1. … is difficult since predicates are often abstract not as specific ? is difficult since predicates are often abstract “and” not as specific. 2. Furthermore, unlike many object ZSL methods … ? This line to the end is very complicated and hard to understand. Related Works: 1. Visual Relationships: I would cite “Graph R-CNN for scene graph generation” since it is the most relevant work regarding the similarity of pipeline (using GCNs). 2. External Knowledge bases (KB): I would cite “Improving Visual Relationship Detection using Semantic Modeling of Scene Descriptions” since it is one of the most relevant works using knowledge graph modelings to improve visual relation detection. Problem Setup: Do you plan to provide the proposed dataset splits so others can work on this setting? I consider this very important given your paper’s contribution. Maybe it is better if it is also mentioned in the paper. Pipeline: 1. Paragraph 2: … the output of which is fused with …: Given Figure 2, it does not seem like is being created by fusing and . It looks more like it is extracted directly from the image (union of bounding boxes). 2. In Figure 2: In the representation of Pipeline (A), the graph is colored by dark blue for objects and light blue for predicates. The represented graphs show Object to Object and Predicate to Predicate connections which I’m not sure if it is correct. Shouldn’t we always have a light blue between every pair of dark blue connections? Evaluation: 1. Please consider the mentioned points in the Main Discussions. 2. In Table 1, I suggest re-naming “embedding” to “initial embedding”. 3. In Table 1, Hit@k should be Hits@k. 4. Please define the metrics clearly (Hits@k). Extra: I have a question regarding the ablation studies with GloVe, Normal and InferSent initialization. The question is whether this initialization is necessary? It seems like in the setting “W/O KG”, even though the embeddings are initialized with GloVe, there is no gain (all of the Hits@k values are 0.0). So GloVe embedding without KG bring no external semantic knowledge? Then why use them? Regarding that, I can see that a GCN, initialized with normally distributed embeddings (row 14 in Table 1) has given 0.0 accuracies, but I find this very counter intuitive, as graph convolution layers already have trainable weights capable of compensating for the lack of ‘proper’ initialized embedding and getting 0.0 does not make sense to me. Conclusions and Future Work: 1. two lines before the last: please use “citep”.