Learning to Recognize the Unseen Visual Predicates

1 INTRODUCTION . Visual relationship recognition ( Johnson et al. , 2015 ; Lu et al. , 2016 ; Xu et al. , 2017 ) aims to estimate the relationships between pairs of localized entities , i.e. , performing the recognition of triplets < subject , predicate , object > . It structurally describes images , which provides rich semantic information of an image to many applications including visual question answering ( VQA ) ( Li et al. , 2018 ) , image captioning ( Yang et al. , 2019 ) and image retrieval ( Johnson et al. , 2015 ) . The relationship recognition methods are mainly supervised to recognize the entities and then combine various entities in pairs to identify predicates between them . There is an increasing interest in relationship zero-shot learning ( ZSL ) that learns to recognize the unseen relationship triplets , where the studies ( Lu et al. , 2016 ; Yu et al. , 2017 ) on this ZSL problem setting assume the components ( subject , predicate , and object ) of the relationship triplet are seen . However , almost all of them only focus on dozens of frequent predicates and do not study the generalization of the seen predicates to the unseen ones . In this work , we propose the predicate zero-shot learning ( PZSL ) problem setting focusing on recognizing the unseen predicates ( no manual annotations or image samples ) . For example , given no instance of chew in the training data , the model is expected to recognize it during testing . Recognizing diverse even unseen predicates is significant for providing very rich relationship information , describing the complex scenes , and analogizing the known abstract concepts to the novel ones . The solution of the PZSL problem will greatly promote many downstream tasks , such as generating image caption with vivid predicates which are even unseen in the description corpus ( image captioning ) and answering the open questions ( with novel predicates ) on the complex scene ( VQA ) . Although zero-shot learning in image classification has received increasing attention ( Larochelle et al. , 2008 ) , PZSL is not explored . Furthermore , the PZSL problem is more challenging in the following aspects . a ) Recognizing predicates is difficult since predicates are often abstract not as specific as objects . Analogizing the seen abstract predicates to the unseen ones further escalates the difficulty . Many object ZSL methods ( Lampert et al. , 2014 ; 2009 ) adopt the pre-defined attributes of objects to recognize the unseen object . However , it is difficult to define the attributes of predicates . b ) Predicates of existing datasets follow an extreme long-tail distribution ( 92.26 % predicates with the number of instances lower than 10 in Visual Genome ( Krishna et al. , 2017 ) ) . The statistics of the dataset ( VG-Zero ) in this work are shown in Fig . 4 . Under this distribution , the model tends to collapse to output few frequent predicates . Note that if the infrequent predicates are not recognized , the unseen predicates are more unlikely to be recognized . To address the PZSL problem , we introduce a basic model to perform compatibility learning ( Frome et al. , 2013 ; Akata et al. , 2016 ; 2015 ) ( Fig . 1 ) , leveraging the linguistic priors from the corpus and knowledge base ( Wang et al. , 2018 ; Kampffmeyer et al. , 2018 ) . To represent the abstract predicates , we adopt the pre-trained word ( sentence ) vectors to initialize the predicates , connect them with linguistic relations defined in knowledge bases , and map them into a semantic embedding space ( middle of Fig.1 ) . A visual module is then applied to map paired image regions ( left of Fig . 1 ) into a visual feature space . The visual feature and semantic embedding spaces fall in the common space ( top of Fig . 1 ) . During training , the visual feature and semantic embedding space are aligned with the seen predicates as anchors , i.e. , a visual feature and semantic embedding labeled with the same predicate fall onto the same point/area in the common space . During testing , the samples in the test set are mapped into the visual feature space and matched with the nearest neighbor semantic embeddings of predicates ( like chew ) . Furthermore , to tackle the long-tail distribution , an unbalanced sampled-softmax is developed to adjust the gradient penalty of the infrequent predicates . The main contributions include : a ) We define the predicate zero-shot learning ( PZSL ) problem and introduce the corresponding dataset ( based on Visual Genome ) . b ) We propose a basic model to address the PZSL problem by compatibility learning leveraging the linguistic priors from the corpus and knowledge base . c ) We develop an unbalanced sampled-softmax for handling the extreme long-tail distribution of predicates . d ) The dataset and source code of this work will be released . 2 RELATED WORK . Visual relationships have been studied from various aspects including statistical motifs ( Zellers et al. , 2018 ) , entity-relationship dependencies ( Xu et al. , 2017 ) , spatial priors ( Dai et al. , 2017 ) , language statistics ( Li et al. , 2017 ) . Almost all of them focus on recognizing dozens of the most frequent predicates . By contrast , our work explores to train a model with about 1000 predicates and test it with about 100 unseen predicates . The two most relevant problem settings are relationship zero-shot learning setting ( Lu et al. , 2016 ) and open vocabulary setting ( Zhang et al. , 2018 ) . Lu et al . ( 2016 ) try to recognize the unseen relationships ( e.g. , < elephant , stand on , street > ) by transferring knowledge from similar relationships ( e.g. , < dog , stand on , street > ) . Note that all the test predicates and entities are seen in the training set . By contrast , the main difficulty in our problem setting is that all test predicates are unseen in the training set . Zhang et al . ( 2018 ) perform visual relationship recognition with an open vocabulary setting focusing on large-scale recognition problem without study on ZSL . The most relevant approach in visual relationship recognition is Graph R-CNN ( Yang et al. , 2018 ) which adopts a graph convolutional network ( GCN ) ( Kipf & Welling , 2016 ) to capture contextual information between visual objects and relations . By contrast , we utilize GCNs to leverage the relations of categories to generate meaningful semantic embeddings . Zero-shot learning ( ZSL ) . To recognize unseen objects , compatibility learning frameworks ( Frome et al. , 2013 ; Fu et al. , 2015 ; Fu & Sigal , 2016 ) map visual and semantic features into the common space and align the visual and semantic manifolds with the seen categories . During testing , these methods recognize the given visual feature by performing a nearest neighbor search on the semantic embeddings of the categories . Recent works ( Wang et al. , 2018 ; Kampffmeyer et al. , 2018 ) utilize the linguistic relations between seen and unseen categories in a knowledge graph ( KG ) for ZSL . Wang et al . ( 2018 ) propose to train a GCN supervised by the classifier ’ s weights of neural network . Our method can be considered as a hybrid of compatibility learning frameworks and knowledge graph based methods . The node embedding of KG is mapped into the same space with the visual feature . External knowledge bases ( KB ) , such as Wikipedia and ConceptNet ( Speer & Havasi , 2013 ) , has been introduced in visual relationship recognition to provide linguistic and commonsense priors . Yu et al . ( 2017 ) extract the < subject , predicate , object > triplets from Wikipedia and leverage the statistics P ( pred|sub , obj ) to help recognizing the unseen relationship triplets . Baier et al . ( 2017 ) train a semantic model from a KB to improve the performance . Gu et al . ( 2019 ) takes the detected objects to retrieve on the ConceptNet to obtain a set of triplets to enhance the visual features . Unlike the prior works essentially using the statistics of < subject , predicate , object > from the external KB , our work leverages the linguistic relations of predicates defined in WordNet ( Miller , 1992 ) to explicitly connect the predicates , such as < attack , is a hyponym of , fight > , for recognizing the unseen predicates . 3 PROBLEM SETUP . Setting : Let the full predicate vocabulary as Vpred = Vtrpred ∪ Vtepred and entity vocabulary as Ven , where Vtrpred and Vtepred are the training and test predicate vocabulary respectively , and “ entity ” refers to “ subject ” and “ object ” . The training and test predicates are disjoint , i.e. , Vtrpred ∩ Vtepred = φ . The dataset is denoted as D = { ( Ii , 〈bsij , sij ; pij ; boij , oij〉 ) } , where sij , oij ∈ Ven denote subject and object labels of the j-th relationship in i-th image Ii ( the green box in Fig . 2 ( A ) ) , bsij , b o ij are the corresponding boxes of the subject and object , and pij ∈ Vtrpred∪Vtepred is the corresponding predicate label . Any image that contains a test predicate p ∈ Vtepred is assigned to the test set Dte , and only the regions with test predicate p ∈ Vtepred are used for evaluation . The rest data is split into the training set Dtr and validation set Dval . During testing , given an image and pair of subject and object boxes from the test set Dte , the model ( trained on Dtr ) recognizes a triplet < subject , predicate , object > , where the accuracy of predicates is in concern . Assumption : We assume that , for any test predicate pte ∈ Vtepred , there exists training predicate ptr ∈ Vtrpred having semantic association with pte . For example , if chew is a test predicate , the predicates meaning an action using teeth ( like bite ) or intaking food ( like eat ) is expected to be included in the training set . Let the visual features and semantic embedding of bite are aligned , so does that of eat . As a result , the visual feature of chew , visual similar to that of bite and eat , is able to match to the predicates likes bite and eat in the semantic embedding space . To satisfy this assumption , the training predicates should be in large-scale to cover as much semantics as possible . 4 APPROACH . In this section , we first present the pipeline of our basic model , then the Simplified PinSage ( a fast graph convolutional network ) for propagating on the large-scale knowledge graph , and finally unbalanced sampled-softmax to handle the extreme long-tail distribution . 4.1 PIPELINE . The pipeline of our method consists of visual and knowledge modules , refer to Fig.2 . They are modeled by a Faster R-CNN ( Ren et al. , 2017 ) and two GCNs ( Kipf & Welling , 2016 ) respectively . Visual module aims to extract visual features of entity and predicate . Given an image as input , the corresponding features of the subject , object and context region are cropped out , where context region refers to the union of subject and object regions . All these feature regions are ROI aligned ( He et al. , 2017 ) as ROI features with fixed size ( 7×7 ) . The ROI features of subjects ( blue ) and objects ( yellow ) are then mapped into visual entity features Vs and Vo by the same multilayer perceptron MLPen . Furthermore , the ROI feature of context region ( green box ) is fed to MLPp and the output of which is fused with Vs and Vo to generate visual predicate feature Vp . All these features Vs , Vo and Vp will be aligned with the corresponding semantic embeddings Es , Eo and Ep . Knowledge module aims to generate the meaningful semantic embeddings of the categories ( including predicates and entities ) . To introduce the language prior and implicit association of the categories , the embeddings of the categories are initialized with word ( sentence ) vectors pre-trained on a large-scale corpus ( like GloVe ( Pennington et al. , 2014 ) ) . The categories are then connected by the linguistic relations ( defined by a KB , WordNet ) to build a knowledge graph . Note that many categories are not directly connected , but can be indirectly connected through categories within the KB outside the dataset . Thus the knowledge graph contains a huge number of nodes and provides rich linguistic information . Taking the knowledge graph as input , the semantic embeddings of predicate Ep and entity Een are generated by graph convolutional networks GCNp , GCNen respectively . For this part , the Simplified PinSage is introduced for fast processing on the large-scale graph , refer to § 4.2 . Loss function is defined as a summation of the entity and predicate terms as follows : L = Lentity + Lpredicate , ( 1 ) where Lpredicate is designed with the proposed unbalanced sampled-softmax for tackling the long-tail distribution of predicates , refer to § 4.3 , while Lentity is a negative log likelihood with softmax : Lentity = E ( Vc , c ) [ −log exp ( Vc · Ec ) ∑ k∈Ven exp ( Vc · Ek ) ] , ( 2 ) where c ∈ Ven is the label of the visual entity feature Vc .

Summary: This paper creates a new task for zero-shot learning of predicates (specifically in cases where the individual predicate components have never been seen in the training, rather than the more traditional setting where the full s-v-o relationship is unseen but each component is).  They create a new subset of the Visual Genome dataset specifically targeted towards predicting unseen predicates.  They also provide results using a knowledge graph (in this case WordNet), to integrate linguistic and visual features for prediction.  Interestingly, their pipeline introduces a new softmax variant, the unbalanced sampled softmax, which addresses the problem of over-predicting common predicates.