Independent Prototype Propagation for Zero-Shot Compositionality

1 Introduction . As humans , hearing the phrase ‘ a tiny pink penguin reading a book ’ can conjure up a vivid image , even though we have likely never seen such a creature before . This is because humans can compose their knowledge of a small number of visual primitives to recognize novel concepts [ 1 ] , a property which Lake et al . [ 2 ] argue is one of the key building blocks for human intelligence missing in current artificial intelligence systems . Machines , on the other hand , are largely data-driven , and usually require many labeled examples from various viewpoints and lighting conditions in order to recognize novel concepts . Since visual concepts follow a long-tailed distribution [ 3 , 4 ] , such an approach makes it near impossible to gather sufficient examples for all possible concepts . Compounded by that , in the absence of sufficient data , vanilla convolutional neural networks will use any correlation they can find to classify training samples , even when they are spurious [ 5 ] . In this work , we aim to tackle both of these issues . Compositional Zero-Shot Learning ( CZSL ) [ 6 ] is the problem of learning to model novel objects and their attributes as a composition of visual primitives . Previous works in CZSL [ 6 , 7 , 8 ] largely ignore the dependencies between classes with shared visual primitives , and the spurious correlations between 1https : //github.com/FrankRuis/protoprop 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . attributes and objects . More recently , Atzmon et al . [ 9 ] tackle the latter by ensuring conditional independence between attribute and object representations , while Naeem et al . [ 10 ] explicitly promote dependencies between the primitives and their compositions . While the independence approach improves generalization , it hurts accuracy on seen classes by removing useful correlations . The explicit dependencies , on the other hand , can share these useful correlations with unseen classes , but there will always be some that are spurious , hurting generalization . In this work , we propose to take advantage of the strengths of both approaches , respectively , by learning independent visual representations of objects and attributes , and by learning their compositions for the target classes . First , we represent visual primitives by learning local independent prototypical representations . Prototype networks [ 11 ] learn an embedding function , where inputs of the same class cluster around one prototypical representation of that class . Here we adopt such a function for learning prototypes of objects and attributes . Next , we leverage a compositional graph to learn the dependencies between the independent prototypes on the one hand , and the desired seen and unseen classes on the other , by propagating the prototypes to compositional nodes . Here , the compositional graph allows some information to be shared between objects that share attributes , e.g. , between tigers and zebras . Sylvain et al . [ 12 ] show the importance of locality and compositionality for model generalization in zero-shot learning . They also propose a measure for the compositionality of a representation , which is equivalent to our compositional loss function . The proposed method , ProtoProp , is outlined in Figure 1 . Our main contributions are : 1 ) We propose a novel graph propagation method that learns to combine local , independent , attribute and object prototypes into one compositional prototype that can accurately detect unseen compositional classes . 2 ) A spatial attention-based pooling method that allows us to obtain differentiable attribute and object patches for use in an independence loss function . 3 ) Our method effectively deals with bias from an underspecified dataset by learning independent representations that then take on the dependencies of the desired target distribution . 4 ) We validate through ablations the importance of each part of the method ( local prototypes vs semantic embeddings , independence loss , backbone finetuning ) and their contribution to the final results . 5 ) We show that we improve on state-of-the-art results on three challenging compositional zero-shot learning benchmarks : 2.5 to 20.2 % harmonic mean improvement on AO-Clevr [ 9 ] , 3.1 % harmonic mean improvement on UT-Zappos [ 13 ] , and a slight improvement on C-GQA compared to the best existing method . 2 Related work . Compositional zero-shot learning ( CZSL ) methods aim to recognize unseen compositions from known visual primitives in the form of attributes and objects . One line of work considers embedding the visual primitives in the image feature space . Misra et al . [ 6 ] use the weight vectors of linear SVMs as embeddings for the visual primitives , which they transform to recognize unseen compositions . Li et al . [ 8 ] look at attribute-object compositions through the lens of symmetry inspired by group theory . A different line of work considers a joint embedding function on the image , attribute , and object triplet , allowing the model to learn dependencies between the image and its visual primitives . Purushwalkam et al . [ 7 ] train a set of modular networks together with a gating network that can ‘ rewire ’ the classifier conditioned on the input attribute and object pair . Atzmon et al . [ 9 ] take a causal view of CZSL , trying to answer which intervention caused the image . They apply conditional independence constraints to the representations of the visual primitives , sacrificing accuracy on the seen data but performing well on unseen data by removing correlations that are useful but hurt generalization to novel compositions . Naeem et al . [ 10 ] , on the other hand , explicitly promote the dependency between all primitives and their compositions within a graph structure , though they rely on pretrained semantic embeddings which may differ in distribution from the visual concepts they describe . Our proposed method combines the strengths from Atzmon et al . [ 9 ] and Naeem et al . [ 10 ] by first learning independent representations of the visual primitives , which are then propagated through a compositional graph to learn the dependency structure of the desired target distribution . Unlike most other methods , we are not reliant on pretrained word embeddings , but instead learn the representations for our visual primitives directly from the training data . Instead of a linear kernel like Atzmon et al . [ 9 ] , we use a Gaussian kernel for our independence loss as we find that its ability to capture higherorder statistics is beneficial in terms of accuracy . Like Naeem et al . [ 10 ] , we train our model fully end-to-end , including the feature extractor , as learning a good embedding is often more beneficial than an overly complicated method applied to suboptimal embeddings [ 14 ] . Prototypical networks [ 11 ] aim to learn an embedding function where inputs of the same class cluster around one prototypical representation of that class . They measure L2 distance between samples in pixel space , which is sensitive to non-semantic similarities between images , such as objects from different classes with a similar background and light conditions . Li et al . [ 15 ] move the similarity metric to latent space and utilise an autoencoder to directly visualize the prototypes , but they only test on simple MNIST-like benchmarks . Chen et al . [ 16 ] extend the method to multiple local prototypes per class , where prototypes are compared to local image patches instead of the entire average-pooled output of a CNN , and evaluate on the more complicated fine-grained bird classification dataset CUB [ 17 ] . We take a similar local-prototype approach , but we use direct attribute and object supervision , enabling parameter sharing between classes and allowing the prototypes to be used for zero-shot classification . While for most methods these prototypes are used for the final classification , in our case they are an intermediate representation . Graph neural networks ( GNNs ) are models that can work directly on the structure of a graph . They have been first proposed by Gori et al . [ 18 ] , later elaborated upon by Scarselli et al . [ 19 ] and popularised by Kipf and Welling [ 20 ] in their work on the Graph Convolutional Neural Network ( GCN ) . GNNs grow more popular every year , and a lot of improvements have been proposed recently [ 21 ] . Just like the GCNs were inspired by CNNs , most improvements are inspired by other areas of deep learning such as attention mechanisms in Graph Attention Networks [ 22 ] , but this jump from existing deep learning fields to GNNs has overlooked simpler methods . Wu et al . [ 23 ] have taken a step back and stripped down GNNs to their simplest parts by removing nonlinearities and collapsing weight matrices until all that was left was feature propagation followed by a linear model . In the same vein , Huang et al . [ 24 ] show that a simple Multilayer Perceptron ( MLP ) ignoring graph structure followed by a label propagation post-processing step often greatly outperforms GNNs . These simplified methods are mostly limited to transductive settings ( test nodes are available at train time ) and graphs that exhibit strong homophily ( similar nodes are connected ) . Many real-world graphs fit those criteria , including the graph we use in this work . 3 Method . In compositional zero-shot learning ( CZSL ) , we have a set of images X and compositional classes with compositional labels Y ⊆ A × O , where A is a set of attribute labels and O a set of object labels . The labels are subdivided into Y = Ys ∪ Yu , where Ys are the seen labels for the training set and Yu unseen labels for the validation and test sets , with Ys ∩ Yu = ∅ . We denote the training data consisting of only seen compositional classes as Xs . Finally , CZSL assumes that each attribute and object is seen in at least one training data point , or more formally ∀p ∈ A∪O ∃y ∈ Ys s.t . p∩ y 6= ∅ . 3.1 Prototype-based representations . In this section we describe how we learn local attribute and object prototypes , before they are propagated and combined into compositional prototypes in Section 3.3 . Prototypes are an average representation of a target class , with strong generalization and interpretability properties . We employ a local prototype approach similar to Chen et al . [ 16 ] , though we use direct supervision , encouraging the feature extractor to learn local image representations that encode the attribute and object labels . We use a ResNet-18 [ 25 ] backbone for fair comparison to other CZSL methods , but the method is backbone agnostic . Figure 2 shows an example of a prototype layer . The layer has a set of prototype vectors P = { pj } kj=1 , pj ∈ RC where each target class is represented by one prototype . k = |Yi| is the number of attribute or object targets , and i ∈ { 0 , 1 } indicates if we are training on attribute or object labels respectively . The layer takes as input the H ×W × C output of a CNN just before its final pooling layer , and calculates a compatibility score s = 〈xij , pk〉 ( e.g. , cosine similarity , L2 norm , dot product ) between the input patches xij ∈ RC over the spatial dimensions H×W and the prototypes . We find that a dot product similarity metric leads to better generalization . To save on parameters and avoid overfitting , we do not use a fully connected layer to combine prototype similarity scores into a final prediction . Instead , the maximum patch-prototype similarity score is used directly as the compatibility score for the corresponding prototype ’ s class . This compatibility score is optimized through a cross-entropy loss function : LCE = 1 |S| ∑ s , y ∈S − log ( exp ( s [ yi ] ) ∑|s| j exp ( s [ j ] ) ) ( 1 ) Here , with fp as our prototype layer , S = { s , y ∈ ( fp ( Xs ) | Ys ) } is the set of prototype similarity scores and target labels for the training data . i ∈ { 0 , 1 } indicates if we are training on attribute or object labels respectively . Cross-entropy loss naturally encourages clustering between samples of the same class and separation from other classes , but we find that an additional loss to push these properties even more is beneficial . For that purpose we adopt the cluster and separation costs proposed by Chen et al . [ 16 ] : Clst = 1 |X| |X|∑ i=1 min j : pj∈Pyi min z∈xij ‖z − pj‖22 Sep = − 1 |X| |X|∑ i=1 min j : pj /∈Pyi min z∈xij ‖z − pj‖22 ( 2 ) The costs Clst and Sep encourage a higher degree of clustering between prototypes of the same class and a greater distance to prototypes of different classes , respectively . The separation cost is only applied to the object prototypes , as the attribute prototypes benefit from less distant representations . Our Hilbert-Schmidt Independence Criterion ( HSIC ) loss ( details in Section 3.2 ) requires the input patches with the highest prototype similarity scores as its input , but that would require the use of the argmax function which has a gradient of 0 almost everywhere . Similar to spatial attention , we use the softmax function over the similarity map as a differentiable proxy . The softmax outputs used as weights in a weighted sum of the input patches xij serve as an approximation of the input patch with the highest similarity score ( cf . Figure 2 ) , and ensure that each patch containing information about attribute or object labels is affected by the independence loss proportional to the strength of their appearance ( i.e . their similarity score ) .

Summary: The paper deals with the problem setup of compositional zero-shot recognition. Where a visual object is specified by two labels: an attribute name and an object name. Not all attribute-object combinations are available at training time, and at test time, a model may observe new (unseen) combinations.  The approach combines ideas from two recent publications: [1] Atzmon et al, 2020, and [2] Naeem et al, 2021: It uses the idea of encouraging independent prototypes for the attribute and object (Atzmon), and then it uses them as the node features for the graph-NN used by (Naeem), replacing than the word-embedding used in (Naeem). In addition, it proposes a spatial attention mechanism over CNN feature maps for extracting the prototypes, rather than the average-pooling used by previous baselines.