Independent Prototype Propagation for Zero-Shot Compositionality
Authors: Frank Ruis
Gertjan J. Burghouts
Doina Bucur
Venue: NeurIPS 2021 Poster (2021)
Keywords: compositional zero-shot learning
prototypes
graph neural networks
independence

Abstract: Humans are good at compositional zero-shot reasoning; someone who has never seen a zebra before could nevertheless recognize one when we tell them it looks like a horse with black and white stripes. Machine learning systems, on the other hand, usually leverage spurious correlations in the training data, and while such correlations can help recognize objects in context, they hurt generalization. To be able to deal with underspecified datasets while still leveraging contextual clues during classification, we propose ProtoProp, a novel prototype propagation graph method. First we learn prototypical representations of objects (e.g., zebra) that are independent w.r.t. their attribute labels (e.g., stripes) and vice versa. Next we propagate the independent prototypes through a compositional graph, to learn compositional prototypes of novel attribute-object combinations that reflect the dependencies of the target distribution. The method does not rely on any external data, such as class hierarchy graphs or pretrained word embeddings. We evaluate our approach on AO-Clevr, a synthetic and strongly visual dataset with clean labels, UT-Zappos, a noisy real-world dataset of fine-grained shoe types, and C-GQA, a large-scale object detection dataset modified for compositional zero-shot learning. We show that in the generalized compositional zero-shot setting we outperform state-of-the-art results, and through ablations we show the importance of each part of the method and their contribution to the final results. The code is available on github.

Independent Prototype Propagation for Zero-Shot Compositionality
Frank Ruis University of Twente & TNO research0frankkrutt ml
Gertjan J. Burghouts TNO
Doina Bucur University of Twente
gertjan an. www.boutsttnno..m.
bucur@utwente.
Abstract
Humans are good at compositional zero-shot reasoning; someone who has never seen zebra before could nevertheless recognize one when we tell them it looks like horse with black and white stripes. Machine learning systems, on the other hand, usually leverage spurious correlations in the training data, and while such correlations can help recognize objects in context, they hurt generalization. To be able to deal with underspecified datasets while still leveraging contextual clues during classification, we propose ProtoProp, a novel prototype propagation graph method. First we learn prototypical representations of objects (e.g., zebra) that are independent w.r.t. their attribute labels (c.g., stripes) and vice versa. Next we propagate the independent prototypes through compositional graph, to learn compositional prototypes of novel attribute-object combinations that reflect the dependencies of the target distribution. The method does not rely any external data, such as class hierarchy graphs or pretrained word embeddings. We evaluate our approach on AO-Cleve synthetic and strongly visual dataset with clean labels, UT-Zappos, a noisy real-world dataset of fine-ge fine-grained shoe types, and C-GQA, a large-scale object detection dataset modified for compositional zero-shot learning. We show that in the generalized compositional zero-shot setting we outperform state-of-the-ard results, and through ablations we show the importance of each part of the method and their contribution to the final results. The code is available on github'.
Introduction
As humans, hearing the phrase 'a tiny pink penguin reading a book' can conjure up a vivid image, even though we have likely never seen such a creature before. This is because humans can compose their knowledge of small number of visual primitives to recognize novel concepts [1], a property which Lake et al. [2] argue is one of the key building blocks for human intelligence missing current artificial intelligence systems. Machines, on the other hand, are largely data-driven, and usually require many labeled examples from various viewpoints and lighting conditions in order to recognize novel concepts. Since visual concepts follow a long-tail distribution [3, 4], such an approach makes it near impossible to gather sufficient examples for all possible concepts. Compounded by that, in the absence of sufficient data, vanilla convolutional neural networks will use any correlation they can find classify training samples, even when they spurious [5]. In this work, we aim tackle both of these issues.
Compositional Zero-Shor Sero-Shot Learning (CZSL) [6] is the problem of learning to model novel objects and their attributes as a composition of visual primitives. Previous works in CZSL [6, 7, 8] largely ignore the dependencies between classes with shared visual primitives, and the spurious correlations between
httpp:///ithhb... http:/chhw..cccoeeeeereessee/eee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Tiger
Hone
Lear
Prototyoical
Condioond Infogaddeic
Tran
ProtoProp
Propagate
Pradkt
Novel Compostion
Input Bay esample datia)
Prototyyicall Representations
Str
White
Orange
Figure I: ProtoProp A sketch of our proposed method; we learn independent prototypical repre- sentations of visual primitives in the form of objects (e.g., horse) and attributes (e.g.. stripes). The prototypes are then propagated through compositional graph, where they are combined into novel compositional prototypes to recognize both seen and unseen classes (e.g., zebra).
attributes and objects. More recently, Atzmon et al. [9] tackle the latter by ensuring conditional independence between attribute and object representations, while Naeem al. [10] explicitly promote dependencies between the primitives and their compositions. While the independence approach improves generalization, it hurts accuracy on seen classes by removing useful correlations. The explicit dependencies, on the other hand, can share these useful correlations with unseen classes, but there will always be some that are spurious, hurting generalization.
In this work, we propose to take advantage of the strengths of both approaches, respectively, by learning independent visual representations of objects and attributes, and by learning their compo- sitions for the target classes. First, we represent visual primitives by learning local independent prototypical representations. Prototype networks [11] learn an embedding function, where inputs of the same class cluster around one prototypical representation of that class. Here we adopt such a function for learning prototypes of objects and attributes. Next, we leverage compositional graph to learn the dependencies between the independent prototypes on the one hand, and the desired seen and unseen classes on the other, by propagating the prototypes to compositional nodes. Here, the compositional graph allows some information to be shared between objects that share attributes, e.g., between tigers and zebras. Sylvain al. [12] show the importance locality and compositionality for model generalization in zero-shot learning. They also propose a measure for the compositionality of a representation, which equivalent to our compositional loss function. The proposed method, ProtoProp, is outlined in Figure 1.
Our main contributions are: We propose novel graph propagation method that learns combine local, independent, attribute and object prototypes into one compositional prototype that can accurately detect unseen compositional classes. A spatial attention-basel pooling method that allows us to obtain differentiable attribute and object patches for use in an independence loss function. Our method effectively deals with bias from an underspecified dataset by learning independent representations that then take on the dependencies of the desired target distribution. 4) We validate through ablations the importance of each part of the method (local prototypes vs semantic embeddings, independence loss, backbone finetuning) and their contribution to the final results. 5) We show that we improve on state-of-the-art results on three challenging compositional zero-shot learning benchmarks: 2.5 to 20.2% harmonic mean improvement on AO-Clevr [9]. 3.1% harmonic mean improvement on UT-Zappos [13], and a slight improvement on C-GQA compared the best existing method.

---

2
Related work
Compositional zero-shot learning (CZSL) methods aim to recognize unseen compositions from known visual primitives in the form of attributes and objects. One line of work considers embedding the visual primitives in the image feature space. Misra al. [6] use the weight vectors of linear SVMs as embeddings for the visual primitives, which they transform to recognize unseen compositions. Li et al. [8] look at attribute-object compositions through the lens of symmetry inspired by group theory. A different line of work considers joint embedding function on the image. attribute, and object triplet, allowing the model to learn dependencies between the image and its visual primitives. Purushwalkam et al. [7] train a set of modular networks together with gating network that can 'rewire' the classifier conditioned on the input attribute and object pair. Atzmon et al. [9] take a causal view of CZSL, trying answer which intervention caused the image. They apply conditional independence constraints to the representations of the visual primitives, sacrificing accuracy on the seen data but performing well on unseen data by removing correlations that are useful but hurt generalization to novel compositions, Naeem et al. [10], on the other hand, explicitly promote the dependency between all primitives and their compositions within a graph structure. though they rely on pretrained semantic embeddings which may differ in distribution from the visual concepts they describe.
Our proposed method combines the strengths from Atzmon et al. [9] and Naeem et al. [10] by first learning independent representations of the visual primitives, which then propagated through compositional graph to learn the dependency structure of the desired target distribution. Unlike most other methods, we are not reliant on pretrained word embeddings, but instead learn the representations for our visual primitives directly from the training data. Instead of a linear kernel like Atzmon et al. [9], we use Gaussian kernel for our independence loss as we find that its ability to capture higher- order statistics is beneficial in terms of accuracy. Like Nacem et al. [10], we train our model fully end-to-end, including the feature extractor, as learning a good embedding often more beneficial than an overly complicated method applied to suboptimal embeddings [14].
Prototypical networks [11] aim to learn an embedding function where inputs of the same class cluster around one prototypical representation of that class. They measure L.2 distance between samples in pixel space, which is sensitive to non-semantic similarities between images, such objects from different classes with a similar background and light conditions. Li et al. [15] move the similarity metric latent space and utilise an autoencoder to directly visualize the prototypes, but they only test on simple MNIST-like benchmarks. Chen et al. [16] extend the method to multiple local prototypes per class, where prototypes are compared to local image patches instead of the entire average-por -pooled output of CNN. and evaluate on the more complicated fine-grained bird classification dataset CUB [17]. We take similar local-prototype approach, but we use direct attribute and object supervision, enabling parameter sharing between classes and allowing the prototypes to be used for zero-shot classification. While for most methods these prototypes are used for the final classification, in our case they are an intermediate representation.
Graph neural networks (GNNs) are models that can work directly on the structure of a graph.
They have been first proposed by Gori et al. [18], later elaborated upon by Scarselli et al. [19] and popularised by Kipf and Welling [20] in their work on the Graph Convolutional Neural Network (GCN). GNNS grow more popular every year, and a lot of improvements have been proposed recently [21]. Just like the GCNs were inspired by CNNs, most improvements are inspired by other areas of deep learning such as attention mechanisms in Graph Attention Networks [22], but this jump from existing deep learning fields GNNs has overlooked simpler methods. Wu et al. [23] have taken a step back and stripped down GNNs to their simplest parts by removing nonlinearities and collapsing weight matrices until all that was left was feature propagation followed by linear model. In the same vein, Huang et al. [24] show that a simple Multilayer Perceptron (MLP) ignoring graph structure followed by a label propagation post-processing step often greatly outperforms GNNs. These simplified methods are mostly limited to transductive settings (test nodes are available at train time) and graphs that exhibit strong homophily (similar nodes are connected). Many real-world graphs fit those criteria, including the graph we use in this work.
3

---

softmax
P
W
max
S
LeE
P
ResNict 18
Σ
RI R 1T
H5IC y,
Low Sim.
High Sim.
Hadamard (Pointwise) Product
Dot Product
Local prototypes with softmax pooling: Local image patches Tay from the layer just before the average pooling layer of ResNet-18 are compared to prototype vectors pk: through similarity function, here a dot product. This outputs a compatibility score s. which is optimized via Loe (ef. Section 3. Similar to spatial attention, these match-prototype compatibility scores are passed through softmax function and via Hadamard (pointwise) product a weighted sum 2 of the original feature map is calculated, which are passed to the HSIC function to promote independence between object and attribute prototypes (ef. Section 3.2)
Figure 2:
Method
In compositional zero-shot learning (CZSL), we have set of images X and compositional classes with compositional labels S A x O, where A is a set of attribute labels and a set of object labels. The labels subdivided into Y Y, U Y., where Y, the seen labels for the training set and Y, unseen labels for the validation and test sets, with Y, nnnn Y. = 0. We denote the training data consisting of only compositional classes as X,. Finally, CZSL assumes that each attribute and object is seen in at least one training data point, or more formally Vp E AUO By E Y, s.t. pn + 0.
3.1 Prototype-based representations
In this section we describe how we learn local attribute and object prototypes, before they are propagated and combined into compositional prototypes in Section 3.3. Prototypes representation of target class, with strong generalization and interpretability properties. We employ direct supervision, encouraging the feature extractor to learn local image representations that encode the attribute and object labels. We use ResNet-18 [25] backbone for fair comparison other CZSL methods, but the method is backbone agnostic.
average
local prototype approach similar to Chen al. [16], though
Figure 2 shows example of a prototype layer. The layer has a set of prototype vectors P {p, } - P, RC where each target class is represented by one prototype.
|Y] is the number of attribute or object targets, and E {0. 1) indicates if we are training on attribute or object labels respectively. The layer takes as input the H W calculates compatibility score s = (ıaj. Pic) (e.g., cosine similarity, 1.2 norm, dot product) between the input patches Izy C over the spatial dimensions H x W and the prototypes. We find that a dot product similarity metric leads to better generalization.
output CNN just before its final pooling layer, and
save on parameters and avoid overfitting, we do not use fully connected layer to combine prototype similarity scores into final prediction. Instead, the maximum patch-prototype similarity score is used directly as the compatibility score for the corresponding prototype's class. This compatibility score is optimized through cross-entropy

---

loss function:
1 |S| ES
Σ log
exp(s(va) En exp(s(i))
LCE
(I)
Here, with fp our prototype layer, {s, E (fp(X.) Y.) is the set of prototype similarity scores and target labels for the training data. E (0, 1 indicates we are training on attribute or object labels respectively.
Cross-entropy loss naturally encourages clustering between samples of the same class and separation from other classes, but find that an additional loss push these properties even more is beneficial. For that purpose we adopt the cluster and separation costs proposed by Chen et al. [16]:
X
X
Clst
min min I|= p2112 PP v, ZEmy
Sep
min || Py J. SERI
min
X
X
The costs Clst. and Sep encourage a higher degree of clustering between prototypes of the same class and greater distance prototypes of different classes, respectively. The separation cost is only applied to the object prototypes, as the attribute prototypes benefit from less distant representations.
Our Hilbert-Schmidt Independence Criterion (HSIC) loss (details in Section 3.2) requires the input patches with the highest prototype similarity scores as its input, but that would require the use of the argmax function which has gradient of 0 almost everywhere. Similar to spatial attention, we use the softmax function over the similarity map as a differentiable proxy. The softmax outputs used as weights in a weighted sum of the input patches Tij serve an approximation of the input patch with the highest similarity score (cf. Figure 2), and ensure that each patch containing information about attribute or object labels is affected by the independence loss proportional to the strength of their appearance (i.e. their similarity score).
3.2 Attribute-object independence
To combat the bias in the training data we leverage a differentiable independence metric such that the model can learn representations for the attributes of images that are uniformly distributed w.r.t. their object labels and vice versa.
The Hilhert-Schmidt Independence Criterion (HSIC) [26] is kernel statistical test of independence between two random variables and B. In the infinite sample limit HSIC A, B) =0 + I1. B long the chosen kernel is universal in the sense of Steinwart [27]. We
Gaussian kernel, as we find that its ability to capture higher-order statistics is beneficial in terms of accuracy, and follow Gretton et al. [26] in setting the kernel size to the median distance between points in a batch. See the Appendix for more details. We define our independence loss function as follows, slightly deviating from Atzmon et al. [9]:
An HSIC O) + HSSCC A)
Chsic
(3)
2
where > is hyperparameter controlling the contribution of Cnsie to the final loss, 2, is the softmax- pooled output of the attribute prototype layer (ef. Section 3.1), zo the softmax-pooled output of the object prototype layer, and and the corresponding one-hot encodings of the object and attribute labels respectively.
3.3 Prototype propagation graph
Figure 3 shows the full architecture. Two types of prototype layers are trained, one on the object labels and one on the attribute labels, leaving us with centroids with confounding information removed. The prior knowledge about the compositional target classes and the attributes they afford can be represented as a compositional graph G = (P. U P. SUC.C. C. y. E). It consists of the attribute prototypes P.. object prototypes P.. and compositional classes Cy C Ax (a subset of which have no training samples). The edges are undirected, E (r, E P. UP.. E C. E y), i.e. all attribute (e.g., red) and object (e.g., cube) nodes are connected to the compositional classes they are part of (c.g., red cube). This corresponds bipartite graph with the attribute and object prototypes in one
5

---

Attribute Prototypes
Compositional Graph
Pa
Red
max
= RR Loe
Red Sphere
P.JA
Green
bero-shot)
Chaie
Object Prototypes
Sphere
Pai
Green Cube
max,
) E 0
Lee
Cube
P..o
ResNet 18
Compositional Nodes
HxW ΣΣ
S. MM
LCB
Classification argmax(s)
Average Pooling
Dot Product
Figure 3: ProtoProp: overview of proposed method. Local attribute and object prototypes are trained with independence (HSIC) loss and mapped to nodes in simplified graph neural network. which learns to combine them into compositional prototypes of both seen and unseen classes. The maximum score in the product similarity map between the compositional prototypes and average- pooled backbone output is the final classification prediction.
set and the compositional classes in the other. In the case AO-Clevr is a complete bipartite graph, but for real-world datasets such as UT-ZZapos this is not the case as, e.g., all shoes available in all materials.
As defined above, the independent prototypes mapped directly to the nodes of the graph (cf. Figure 3), and through shared weights a Graph Neural Network (GNN) learns to propagate the prototypes to the compositional nodes to form new compositional prototypes. The compositional nodes are initialized with zeros. Our GNN is a 2-layer GCN [20]; our findings are in line with the experiments of Naeem et al. [10], where they show that 2-layer GNN performs best. Inspired by SGC [23] we remove all nonlincarities, as we find that simple linear combinations lead to better generalization:
X = --1-AAA
Here X are the input nodes, O a learnable weight matrix, Å = A + denotes the adjacency matrix with inserted self-loops, and D. = Σ; Aij its diagonal degree matrix.
We then compute another dot product similarity map s. between the compositional prototypes and the average- pooledd output of the backbone, which is again optimized via cross-entropy loss function. Our final compositional classification prototype with the maximum similarity score. The shared weights in the GNN ensure that the model learns a general composition of attributes and objects that can generalize to unseen compositional classes. By initializing the node features with independent prototypes, the graph is able to learn the dependencies encoded by the compositional graph, including the novel zero-shot classes, instead of the biases of the training data.
= argmax(s.). i.e., the class corresponding to the compositional

---

4 Experiments
4.1 Implementation
We implement our model using the PyTorch library [28], using some of the boilerplate code provided by Nagarajan and Grauman [29] and Purushwalkam et al. [7]. The GNN is implemented using PyTorch Geometric [30]. For our independence loss we use normalized implementation of HSIC provided by Ma et al. [31]. We use the Adam [32] optimizer. Experiments have been run on an 11GB GeForee GTX 1080 Ti graphies card. On AO-Clevr we reach our best result in 1-5 epochs on all splits, at -5 minutes per epoch. On UT-Zappos we reach our best result in -30 epochs at -1.5 minutes per epoch. On C-GQA we reach our best result in 5-10 epochs. See the Appendix for detailed hyperparameters and grid search ranges.
4.2 Evaluation
We evaluate our approach on three CZSL datasets. Previous works also evaluate on MIT-States [33], but Atzmon et al. [9] conclude through large-scale user study that the dataset has a level of -70% label noise, making it too noisy for evaluating compositionality. All three datasets evaluated in this paper do have some limitations that are relevant for future works, which we describe in the Appendix.
AO-Clevr [9, 34] is synthetic dataset consisting of types of objects (sphere, cube, cylinder) and attributes (red, purple, yellow, blue, green, cyan, gray, brown), with 24 compositional classes in total. There are splits with a varying ratio of unseen to seen classes, ranging from 2:8 to 7:3, allowing insights into the performance of models as the proportion of unseen classes increases.
UT-Zappos [13] is a fine-grained dataset of types of shoes with 12 objects (e.g., boat shoes), 16 attributes (e.g., leather), and 1 16 compositional classes. We use the split proposed by Purushwalkam al. [7], with 83 seen classes in the training set, 15 seen and 15 unseen classes in the validation set, and 18 seen and 18 unseen classes in the test set.
C-GQA [10] is based on GQA [35], large-scale object detection dataset, that has been modified to fit into the CZSL setting by filtering on bounding boxes of objects that are only described by single attribute.
Metries: like other recent works we adopt the generalized zero-shot evaluation protocol proposed by Purushwalkam et al. [7], Chao et al. [36]. Instead of evaluating only on the unseen classes as in the Closed setting, the Generalized setting considers both seen and unseen classes the validation and test sets. To account for the inherent bias towards seen classes, Chao et al. [36] add calibration bias term to the activations of unseen classes. As the value of the bias is varied between -oo and +oo, they draw curve with the accuracy on the unseen classes on the axis and the accuracy the seen classes on the x axis, and report the Area Under the Curve (AUC). The harmonic mean is defined as 2 Acc. Accu, where Acc, is the accuracy on the seen classes and Acc, the accuracy on AAc,,
the unseen classes. penalizes large differences between the two metries and as such indicates how well the model performs on both seen and unseen classes simultaneously. We also follow prior works in reporting the Closed seen accuracy where only the seen classes are considered (corresponding to a bias of (o) and closed unseen accuracy where only the unseen classes considered (corresponding to bias of +oc). For the results on AO-Clevr we report the seen and unseen accuracy components of the harmonic mean in accordance to Atzmon et al. [9]. Atzmon et al. [9] do not perform post-hoc bias calibration, but instead handle the seen-unseen bias during training.
Like Naeem et al. [10], we train our feature extractor end-to-end with the rest of our model. Other methods keep the backbone fixed, but Naeem al. [10] have shown that they perform worse when allowed to finetune they will then start to overfit. While our method is designed to take full advantage of the backbone, we show our results with a fixed backbone in our ablations.
4.3 Results
For each of the results we select the best model by the best harmonic mean on the validation set and report the results for all metrics on the test set. The results with error bars come from training runs with random initializations. We note that, unlike other methods, Causal [9] does not perform post-hoc
7

---

Table 1: UT-Zappos and C-GQA results, ProtoProp improves on state-of-tte-arr results in the AUC and harmonic mean metrics.
UTPZaapps
C-GQA
Method
AUC
Seen
Unseen
Harmonic
AUC
Seen
Unseen
Harmonic
AuOp [29]
59.8
LE+ [6]
25.7
53.0
61.9
41.0
1.2
16.1
5.0
5.3
SymNot [8]
23.9
53.3
57.9
39.2
25.2
9.2
TMN [7]
24.7 4.4
58.8 1.4
49.4 4.7
40.7 1: 2.0
2.2
21.6
6.3
7.7
Causal [9]
23.3
53.4 +: 0.8
31.8 1 1.7
OGE [[]]
32.5
61.0+
65.9
1.2
47.1
31.4
14.0
14.5
ProtoProp" (ours)
34.7 1: 0.8
62.1 1: 0.9
65.5
1: 0.2
50.2 1: 1.3
3.7 1 0.1
26.4 4: 1.2
18.1
0.9
15.1 1 0.2
Usiin finetuned features
bias calibration, but instead tackles the bias against seen classes during training, which is a benefit of their approach.
AO-Clevr Figure shows results on AO-Clevr. We also report the results for CGE [10] (after grid search using their own codebase) as they have not reported experiments on this dataset. See Appendix for the exact values and error bars. observe that for the 3:7, 5:5, 6:4 and 7:3 splits, one or more colors are not part of the training data, making a large portion of the validation and test classes impossible to classify through compositional methods on those splits. Our method performs better than the compared methods on all splits, with minimal impact the seen accuracy. The improvements are in the range 2.5 to 20.2% for the harmonic mean and 3.3 to 17.0% for the unseen accuracy, with higher gains the proportion of unseen classes increases significantly.
Harmonic performance
Unseen performance
performance
80
Accuracy (%)
60
40
ProtoProp
ProtoProp
ProtoProp CGE
CGE
CGE
20
Causal
Causal TMN
Causal TMN
TMN
2:8
3:7
4:6
5:5
6:4
7:3
2:8
3:7
4:6
5:5
6:4
7:3
2:8
3:7
4:6
5:5
6:4
7:3
Ratio Unseen:Seen
Ratio Unseen:Seen
Ratio Unseen:Seen
Figure 4: A0-Clevr results Plots of the seen and unseen accuracy and their harmonic mean (y- axis) as the ratio of unseen:seen compositional classes (x-axis) increases. ProtoProp consistently outperforms state-of-the-au methods, especially when the portion of unseen classes grows.
UT-Zappos Table shows the results on UT-Zappos. Because earlier works only report single best run, we report the average over random initializations including standard error for the two best per- forming previous models (CGE and TMN), using their own codebase and reported hyperparameters. We find that, especially for this dataset, the error bars are important since it highly susceptible to random initialization, where 'lucky' initialization can result in significantly larger harmonic mean. Our method outperforms earlier methods on all metrics except for the closed unseen accuracy where CGE performs slightly better. Our method performs especially well when both seen and unseen classes are taken into account, as evidenced by the AUC and harmonic mean improvements.
C-GQA Table shows the results on C-GQA. We report the average of our results over 5 random intializations. Because of the scale of the dataset we were not able do the same for related work. as we do not have enough compute to run experiments using their preferred hyperparameters. Our method trades some seen accuracy for more aceuracy on the unseen data, and slightly outperforms the previous best method. We find that the C-GQA dataset has some limitations that makes it less suitable for object-centric classifiers or a single-label setting such as CZSI., which we discuss the Appendix.

---

Table 2: Ablation on the AO-Clevr 4:6 split: Checkmarks indicate whether prototypes are used as node features, semantic vectors as node features, local prototypes are trained with independence loss, or the backbone is frozen respectively. Prototypes trained when semantic node features are used are not part of the final classification but still improve the feature extractor output.
Node Features
Indep. Proto
Finetune
Seen
Unseen
Harmonic
Visual
94.5 + 0.1
77.0 = 2.8
$4.81.1
Visual
78.2 + 1.1
73.0 = 1.0
75.5 0.8
Visual
V
97.9 + 1.0
95.5
96.7 = 0.7
Semantic
95.4 1.1
82.4
88.4 0.1
Semantic
97.3 + 1.2 84.5 + 1.4
90.4 + 0.9
Table 3: Ablation on the AO-Clevr 4:6 split: Additional ablations on the linear vs. gaussian kernll for Chsic. dot vs. cosine distance function, and linear vs. nonlinear GNN.
Ablation
Seen
Unseen
Harmonic
Linear kernel
97.0 0.9
71.6 + 1.2
82.4 + 1.4
Cosine
89.2
64.0 + 2.7
74.5 3.8
Nonlinear GNN
95.0 + 0.7
76.2 0.4
84.6 + 0.8
ProtoProp
97.9 1.0
9.9
96.7 0.7
4.4
Ablations
To determine the effect of our visual features in the form of prototypes, in contrast to the semantic features in most prior works, we perform ablations where replace our prototypes with pretrained word embeddings. We also take a look at the importance of the independence loss function and finetuning the backbone.
Early ablations Table 3 shows the results of our ablations for several design choices that were mentioned in the method section, specifically the linear vs. gaussian kernel for LHSIC. dot vs. cosine distance function, and linear vs. nonlinear GNN, with all other parameters equal to the final ProtoProp architecture. We note that these choices have been made early on after empirically choosing the option that performed best. The cosine distance metric, for example, was more difficult to optimize and takes twice long converge compared to the dot product. Similarly, nonlinear GNN may be easier to optimize for our method because the node features are prototypes which (as opposed to semantic features) are already in the visual domain, and therefore don't 't require nonlinear transformations. As such, there may be more optimal settings where these alternative options perform better than we have reported here.
Importance of prototypes Table 2 shows the results of our ablations on the 4:6 split on AO-Clevr. Row shows the results for our proposed method without the independence loss function, in row 2 we add the independence loss but freeze the backbone, and row 3 shows the results with both independence and finetuning, described in the method section. The prototypes perform worse than semantic embeddings when used as node features without the independence loss, but with the independence loss they receive significant gains and come out top with 8.3% increase in harmonic mean accuracy over the semantic features. With a frozen backbone the harmonic mean for method is just slightly higher (0.8%) than Causal on the same split, though it reaches that accuracy in a fraction of the time.
Visual vs. semantic For the ablation in row 4 we use semantic word embeddings (word2vec [37]) as node features, which is equivalent to Nacem et al. [10] with a simplified GCN. In row 5 we also train our local prototypes but don't It use them during classification, only to influence the local features the feature extractor learns. Training the local prototypes improves the results slightly even when they are not used for the final classification, by encouraging the backbone to learn local features that represent the target attributes and objects.
Effect of independence Table 2 shows that the prediction accuracy breaks down when not using the independence loss. Figure 5 provides some intuition for this through a t-SNE [38] plot of softmax- pooled attribute (in this case color) representations from our model on AO-Clevr. Without the loss there are three strongly separated clusters per color (one per shape), and many colors are embedded closer to different colors of the same shape. When do use the independence loss, each of the

---

Without HSIC
With HBC
Figure 5: t-SNE plot of cettmax-pooledd color patches for AO-Clevr that were trained with (left) and without (right) HSIC loss, where with the HSIC loss we get homogeneous grouping of the attributes and fragmentation by spurious confounders.
colors is part of their own single cluster and the only correlations left are those between visually similar colors, improving the accuracy of the compositional prototypes after the propagation step.
Limitations
Like typical CZSL works, our method is limited to a single attribute and object label per image, and each individual attribute and object needs to be seen in at least training data point. Existing attribute datasets that we have evaluated are often either noisy or do not work well in the aforemen- tioned single-attribute setting. But if this initial hurdle of quality datasets is overcome, compositional methods make it easier to handle the addition of new classes, especially for rare objects with little to available images. To extend the scope we would extend the method to multiple attributes, e.g., through attribute prototype regression [39], or by leveraging textual descriptions that can be scraped from the web in semi-supervised fashion [40].
realistic settings and other datasets, in future work
The hyperparameters can be difficult to tunc, as there are hyperparameters for the backbone, pro- totype layer, independence loss, and compositional GNN. We do find, however, that the prototype layers can be tuned separately from the rest, as the best performing individual prototypes and the hyperparameters that led there also perform best when used in conjunction with the GNN. As such, most hyperparameters can be adopted from existing prototype-baa bbatee classification works that have been trained similar dataset.
Conclusion
In order to have chance at recognizing the long-tailed distribution of visual concepts, our models need to become better at recognition through shared visual primitives. To this end, we have proposed a novel prototype propagation method for compositional zero-shot learning. Our method learns prototypes of visual primitives that are independent from the other visual primitives they appear with, and propagates those prototypes through a compositional graph in order to recognize unseen compositions. The method works with just the attribute and object annotations, and, as such, is not reliant on external sources of information, such as hierarchy graphs or pretrained semantic embeddings. We evaluate work on two CZSL benchmarks, and improve ffate-ofthhe-ar results, especially with large fractions of unseen classes, with minimal impact on the accuracy of the seen classes.
Broader impact Although not intended or designed for particular applications that are harmful, our method could be misused, deployed as part of unethical applications in the real world; this can only be prevented by an ethical review for each application. Furthermore, while we attempt to deal with bias in recognition, this not silver bullet, and as such in each application continuous analysis needed to ensure that all assumptions continue to hold.
10

---

References
[1] Martin N Hebart, Charles Zheng, Francisco Pereira, and Chris Baker. Revealing the multidi- mensional mental representations of natural objects underlying human similarity judgements. 2020.
Nature Human Behaviour, 4
[2] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[3] Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum. Learning to share visual appearance for multiclass object detection. In CVPR 2011, pages 1481-1488. IEEE, 2011.
[4] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[5] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9012-90200 2019.
[6] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composi- tion with context. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1792-1801, 2017.
[7] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3593-3602, 2019.
[8] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compositions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11316-11333 11325, 2020.
[9] Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33, 2020.
[10] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. arXiv preprint arXiv:2102.01988 2021.
[11] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 3/ International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[12] Tristan Sylvain, Linda Petrini, and Devon Hjelm. Locality and compositionality in zero-shot learning. In International Conference on Learning Representations. 2019.
[13] Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.
[14] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Re- thinking few-shot image classification: a good embedding is all you need? arXiv preprint arXiv:2003. 11399, 2020.
[15] Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: neural network that explains its predictions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[16] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32:8930-8941, 2019.
[17] C. Wah, Branson, Welinder, P. Perona, and Belongie. The Caltech-UCSI Birds-200-2011 Dataset. Technical Report CNS-TR-2011-00 California Institute of Technology, 201

---

[18] Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729-734. IEFE, 2005.
[19]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions Neural Networks, 20 (0)1))61-80 2008.
[20] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[21] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[22] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference Learning Representations, 2018. URL httph:t//ppptrwweieeee
wtntt/orrmmiddi-i JXMpikkk..
[23] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning, pages 6861-6871. PMLR, 2019.
[24] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label propagation and simple models out-performs graph neural networks. In International Con- ference Learning Representutions, 2021. URL https: /tttps://openrevieww wttpeviwwwwwweeeeeeeeemmmeeeeeee mest/forum?id 8E1 HE1-f3VhX1o.
(25) Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[26] Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song. Bernhard Schölkopf, Alexander J Smola, al. A kernel statistical test of independence. volume 20, pages 585-592. NeurlPS, 2007.
[27] Ingo Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of machine learning research, 2(Nov):67-93, 2001.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and Gamnet,, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http:///pppppe neurips cc/paper/ 9015-pytor an- imperative- htwprrreewettte-ieggeeeeerrrrrre deppipp-eerrrrraa-deeeeee--p-p Learning wwwaarninnnnnniarrrrrreeaa library. pdf.
[29] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 169-185, 2018.
[30] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[31] Kurt Wan-Duo Ma, Lewis, and W. Bastiaan Kleijn. The HSIC bottleneck: Deep learning without back propagation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5085-5092. AAAI Press, 2020.
URL https:/ :/ //a.aaaaiaaaaaa///aa//aaaa/aaaa/ org/ jjs/ /tt///ndxx.eee//eee www.ndewwpppp/e/ee AAAI/ www.iceeeiiee..ee.eee.....
12

---

[32] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[33] Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383-1391, 2015.
[34] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pages 2901-2910, 2017.
[35] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700-6709 2019.
[36] Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis generalized zero-shot learning for object recognition the wild. In European conference computer vision, pages 52-68. Springer, 2016.
[37] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26:31 1-3119, 2013.
[38] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[39] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. 34th Conference on Neural Information Processing Systems. Curran Associates, Inc., 2020.
[40] Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1004-1013, 2018
13


Reviews:
[{"review_id": "tnREwDEMNN6", "reviewer": "Reviewer_nHAS", "summary": "The paper deals with the problem setup of compositional zero-shot recognition. Where a visual object is specified by two labels: an attribute name and an object name. Not all attribute-object combinations are available at training time, and at test time, a model may observe new (unseen) combinations.\n\nThe approach combines ideas from two recent publications: [1] Atzmon et al, 2020, and [2] Naeem et al, 2021: It uses the idea of encouraging independent prototypes for the attribute and object (Atzmon), and then it uses them as the node features for the graph-NN used by (Naeem), replacing than the word-embedding used in (Naeem).\nIn addition, it proposes a spatial attention mechanism over CNN feature maps for extracting the prototypes, rather than the average-pooling used by previous baselines.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Originality: \nThis work provides a novel combination of recent techniques (Atzmon, 2020) and (Naeem, 2021) - the idea is simple, and may be considered as immediate, but that's what I find as appealing in this approach.\n\n### Quality: \nThe approach is technically sound, however there are issues with several key points:\n\n* It will be beneficial to report the results of the model with the recent C-GQA benchmark (Naeem 2021).\n\n* One claim uses a wrong terminology, but it is not a major issue because this is an architecture paper rather than a theoretical paper. Specifically, the paper claims that its loss function promotes *conditional*-independence of prototypes (as in Atzmon), where in fact, the loss term in eq (3) promotes *\"full\"* independence between prototypes. Promoting full independence may be problematic when used as a hard constraint (e.g. with a very large loss coefficient), because the attribute and object are not fully independent in the training data, but empirically it can work to some extent with a careful hyper-param tuning.\n\n* Some claims are supported by the ablation study. However evidence is missing to several other claims. Specifically: (1) Superiority of Gaussian Kernel over Linear Kernel. (2) Superiority of a dot product similarity metric (3) superiority of simple linear combinations (4) benefit of cluster and separation loss.\n\n* The reported metrics for Zappos are missing the Open Seen/Unseen metrics\n\n* It will be more fair to cluster the compared approaches according to fine-tuned vs non fine-tuned.\n\n* It will be beneficial to include an ablation study for the attention mechanism vs average pooling.\n\n### Clarity:\nThe paper is mostly clear. Some sections  were hard to understand:\n\n* Except for the linear activation, does the graph-NN architecture is identical to Naeem et al? Please discuss that in related work.\n* For the HSIC kernel, how was the median distance between points calculated? If I understand correctly, this distance changes every time the model weights are updated. Therefore, it is not clear how to select this value if the distance keeps changing after every gradient update.\n* Figure 2 and its caption is unclear.\n* Paragraph in lines 147-154\n\n### Significance:\nCompositional generalization is a challenging and important problem in machine-learning models. The approach provides a simple combination of recent techniques using spatial attention, and provides promising results.\n\n\n\n### Additional comment to authors:\n* The claim \"we do not use a fully connected layer\" is not accurate: Each of the learned prototypes may be viewed as a vector of a matrix that parametrizes a fully-connected layer. \n\n\n### Post rebuttal:\nI thank the authors for their answers. Following the rebuttal I decided to raise my score to 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations were adequately discussed\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LSiIKefN7JW", "reviewer": "Reviewer_iHTC", "summary": "The paper proposes ProtoProp model for compositional zero-shot learning, which propagates the learned object and attribute prototypes with independency constraints by a compositional graph of GNN. The compositional prototypes are then used for detecting unseen classes. Experiments are done on AO-Clever and UT-Zappos benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The proposed ProtoProp model is validated on two zero-shot learning benchmarks with strong performance improvement.\n\n2) The idea of compositional prototypes from attribute and object prototypes by GNN propagation is novel to me.\n\n3) The paper writing is clear and well organized.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1) Using HSIC loss to maintain the attribute-object independence achieves improved embedding property but it seems a direct adaptation from Atzmon et with different kernels. But it has no experiment table to validate the improvement.\n\n2) Missing detailed discussion or comparison to some related works on learning the prototype-based representations, such as [a, b]. It would be interesting to see the influence of prototypical representations obtained by the different methods.\n[a] PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment. ICCV, 2019.\n[b] Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation. Arxiv.\n\n3) Missing detailed failure cases analysis, and qualitative visual results. \n\n4) In Table 2, missing discussion on why finetuning is critical to visual node features but has smaller influence on the semantic node feature?\n\n5) What's the influence of number GNN layers/design choices to the propagation and final performance.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "J3tnH5OWpj6", "reviewer": "Reviewer_jpXL", "summary": "The submission focuses on compositional zero-shot learning, in this context a type of zero-shot problem where instances are a pair of images and compositional labels (object/attribute information). \nThe authors start with the standard prototypical networks setup, and add auxiliary losses in order to learn object and attribute prototypes. A HSIC loss is added to encourage object-attribute independence (for better compositional generalization).\nFinally, a graph propagation algorithm is applied to combine the prototypes into compositional prototypes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects:\n- The paper is well-written, the ideas are clear.\n- Empirical evaluation is satisfactory: there is an ablation study and a detailed discussion of the impact of the different components.\n\nLimitations/to improve:\n- I disagree with the author's claim that attribute datasets are either noisy or do not work in the single attribute setting. As an example, Caltech Birds (CUB) can be easily modified to account for a single attribute (e.g. color). More generally, one of my main issues is that the datasets considered are quite simple compared to the more complex \"real-world\" datasets more commonly seen in the classical zero-shot literature. Could the authors adapt their model and baselines to e.g. Caltech Birds or a related dataset and show that their approach indeed generalizes beyond toy datasets?\n- The related works section is missing some works, notably [1] which is highly relevant to the discussion on compositional generalization. They propose a method, by no means the only one, to measure the compositionality of a representation. It would be interesting to see if there is indeed a link between a measure of compositionality (not necessarily theirs) and downstream performance of the representations in this submission. In a similar vein, I would mention [2] who take a more applied standpoint on compositional generalization.\n\nAfter rebuttal: looking at the author response, I have raised my score to a 7. Cf. my response to their point for justification.\n\nReferences\n[1] Tristan Sylvain, Linda Petrini, and Devon Hjelm. \"Locality and compositionality in zero-shot learning.\" ICLR 2020\n[2] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata,\"Attribute Prototype Network for Zero-Shot Learning.\" NeurIPS 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations/potential negative societal impact. Indeed, they mention that there are no obvious issues, but that misuse can always happen, and that therefore caution should be exercised on a per-case basis.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not find any ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Compositional generalization is an important problem in machine learning. The problem setup is well motivated and the paper is well written. The proposed method is simple and effective.  It combines several existing techniques with innovations. All reviewers agree the acceptance. 
Decision: Accept (Poster)