Locality and Compositionality in Zero-Shot Learning
Authors: Tristan Sylvain
Linda Petrini
Devon Hjelm
Venue:  (2020)
Keywords: Zero-shot learning
Compositionality
Locality
Deep Learning

Abstract: In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). 
In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSL, no pre-training on different datasets (e.g. ImageNet) is performed.
The results of our experiment show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus on more local-aware models in future research directions for representation learning.

Published as a conference paper at ICLR 2020
LOCALITY AND COMPOSITIONALITY IN ZERO-SHOT LEARNING
Tristan Sylvain
Linda Petrini
Mila, Université de Montréal
University of Amsterdam
Montreal, Canada
Amsterdam, Netherlands
tristan. sy sy v www.esylliilllll..oomommmmmmmmmm com
1 Lindapet r rini @ com
R Devon Hjelm
Microsoft Research, Mila Redmond, USA
ABSTRACT
In this work we study locality and compositionality in the context of learning representations for Zero Shot Learning (ZSL). In order to well-isolate the importance of these properties in learned representations, we impose the additional constraint that, differently from most recent work in ZSI., no pre-training on different datasets (e.g. ImageNet) is performed. The results of our experiments show how locality, in terms of small parts of the input, and compositionality, i.e. how well can the learned representations be expressed as a function of a smaller vocabulary, are both deeply related to generalization and motivate the focus more local- models in future research directions for representation learning.
INTRODUCTION
crucial property of a useful model is to generalize, that is to perform well on test settings given learning on training setting. While what is most commonly meant by generalization is being robust to having limited number of training examples in instribttionally-mathhh 2016). many tasks are designed to address variations in the data between when a model is trained and when it is evaluated. For instance, some classification tasks address distributional changes in the input: from lacking guarantees of distributional match between train and test (e.g., covariate shift, Shimodaira 2000) to having fundamental domain differences (e.g., domain adaptation, Crammer et a 2007: Ben-Davil et al., 2007). A number of tasks have also been designed specifically to understand models terms of their ability to generalize to test situations that are poorly represented during training (c.g., Few-Show learning. Li ct al.) (2066), or even consist of a diverse and entirely novel set of sub-tasks (aaiir et al.) 2018). For supervised classification. Zero-Shot Learning (ZSL, Larochelle et al.) 2008) is among the most difficult of these tasks, as it requires the model to make useful inferences about (e.g., correctly label) unseen concepts, given parameters learned only from seen training concepts and additional high-level semantic information.
settings (Zhang et al.
The fundamental question we wish to address in this work is: What are the principles that contribute to learning good representations for ZSI.? While the most successful 7SL models (Atzmon & Chechik
2019 Wang al.) (019)) use pretrained features from Imagenet (Krizhevsky al.] 2012) Russakovsky et al. 2015), we wish to understand how these features can emerge given only the data provided from
the ZSL task. Specifically, we explore the role of compositionality and locality (Tokmakov et al.)
2018 Stone et al. 2017) as two principles that lead to good generalization. Our study focuses on image representations, so we explore various means of learning representations that are local and compositional for convolutional neural networks (CNNs). We also leverage the structure of CNNs and available annotations from 7SL. datasets as means of interpreting various models terms of these factors. Overall, our results support the hypothesis that compositionality and locality are crucial principles for training models that generalize well.
"Equal contribution: order between first authors is arbitrary.

---

Published as a conference paper at ICLR 2020
Finally, in order to provide cleaner framework for understanding the relationship between the above principles and generalization, we re-introduce Zero-Shot Learning from scratch (ZFS). In this setting, the model is not allowed to be pretrained on another dataset, such as Imagenet, and is evaluated on its ability to perform classification using auxiliary attributes and labels trained only using the data available from the training split of the target dataset. We believe that ZFS will provide researchers with a better experimental framework to understand which principles are important for Zero-Shot generalization.
The contributions of our work
follows:
We introduce Zero-Shot Learning from scratch (ZFS), an extension to ZSL., which we believe will be an important benchmark for understanding which learning principles lead to better generalization. We evaluate several supervised and unsupervised methods on their ability to learn features that generalize in the ZFS setting by training a prototypical network on top of those features (in a similar way to what was done in Snell al. 2017 with Imagenet features). We then relate this generalization performance with different proxies for locality and compositionality of the given
representations, and show that both concepts contribute heavily.
We introduce a novel version of Deep InfoMax (DIM. Hjjelm et al. (018)) which draws local patch
representations from other images with the same label as positive samples.
We introduce novel visualization technique based Mutual Information, that allows to investigate
local properties of the learned representations.
2 PRINCIPLES THAT LEAD TO GOOD ZSL PERFORMANCE
Zero-Shot Learning (ZSL, Larochelle et al., 2008) is an important learning framework for under- standing a model's capacity be used in real world scenarios where many relevant test cases (e.g.. classes) are not known or are infeasible to sample at training time. An important component of ZSL, particularly in Deep Learning, is learning features directly from raw data (e.g.. pixels), that generalize to these test cases. While there a number of commonly-use generalizable features for various tasks in Deep Learning (Neyshabur et al., (017)). we believe that ZSL in particular requires thinking beyond normal classification by incorporating principles such as compositionality (Boole (1844) and locality (Fukushima) 1980).
strategies for learning
How we formulate, exploit, and analyze these principles to learn models that solve image 7SL. tasks will depend
of convolutional neural networks (CNNs) the network architecture for encoding images as well as properties of the data, such as input statistics and available annotations. We will broadly define compositionality and locality, then relate these principles to the tools we have at our disposal from the network architecture and data.
2.1 COMPOSITIONALIIT
Compositional representations have been a focus in the cognitive science literature (Biederman) 1987, [offman & Richards 1984) with regards to the ability of intelligent agents to generalize to new concepts. Applications are found in computational linguistics (Tian et al. 2016), generative models (Higgins et al.] (017b). and Meta Learning (Alet et 2018 Tokmakov et I 2018), to name few, with approaches to encourage compositionality varying from introducing penalties (Toomakov ctt al. 2018) to using modular architectures (Alet et al.) 2018).
In what follows, we will consider a representation to be compositional if it can be expressed as a combination of simpler parts (Andreas) 2019 Montague| 1974). Let P denote the set of possible parts, R the representation space and X the input space. For each x E X, we assume the existence of function 1 mapping x to P' CP, the set of its parts. These parts could be local image features (c.g., wings or beaks). or other generative factors (c.g., size, color, etc). Let g: P P R be a function that maps the parts to representations. Formally, f(x) E R is compositional it can be expressed combination of the elements of {g(p)(p D(z)). The combination operator used is commonly a weighted sum (Brendel & Bethge (019b). although some works learn more complex combinations (Higgins et al. (017b). As we consider representations that are implicitly compositional, the above formalism might be approximately true which motivates our later use of the TRE metric.
2.2 LOCALITY
Local features have been used extensively in representation learning. CNNs exploit local information by design, and locally-aware architectures have been shown to be useful for non-image dataset, such as
2

---

Published as a conference paper at ICLR 2020
graphs (Kipf & Welling 2016) and natural language processing (Yu at. 2018 For supervised image classification, a bag of local features processed independently can surprisingly well compared to processing the local features together (Brendel & Bethge, 2019a). Attention over local features is
commonly used in image captioning (Li et a 2017), visual question answering (Kim al. 2018 and fine-granne gra-igr classification (Sun et al. 2018 Self-attention over local features resulted in large improvements in generative models (Zhang et al. 80) Self supervise methods often exploit local information to learn useful representations: Doersch et al. (2015) proposes to learn representations by predicting the relative location of image patches, Noroozi Norooz & Favaro (2016) solves jigsaw puzzles of local patches, and Deep InfoMax (DIM, Hjelm et al. 201 8) maximizes the mutual information between local and global features.
For our purposes with image data, we loosely define a local representation as one that has information that is specific to patch. This helps motivate choices in architecture and learning principles that encourage locality. In this work, we take the most straightforward approach, and use features of CNNs which have receptive fields that are small compared to the size of the full image. This is similar to the motivations in Bachman et al. (2019), where the architecture is carefully designed to ensure that the receptive fields do not grow too large. However, this choice in architecture does not guarantee locality, as CNN representations might hypothetically contain only global information, such as the class or color of the object, despite having limited receptive field. Therefore, we will evaluate number of different models on their ability to encode only information specific to those locations. We will discuss relevant evaluation in Sections 4.2 4.3
Note also that compositionality as discussed above and locality are not necessarily independent concepts, nor are they necessarily the same, The set of compositional factors could include local factors, such as parts of an object, but also be more "global" factors, such as general properties of a class (e.g., size, color, shape, etc).
2.3 COMPOSITIONALITY AND LOCALITY WITH IMAGE DATA
We focus on three common ZSL dataset that allow us to explore compositionality and locality, namely Animals with Attributes 2 (AwA2, Xian et al.] 2018).
(CUB, Wah et a 2011 D). SUN Attribute (SUN, Patterson & Hays 2012). Typical images from these datasets are shown in Fig. [
CUB a fine-grained dataset, where the object of interest is small relative to the total image. This is in contrast to AwA2, where subjects have variable size in relation to the total image. SUN a scenes dataset, meaning that the object of interest often the whole image.
Figure 1: Typical samples
show how compositionality and locality are expressed differently in the datasets we consider in this study.
AmA2
SUN
In our evaluation of compositionality. we can leverage different annotations provided by the datasets. All of these datasets provide attributes, which roughly correspond to high-level semantic information composed of a set of underlying factors. For CUB, these attributes describe visual characteristics such as wing colour or shape. For Aw these describe both visual and behavioral characteristics, such as the animal's ability to swim or its habitat. For SUN, the attributes are more diverse, ranging from describing the function of the scene, such as playing, to the spatial envelope, for instance man-made. As in Xian et al. (2077). we used &g normalized versions of these attributes as semantic class representations. In addition to attributes, CUB comes with bounding boxes for the whole subject and parts locations for 15 different parts, e.g. beak, tail, belly. These can also be used to assess both locality and compositionality, with the compositional factors being the same as the local ones. The details of each dataset are in the Appendix [Table
3

---

Published as a conference paper at ICLR 2020
3 HERO-SHOT LEARNING FROM SCRATCH
While the original ZSL setting introduced in Larochelle et al. (2008) was agnostic to the exact methodology, more recent image ZSL approaches almost uniformyy m use features from very large "backbone" neural networks such as Inception V2 (Szegedy et al. 2016) and ResNet101 (He et al. 2016) pretrained on the Imagenet dataset. In terms of absolute performance, this approach appears to be well-justified, as state-of-the-art results on various ZSL (Yosinski et al. Sun et al.
201 Huh et al.) 6016|Azizpou et al. 2013) and non-ZSL benchmarks (Li et 2019 20191/Zhng al. 2018b He & Peng 2017) learn on top of similar pretrained backbones.
However, we have many concerns with this approach towards our goal of understanding the principles that contribute good generalization. First, relative success in transfer learning has been shown to be highly dependent on the precise instantiation of the pretrained backbone encoder (Xian) et al. 2018) or the pre-training dataset (Cui et al. 2018). Next, while Imagenet features have been shown to work well for ZSL. tasks with similar image datasets, there no guarantees that a suitable pre- tre-training framework would be available in general ZSL settings. Conversely, it can be hard in practice to meaningfully evaluate a Zero-Shot learner, as performance specific classes is impacted by their presence in the pre-trate pre-training dataset (Xian et al. (0177)
Finally, we believe this approach misses the point, in such a way that makes understanding the learning principles that contribute to good generalization difficult. We believe that ZSL should first and foremost be used as framework for training, understanding. and evaluating models on their ability to reason about new, unseen concepts. Despite the absolute performance gains of the methods above that use Imagenet features, the use of backbones hyper-optimized for supervised performance Imagenet and the Imagenet dataset itself represent nuisance variables a larger effort to understand how to learn generalizable concepts from scratch.
In addition to the ZSL task framework outlined in Larochelle et al. (2008 ZFS simply adds one additional additional requirement: No model parameters can contain information about (e.g., can be learned from) data outside that from the training split the target dataset.
4 METHODS
Given our hypothesis on the importance of locality and compositionality, we consider a wide range of representation learning methods trained using the ZFS setting described in Section To showcase the role of these principles, will introduce set of proxies for compositionality and locality below. We will also consider auxiliary losses that emphasize locality the learned representation. Finally, we will introduce a visualization tool that can help identify which local representations are assigned higher importance by each method.
4.1 GENERAL APPROACH
In this work, we train convolutional image encoders (CNN) using either supervised or unsupervised learning, then use prototypical networks to perform ZSL. transfer on these fixed representations. Prototypical networks are chosen they require minimal parameters are well-studies (Huang al. 2019 Finn et al. 2017). and performance is very close the state of the benchmarks. Our setup is representative of the current state of ZSL models,
tuning,
art for mmagenet-preeraiig most of which (Akata et al. 2015 Changpinyo et al. 2016 Kodirov et al. 2017 Zhang et al. 2017] Sung et al.) 2018) rely on metric learning by applying two steps: (1) learning suitable embedding function that maps data samples and class attributes to a common subspace, (2) performing nearest neighbor classification at test-time with respect to the embedded class attributes.
For our study, we compare pre-training the image encoder with a diverse, yet representative set of models:
Fully supervised: Fully supervised label classifier (FC)
Unsupervised / reconstruction based generative: Variational auto-encoders( VAE, Kingma & Welling 2013), 3-VAE (Higgins al. 2017a), Adversarial auto-encoders (AAE, Makhzami et al. 2015),
Local self-supervision and variants: Augmented Multiscale Deep InfoMax (AMDIM, Bachman al. 2019) and Class Matching DIM (CMDIM).
4

---

Published as a conference paper at ICLR 2020
We pick variants of DIM (Hjelm al.) 2018) opposed to other self-supervision methods (Doersch & Zisserman 2017; Noroozi & Favaro (016) because extensions have achieved state-of-the-eee on numerous related tasks Veličković et al. 20188 Bachman el 2019).
We introduce Class- Matching DIM (CMDIM), novel version of DIM that draws positive samples from other images from the same class. The goal is to learn representations that focus less on the information content of a single input, while extracting information that is discriminative between classes. The hyperparameter p determines the probability of performing intra-class matching, experiment with e (1.0.5,0.13.. A more detailed description is provided in Appendix G
Local classification and attribute auxiliary loss We encourage the image encoder to extract semantically relevant features at earlier stages of the network by introducing an auxiliary local loss to the local feature vectors of a convolutional layer (whose receptive field covers small patch of the image). When used, this auxiliary loss is either from an attribute- saribute-bbasd classifier (AC) or label-based classifier (LC) using the attributes or the labels as supervised signal, respectively. A schematic explanation can been seen in Fig. 7 in the appendix.
4.2 PARTS CLASSIFICATION FOR CUB EVALUATION
For each of the 15 parts labelled in the CUB dataset, we the MTuurk worker annotations to construct 15 boolcan map for each local feature. We project these boolean maps through the CNN to generate ground truth variables that indicate whether the given part is present and visible at a location specific to the CNN encoder features. We then train linear probe for each part, without back-propagating through the encoder, and measure the average Fl score across all locations and parts. This gives us measurement on how well the encoder represents the parts of the image at the correct locations. For more details, please refer to Section in the Appendix.
4.3 MEASURING MUTUAL INFORMATION BETWEEN LOCAL FEATURES OF DIFFERENT IMAGES
As a tool for local interpretability, we propose estimating the mutual information (MI) between global features given from one image and local features from second image. A schematic explanation is provided in Fig. ijnn the Appendix. In order to do this, we rely on MINE (Belghazi et al. (0018) which uses statistics network, Ts, with parameters á to formulate a lower bound to MI, which is effective for high dimensional, continuous random variables. In our case, the statistics network takes two inputs: a global and local feature vector either sampled from the joint, where each comes from the same image, or from the product of marginals, where the global and local features are sampled independently from each other. The statistics network optimizes lower bound to the MI:
(X)o(X);Lo > xx
(I)
Where p(X)==(XX) is the data distribution, and and G random variables corresponding to the local and global feature vectors of the encoder. At optimum, the output of the statistics network, To
P(L]G) p(G.L) log plL)
provides an estimate for the Pointwise Mutual Information (PMI), defined log p[G)p(L]
which roughly gives us measure of how similar the global and local representations are in terms of information content. Normally, we could try to estimate the marginal term p( L) to get an estimate for the conditional density, but will normalize our score across the local patches of the target image to get a relative score of the relatedness of each local feature to a given global fcature. This analysis is similar to that done in Bachhan et al. (2019). which looks at different augmentations of the same image using the same MI estimator used to train the encoder.
4.4 TRE EVALUATION
We focus on a metric of compositionality introduced in Andreas (2019), the Tree Reconstruction Error (TRE), a proxy for compositionality. We will write TRE(X .a for the TRE computed on a dataset X over the set of primitives a. For details on its definition, see Section[ the Appendix.
As we mostly care about the compositionality with respect to attributes in the context of zero-shot learning and some representations are inherently more decomposable than others (such as VAEs due to the gaussian prior), we consider instead the ratio of the TRE computed with respect to attributes and the TRE computed with respect to uninformative variables (random assignment). We define the TRE(X.á)
TRE ratio as: TRE(X.a] where à is random binary matrix (random cluster assignment), and à are the actual visual primitives, attributes in our case.
5

---

Published as a conference paper at ICLR 2020
4.5 EXPERIMENTAL SETUP
Image encoders We considered both an encoder derived from the DCGAN architecture (Rad- ford et al., 2015) and similar in capacity to those used in early Few Shot Learning models such as MAML (Finn al. 2017). We also consider larger AlexNet (Krizhevsky al. 2012) based architec- ture to gain insight on the impact of the encoder backbone. It important to note that overall the encoders we use are significantly smaller than the "standard" backbones common in state-of-the- of-t---rrrrrr Imagenet- pretrained ZSL methods (note that similarly most recent ZSL methods, the encoder fixed after pre-training, and used as a feature extractor). We believe restricting the encoder's capacity decreases the performance, but does not hinder our ability to extract understanding of what methods work from our experiments. detailed description of the architectures can be found in the Appendix (Tables and
Evaluation Protocol We used the ZSL splits constructed in Xian et al. (2017), as they are the most commonly used in the literature. All models evaluated Top-1 accuracy. We pretrain the encoder using each the previously mentioned methods (strictly on the considered dataset, as per the ZSF requirement). We then train Prototypical Network on top of the (fixed) learned representation. All the implementation details are available in the Appendix, in Section
Model
AAE
AMDDM
CHIHM
Figure 2: Parts FI score for all models CUB with DCGAN-based encoder plotted against ZSL accuracy. There is a clear relationship between the two: encoders that have a good understanding of local information (as mea- sured by the parts F1 score) perform better in zero-shot learning. The addition of a local loss increases parts Fl score for all models. This improves generalization for all models except those trained with reconstruction objective.
CHDIM (n-1)
DIM
WAE
beta-VAE
Audo
5 RESULTS AND DISCUSSION
In this section, we describe in more detail our experiments and analyize the results. The full numerical results and plots for all considered models can be found in the Appendix.
5.1 DOES LOCALITY HELP ZSL, AND CAN LOCAL REPRESENTATION BE LEARNED?
Representations that predict parts at the correct location tend to perform better at ZSL. We hy- pothesize that the encoder represents information that is descriptive parts, it should also be able to generalize better. To test this, we compare ZSL performance to the part classification F1 score described 4.2 Fig. 2 the average Fl scores across the 15 classifiers is plotted against the ZSL accuracy for each model. The two measures are clearly correlated (Person' 's correlation doesn't it hold for reconstruction-bet -basd methods such as VAEs, which could be due to these models need- ing to represent information related all pixels, including the background, in order to reconstruct well.
0. 73). This relationship
Encouraging local representations to contain similar information improves ZSL perfor- mance For variants of Deep InfoMax (DIM, Hjelm et al. (018)), AMDIM and CMDIM, the local representations are encouraged be similar global representation through the mutual information maximization objective. While locally specified, this somewhat contrary to our definition of locality in 2.2 Among the models we tested, these variants generally perform very well, with CMDIM performing the best overall.
This indicates that, while important, locality by itself is not sufficient to learn good 7SLL representa- tions. The local representations must also share information, e.g., through a global representation the class. We hypothesize that such constraints help the encoder learn information that present locally, but relevant to discriminating the class or important high-level semantic meaning. The above observation also holds for the local losses (AC and LC introduced in 4.1). These losses both encourage the model to

---

Published as a conference paper at ICLR 2020
Model FC
Model AMDIM
Model CMDIM (pm1)
Model CMDIM (pm0.5)
Model CMDIM (pm0.1)
Reaatin Improvement
Local Loss Actrbbtt
CUB
AWAZ SUN
CUB
AWAA SUN
CUB
AWAZ
SUN
CUB
AWAZ SUN
CUB
AWAZ Dataset
SUN
Dataset
Dalaset
Dataset
Datas at
Figure 3: Relative improvement in terms of ZSL accuracy with respect models trained without the auxiliary loss. Attribute information results in a bigger improvement. Surprisingly. for certain models label information results in a decrease in generalization performance.
rely on local information (local features must capture important semantic information), and for these rep- resentations share information (by having high mutual information with either the attributes or labels).
Adding local losses helps supervised and self-supervised models. We investigate in more detail the effect of encouraging the model to take into account different types of local in information. As can be seen in Fig. 2 the addition of local loss improves both ZSL and parts score for all models except the generative ones (VAE, AAE). Interestingly, for these models the parts score also increases, indicating more locality, but this does not translate to better 7SL performance.
To better investigate why local losses improve generalization for supervised and self-supervised mod- els, in Fig. we the relative improvement of each type of local classifier over the performance of the encoder only trained with its global loss. We can see how for the supervised model, the attribute based auxiliary loss has much bigger impact, which indicates that label information is already exploited by the model, while attributes actually provide more information. For AMDIM, both losses seem to have a consistent positive effect, possibly because the model is unsupervised and hence any label-related infor- mation is useful. For CMDIM, the ICC auxiliary loss actually ly hurts performance. This is likely due to the fact that both CMDIM and the LC loss focus on discriminating classes at the local level, and that the LC objective is inherently less effective than the CMDIM formulation for this task (in terms of downstream ZSL performance). As a result, forcing the model to account for both terms lowers downstream perfor- mance. DIM and AMDIM discriminate instances and not classes so adding class and attribute informa- tion in the form of the AC and Losses helps performance. CMDIM is already exposed to class informa- tion, so only gains from being exposed to the (more informative) attribute information in the form of AC.
5.2 How DO DIFFERENT MODELS ENCODE INFORMATION LOCALLY?
In Fig. we apply the PMI-based visualization technique introduced in Section 4.3 to pairs of images from the CUB dataset. In this case, we are examining the global representation extracted by each model for the top left image (the Pacific Loon) and comparing it with local features from images of various classes. By noticing which patches each model pays attention to (have higher mutual information), we can infer how information is coded locally. The main take-aways are the following:
Supervised models. The fully supervised model in the first row seems be able to focus on relevant semantic details, such as the tail of the Horned Grebe and the head of the Back Tern. Unsupervised models.
In the second and third row can how models based on reconstruction loss seem to fail at highlighting semantic information: for images with patterns and colours similar to the Pacific I Loon, such as the other Pacific Loon or the White breasted Kingfisher, PMI is high across all local features, while scores are very low for the Tree Swallow with the uniform green background. This could possibly indicate that these models focus more on pixel statistics necessary for reconstruction, and are unable to extract as much semantic information.
Self- Supervieee models. AMDIM manages recover some semantic structure, e.g. for the other Pacific Loon or for the Rusty Blackbird, but fails in some other cases. CMDIM on the other hand, especially for high matching probability P. produces heatmaps that are very similar to those of the supervised models, hence managing to recover what discriminative between classes.
Models that encode semantic information well perform well on ZSL generalization. Models that reconstruct pixels well perform worse. To confirm intuitions on how some families of models focus more on semantic information, while others are more sensitive to pixel statistics, we rely again on the parts binary maps described in[ For each pair of images, we compute the ratio between the score assigned patches containing any part (i.e., logical OR computed across all binary maps) and
7

---

Published as a conference paper at ICLR 2020
Pacifii
Pacific
Horndd Grabe
Black
Blackbird
representation
Kingfisher
SImL 1.00
Aie slmo 0.00
SSIRG 8.56
AlE gimo 0.52
SSIN 0.68
Aee alm: 36
AUL aim D.
SSIN 0.50
Global
SSIN:
SSINL
Al
Tarta sooed 10:40
Parto acore: 30.05
Perto 10:22
Parl scane
11.60
Part score:
70.50
Perto acore:
20-23
Parto soone:
Fully Superoiedd
31.50
Variational
Parts score 54566
Putts SCOPEL
26
Pano scon:
22.21
Parts scone
34.92
Part soom
$1.46
Parts score
6:72
Parts SCOR
Adversarial Auroencoder"
Augmented
Muttscale
13:30
Patta McArE:
Perto 13:77
Para
27.34
Part 21.27
Matching DMM 0p
Parto
Parts scoe:
15.52
Parts scane
scone
Parts score: scae
Class
Matching (p-
Figure 4: Mutual Information heatmaps allow to understand which local patches contributed the most the final representation. For each heatmap we plot the absolute values the rightmost plot and the superposition of the heatmap and the original image on the left, to increase interpretability. Yellow corresponds to higher scores.
the overall score of all features. We refer to this measure as the Parts ratio. This ratio is sensitive to both how highly the model scores the relevant parts and to whether the rest of the features assigned a lower score. We then compute two different types of similarity between the considered images: semantic one, defined as the cosine distance between the attributes associated to the images' classes, and an pixel- wise one, measuring the Structural Similarity index (SSIM) between the images. We then measure correlation between the Parts ratio and these measures of similarity. Our interpretation the following:
Positive correlation with attribute similarity: If two images are semantically similar, the part ratio should be higher the model manages to extract the common semantically relevant patches (and we know that they are the parts for CUB).
Negative correlation with SSIM score: If model is too sensitive to pixel similarity, the parts score will be higher for images that are very different, where the only thing in common (pixel-wisse is to depict a bird, while for very similar images the model will just assign a high score to all local features.
We find that for VAEs, the ratio and the attribute similarity are not correlated, but the ratio and the SSIM scores correlate negatively. The effect is reversed for Supervised and Self- Self-Supervieed models. This confirms our intuition that VAEs and reconstruction models are not well suited to learn representations that generalize in context. More details about this experiment and the correlation coefficients are reported in the Appendix, in Fig.
5.3 DO COMPOSITIONAL REPRESENTATION PERFORM BETTER FOR ZSL?
Intuitively, we expect compositionality to be advantage: if model has good understanding of how parts map representations, it can learn to combine known concepts to describe new classes. The experiments show:
Measures of implicit compositionality correlate strongly with ZSL performance. Fig. shows the relationship between the TRE ratio introduced 4.4. and ZSL accuracy. The Pearson correlation coefficients between the TRE Ratio and Z.S.. Accuracy are the following: -0.90 for CUB, -0.60 for AwA2 and -0.30 for SUN.
The relation is strongest when the attributes are strongly relevant to the image. For the AWA2 and CUB, datasets for which the attributes are semantically very meaningful, we observe that there is direct relationship between TRE ratio and 7SL performance. This relationship degrades for
8

---

Published as a conference paper at ICLR 2020
23L. Acarssyy
AMDIM
CMDIM (p-1)
VIE beta-VAE
1.0
122
TRE Pario
Ratio
Ratio
Figure 5: Relationship between TRE ratio and aSccurayyy each dataset (lower TRE ratio is better).
SUN, which the attributes are per-image, and averaged over classes, meaning that they are less likely to map to information actually present in given image.
We also consider the effect of combining local representations directly (instead of relying on the global output of the model). Given local representations, there are several ways to employ them to perform classification: option classify each patch separately and then average this predictions.
option is to create final representation by averaging the local ones, another
An explicitely compostional model based on local features helps ZSL. The results for this be
comparison shown in Fig. G Averaging representations of compositionality: the representation of the whole input is directly built as weighted sum the patch representations (that we can imagine being more similar across different data points) and where the weights uniform. For CUB, and to a lesser extent AwA2, where only few patches encode important information such as beaks, tails, the effect is quite pronounced. There is less a difference for SUN, where the object is usually the whole scene, meaning all patches are expected to contribute.
directly enforcing notion
famiy Superdted
hmll
famil
40
X30 Aocu3s0
50
Avarge Regreeertiions
20
Predicions
10
CUB
AWAA
AWAA
SUN
CUB
AWA2
SUN
Dutaset
Dataset
Dalaset
Figure 6: Comparison between averaging representations and averaging predictions. We how, for the more local model familics, this notion of compositionality is most useful for CUB, where the object of interest is likely only present in few patches.
CONCLUSION AND FUTURE WORK
Motivated by the need for more realistic evaluation settings for SerooShhot Learning methods, we pro- posed a new evaluation framework where training is strictly performed only on the benchmark data, with no pre-training on additional datasets. In the proposed setting, we hypothesize that locality and com- positionality are fundamental ingredients for successful zero-shot generalization. We perform a series of tests of the relationship between these two aspects and zero-shot performance of diverse set of rep- resentations. We find that models that encourage both these aspects, either explicitly (through penalty per instance) or implicitly by construction, tend to perform better at zero-shot learning. We also find that models that focus on reconstruction tasks fail at capturing the semantic information necessary for good generalization, calling into question their applicability general representation learning methods.

---

Published as a conference paper at ICLR 2020
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jun 2015. doi: 10.1 10111099cp:220157779911. URL
httt:://d...ol..or/11...1. htt.p/.........1.111////........ 111p:///R..1//// 2015. 7298911
Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. arXiv preprint
arXiv:1806. 10166, 2018.
Jacob Andreas. Measuring compositionality in representation learning. (eXivv/902.07/0 2019
arXiv preprint
Yuval Atzmon and Gal Chechik. Adaptive confidence smoothing for generalized zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognirion, pp. 11671-11680, 2019.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. Factors of transferability for a generic convnet representation. convnet machine intelligence, 38(9): 1790-1229- 2015.
IEEE transactions on pattern analysis and
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: 00900. 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Ben- gio, Aaron Courville, and R. Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Leurming Research, pp. 531-540, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. http::/prrccediing
URL
ml press....s.............m..m http//000/ll/aaeeeee iat.aaiiila....... html
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations
for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
Irving Biederman.
broognition-by-oompone Psychological review, 94,2):115, 1987.
a theory of human image understanding.
George Boole. An investigation of the laws of thought, which are founded mathematical theories oflogic and probabilities. New York, Dover, 1 edition, 1854.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-feature CoRR, Abs/1904.00760, 2019a. URL
models works surprisingly well on imagenet.
http::/arxiv: org/ abs///b////// 1904 00760
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-feature models works surprisingly well on imagenet. International Conference on Learning Representations, 2019b.
Soravit Changpinyo. Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. 2016 IEEE Conference on Computer Vision and Pattern Recogmition (CVPR), Jun 2016. doi: 10. 10.1000/0p220
URLhttpp://dx dei org/ 10. 1ww19/1111...... 2016.
Koby Crammer, Michael Kearns, and Jennifer Wortman. Learning from multiple sources. In
B. Schölkopf, J. C. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems 19, pp. 321-328. MIT Press, 2007. URL atttp:/ppapers http:/ppppss.........pp cc/ cc/paper/ /aper/
ng- rom-mu 1 iipleeleeeeeee plp-ppsseee wtw.::/lp..lsssu....ppp... .dd
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain -ppeaif- transfer learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4109-4118, 2018.
Carl Doersch and Andrew Zisserman. Multi-task self self-super visual learning. abs/1708. 07860, 2017. URLhttp: http:://wwwwwwwxppp//ooo//soo
CoRR,
Carl Doersch, Abhinav Gupta, and Alexei A. Efros.
leaming by con- text prediction. CoRR, abs/1 hhss/5055 505.05192. 2015. URI httt///tgg///aaiii..or////////// 051
10

---

Published as a conference paper at ICLR 2020
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Kunihiko Fukushima. Neocognitron: A seff-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193 1980.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 2016.
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and language. In Proceedings the IEEEEECCeeee Conference on Computer Vision and Pattern Recognition, pp. 5994 6002, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In /CLR, 2017a.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv; (1t7:007700338 2017b.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. 2018
Donald D Hoffman and Whitman A Richards. Parts of recognition. Cognition, 18(1-3)65-966 1984.
Gabriel Huang, Hugo Larochelle, and Simon Lacoste-Julien. Centooi for few-shot clustering and unsupervised few-shot classification. arXiv preprint arXiv:1902.0666 2019.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv: 1608.0000000000 08614,7700 2016.
Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, pp. 1564-1574, 2018.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. 1rXiv:1312.611 14, Dec 2013.
Thomas N. Kipf and Max Welling. Semissuperviseddaasssicaaaoo with graph convolutional networks. CoRR, abs/1609.02900 2016. BRLLtttp://arrav.oog/aaass//s/s
02907.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jul 2017. doi: 10.1 10.11099/vpr:22117.47.. URLhttp::/ddx doi orrgiiorg//o / 1109//.............
http://ww..ol...rr//0111111///./ 2017.473
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume
2. AAAI08,, pp. http://dl. acm. org/ www.//itttte.oo ctt//iitaaaatt on cfm?i d= 620163 63. 620172
AAAI Press, 2008, ISBN 978-1-57735-368-3. URL
Fei-Fei Li, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594 611, 2006.
Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S. Davis. An analysis of pre-training on object detection, 2019.
Linghui Li. Sheng Tang, Lixi Deng, Yongdong Zhang, and Qi Tian. Image caption with global-local attention. In Thirty-PPie First AAAI Conference on Artificial Intelligence, 2017.
Alireza Makhzani, Jonathon Shlens. Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. CoRR, abs/1511 11111111.08000 105644, 2015. Utt.tt.pp://arxw..oo//////
05644
11

---

Published as a conference paper at ICLR 2020
Richard Montague. Universal grammar. In Richmond H. Thomason (ed.), Formal Philosophy: Selected Papers of Richard Montague, number 222-2222222222222222 22-24. Yale University Press, New Haven, London, 1974.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generaliza- tion in deep learning. In Advances in Neural Information Processing Systems, pp. 5947-5956, 2017.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representatiionn by solving jigsaw puzzles. CoRR, abs/1603.09246, 2016. htt.tttpp/aaxxv..zpppppo
09246
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating and recognizing scene attributes. In Proceeding the 25th Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arwiv:1511..66.. 2015.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint I 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252,, 2015. doi: 10. 10.1000//1
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv: 7700. 077.05555...51.010 2017.
Austin Stone, Huayan Wang, Michael Stark, Li D Scott Phoenix, and Dileep George. Teaching compositionality to cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 505550088000 0088-5067.200 2017.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843-852, 2017.
Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. Multi-attention multi-class constraint for fine-gramd grained image recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 805-821, 2018
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1109/cvpr.2018800111 URL
http:://x.com www.oo.gg//ooo.... 1t0///0.00....00//pp.......... 2015 2018
00131
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Ran Tian, Naoaki Okazaki, and Kentaro Inui. Learning semantically and additively compositional distributional representations. arXiv preprint 4rXin:1600.02266),22 2016.
Pavel Tokmakov, Yu- Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. arXiv preprint arXiv: (et::::2220000 2018.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv:1809.103:: 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Xinsheng Wang, Shanmin Pang. Jihua Zhu, Zhongyu Li. Zhiqiang Tian, and Yaochen Li. Visual space optimization for zero-shot learning. arXiv preprint 1907 2019.
12

---

Published as a conference paper at ICLR 2020
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning A comprehensive evaluation of the good, the bad and the ugly. CoRR, abs/ 006000 2017. URL
http://arxiv. org.bbs//r/////o/o/o/o/oo/.. abs 707 .06000
Yongqin Xian, Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions pattern analysis and machine intelligence, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
Adams Wei Yu, David Dohan, Minh- Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv::804. 804.09541, 2018.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 11..5:161 2016.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 1805.000.833 08 18, 2018a.
Li Zhang. Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1 10.100//vrr.2111..30 URLhttp://dae doi orq 10.z//1... 10. 11C9/CVPR 2017. 321
Yabin Zhang, Hui Tang, and Kui Jia. Fine-grained visual categorization using meta-learning optimization with sample selection of auxiliary data. Proceedings of the European Conference on Computer Vision (ECCV), pp. 233-248, 2018b.
13

---

Published as a conference paper at ICLR 2020
A EXTRACTING LOCAL AND GLOBAL FEATURES FROM A CNN ENCODER
A.1 EXPLAINING LOCAL AND GLOBAL FEATURES
kggg
Figure 7: The convolutional encoder takes in-
put an image and outputs a global representation
modde
used to compute the model loss Cmodel. To encourage locality and compositionality, label or attribute based classification is performed on
the activations from early layers(Ciocal).
Ciocol
A.2 EXPLAINING THE ROLE OF LOCAL AND GLOBAL FEATURES FOR MI COMPUTATION
Global feature
Figure 8: Local and global features are extracted from different images (where local features activations from early layer in the CNN encoder), then scored against each other. A high score means the two are considered likely to be extracted from the same image by the model, giving us insight in what information encoded by the global representation. A heatmap of the scores is used make the result interpretable.
Pointwise
Heatmap
Local teatures
B IMPLEMENTATION DETAILS
All models used in this paper have been implemented in Py Torch, and code will be made publically available. All images were resized to size 128 128, random crops with aspect ratio 0.875 were used during training, and center crops with the same ratio were used during test. While most ZSL approaches do not use crops (due to the fact that they used pre-computed features), this experimental setup shown to be efficient in the field of text image synthesis (Reed et al. 2016). All models are optimized with Adam with learning rate of 0.0001 with batch size of 64. The final output of the encoder was chosen to be 1024 across all models. Local experiments were performed extracting features from the third layer of the network. These features have dimension 27 27 x 384 for the AlexNet based encoder and 14 x 14 x 256 for the DCGAN encoder.
C MI HEATMAPS COMPARING CMDIM WITH DIFFERENT LOCAL LOSSES
In Fig. 9 we show how local losses affect the type of information CMDIM extracts. We can how some cases, e.g. the Nashville Warbler and Rusty Blackbird, the label-based local loss (LC) results in the encoder focusing more on the background and missing out on discriminative features. On the other hand, the fabel-based (LC) local loss helps the model focus more localised distinctive patches, especially for the Rusty Blackbird and the Rufous Hummingbird.
D PARTS RATIO AND RELATIONSHIP TO DIFFERENT MEASURES OF IMAGE SIMILARITY
In Fig. I0 plot the Parts ratio against the two different measure image similarity considered in our experiments and we report the correlation between them across the considered families of models. The correlation computed over 20.000 pairs of images for each family. While not being strong correlation, our experiments show how it's statistically significant one, with associated p-value (expressing the probability of not-correlated sample resulting in the reported correlation coefficients) of less than 10-6
14

---

Published as a conference paper at ICLR 2020
Local features
Plah Croe
Fal Crow
Jn Crow
Margrove Costoo
Maaha Warbe
Ho Shoo Sparrow
Tualy Blackbedd
Rafoca Harsmingars
Cloba
22.33
Mallaey DIM nome
Parts
Parts 19.20
24-03
Parta 23.76
Clocs
Parts
Parts scems 22.14
Parts scone 14.29
Parts score:
Varriing
Figure 9: Visualization of locality comparing encoders trained with CMDIM's loss and the proposed local losses. The Figure highlights the impact of local losses on the content extracted by the encoders.
Altbus
Unspenvied
Scperteed
Seh-Supemieed
4.3 404 080
(b)
Figure 10: Relationship between the parts and different measures of similarity. On the left, the parts scores is plotted against the two different measures of similarity. We can see there is a clear trend for all the models: the parts score increases for more semantically similar images, and decreases the images become more similar pixel-wise. The figure on the right shows Pearson's 's correlation coefficient between the metrics for different models
Model family Su Survivieed
Model family = Unsupervised
Model family = Self-Supariiad
50
Accursoy
Recuplive fuld
Blg Small
CUB
AWAR
SUN
CUB
AWA2
SUN
CUB
AINA2
SUN
Figure 11: Comparing pre and post-pooling (respectively Small and Big) features in terms of ZSL accuracy. The effect of varying receptive field size strongly depends the model type and on the dataset, highlighting how locality is expressed differently in the datasets.
E How LOCAL DO THE REPRESENTATIONS NEED TO BE?
somewhat alternative way to test local information is relevant for generalization is to explicitly only consider local information. To achieve this, consider performing classification directly features whose receptive field doesn't 't cover the entire image. our experiments we consider the features extracted from the AlexNet based encoder right before the flattening and final linear layers. To the effect of varying the receptive field, perform the same experiment and pre and post last
15

---

Published as a conference paper at ICLR 2020
pooling layer, going from a receptive field of 65 pixels (referred to small in the plots) to 85 pixels (big), out of 128 in the original input. The way these local prediction are combined is described in the following section. The results are summarized in Fig. [T] We can see how the dataset seems to make quite difference for class-matchin the object is in a small part input, while for SUN, where the whole image tends to be relevant as the images depict scenes, pooling either does not affect or has a negative effect on performance. For reconstruction based models on the other side we see a different trend, where not performing pooling consistently results in better performance across all datasets.
DIM that benefits from pooling for the datasets where usually
F
BIRD PARTS LOCATION MAPS
The parts annotations provided with the CUB dataset give us the ability to explicitly quantify whether the encoder is learning to extract meaningful local information. To evaluate this, we train a classifier for each part, that takes input local features extracted from a specific layer of the CNN encoder and outputs the probability of that part being present within the receptive field of the local feature. To construct ground truth for this evaluation, we pre-process the parts clicks annotations as follows: the datasets provides, for each input and part, a list of multiple parts location perceived by multiple MTurk workers. Each anno- tation is provided as (x.y) coordinates of the center of the part and a boolean feature visible indicating whether the part is hidden in the considered input. The ground truth for the classifiers is obtained by con- verting each part annotation into a boolean semantic map, where a truth value assigned to square of side 10 pixels centered in all the locations provided by different MTurk users for each part when visible. This process is repeated separately for all the 15 parts. The obtained boolean masks are then processed through CNN project them to the size compatible with the extracted features, so that the classifier's loss can be computed. Importantly, this loss is never backpropagated through the encoder, as these classifiers are meant to only evaluate whether the considered local features are predictive of the parts.
G CLASS MATCHING DIM
Deep InfoMax (Hjelm et al. 2018) is a self-supervised representation learning algorithm that is trained by maximizing Mutual Information (MI) between local and global features extracted from the network. More specifically, DIM's objective is lower bound to MI based on the Donsker-Varadhan representation of the KL divergence that computes two expectations: one over the joint distribution of local and global features, and one over the product of the marginals. In the original DIM setting, samples from the joint distribution (positive samples) are defined as local-global pairs extracted from the same input, while for the product of marginals (negative samples) local and global features are extracted from different inputs.
Class Matching DIM performs a similar operation, but samples from the joint distribution are defined to be pairs of local-global features extracted from different iuputs belonging to the same class, and negative samples are pairs where features are extracted from inputs of different classes. Moreover, we add a hyper-ppare parameter p that allows to control the interplay between DIM and CMDIM, so that positive samples are extracted from inputs of the same class with probability p and from the same input otherwise. Intuitively, this would push the encoder to extract features relevant a specific input while identifying what features are shared across a single class.
H DEFINITION OF TRE
We introduce the following notations:
X' is dataset, split into train Xr and test X sets.
For e X belonging to a class with binary attributes a; (for continuous attributes we threshold them beforehand), we define D(x) ) to be the set of 1-valued attributes (present attributes).
Each attribute a; is assigned learnable vector representation fr(a,)= =n))
5(-r) is distance function, chosen to cosine similarity as in Andreas (2019).
As in Andreas (2019) we combine individual attribute representations by summation:
fn)()
Σ
(a.)
a. e.E
16

---

Published as a conference paper at ICLR 2020
We can now define:
TRE( = 8 ((
TRE(A",a;))
TRE(
#EX
We compute 7 -= 7
and omit it in what follows by abuse of notations.
I
ARCHITECTURES AND DATASETS DETAILS
Table 1: Details of the datasets used
Dataset
#Images
#Attributes
#Classes
#Train classes
#Test classes
CUB
11,788
312
200
150
50
AWA
30,475
85
50
40
10
SUN
14,340
102
717
645
72
Table 2: Basic 128x128 architecture details.
Layer
Layer type
Layer params
pooling
activation
соnv
(64, 2, 1), batch norm
ReLU
conv
(128, 4, 2, 1), batch norm
ReLU
conv
(256, 4, 2, 1), batch norm
ReLU
(512, 2, 1), batch norm
ReLU
conv
(1024, 4, 2, 1), batch norm
ReLU
flatten
linear
(1024), batch norm
ReLU
Table 3: AlexNet 128x128 architecture details.
Layer
Layer type
Layer params
pooling
activation
conv
(96, I, 1), batch norm
(MaxPool2d, (MaxPool2d, 3,
ReLU
conv
(192, 1, 1), batch nomn
ReLU
(384, 1), batch norm
ReLU
conv
(384, 3, 1, 1), batch norm (192, 3, 1), batch norm
ReLU
conv
(MaxPool2d, 2)
ReLU
conv
(192, 3, 1), batch norm
(MaxPool2d, 3,
ReLU
flatten
linear
(4096.), batch norm
ReLU
linear
(4096,), batch norm
ReLU
TRE RATIO VALUES
Table 4: TRE ratio values for the CUB dataset
Normal
AC
LC
Normal_a
ACaa
LC_a
Model
Basic
Alex
classifier full
0.761
0.737
0.807
0.758
0.787
0.758
vae vac beta
1.062
1.042
1,066
1.111
1.079
1.11
1.04
1.044
1,044
1.12
1.087
1.1
aae
1.011
0.989
1.036
1.003
0.974
0.999
local dim
0.771
0.875
0.798
0.755
0.783
0.725
local. _wopass
0.881
0.913
0.85
0.762
0.804
0.739
twopass.wssssccoomooom class matching
0.639
0.633
0.752
0.676
0.649
0.624
17

---

Published as a conference paper at ICLR 2020
Table 5: TRE ratio values for the AWA2 dataset
Normal
AC
LC
Normal
AC a
I.C.a
Model
Basic
Alex
classifier_ful
0.88
0.873
0.913
0.882
0.878
0.909
vae
1.395
1.292
1.379
1.566
1.499
1.681
vae_beta aae
1.325
1.33
1.372
1.631
1.485
1.567
0.886
0.941
0,986
1.282
0.945
0.913
local_dim
1.211
1.206
1.174
1.154
1.264
1.062
twopass
1.103
1.072
1.093
1.151
0.998
1.105
local. twopass class matching
0.996
1.155
0.887
1.054
1.118
1.07
Table TRE ratio values for the SUN dataset
Normal
AC
LC
Normal
AC a
LC
Model
Basic
Alex
classifier_full
0.85
0.803
0.82
0.839
0.794
0.848
1.005
1,005
1,002
1.04
1.036
1.035
vae_beta
0.989
0.995
0.989
1.038
1.031
1.031
aae
0.963
0.976
0.975
0.954
0.926
0.958
local.dim
1.132
0.977
1.043
1.156
0.995
1.036
local. awooass
1.047
0.912
0.969
1.123
1.042
1.057
0.832
0.755
0.877
0.859
0.82
0.956
K Fl PART AVERAGE SCORES
Table 7: Part average F1 score, CUB dataset, basic encoder.
Loss
Normal
AC
LC
Model
FC
0.198
0.368
0.284
VAE
0.07
0.334
0.265
beta- VAE
0.067
0.353
0.255
AAE
0.086
0.085
0.024
DIM
0.235
0.393
0.304
AMDIM
0.311
0.406
0.319
CMDIM (p=1)
0.313
0.406
0.314
CMDIM (p=0.5)
0.295
0.397
0.321
CMDIM (p=0.1)
0.315
0.382
0.312
PN
0.288
18

---

Published as a conference paper at ICLR 2020
Table 8: ZSL. accuracy, comparing the different local losses.
Encoder
alex128x128
basic 28x
Loss
Normal
AC
LC
Normal
AC
LC
Model
Dataset
FC
30.47
34.92
32.40
27.44
32.17
30.44
VAE
12.08
13.41
12.51
12.13
13.46
I0.27
beta-VAB
11.75
11.77
12.33
12.03
12.85
12.52
AAE
15.16
15.00
13.49
9.12
12.36
9.80
DIM
23.93
33.35
31.84
24.42
32.54
29.17
CUB
AMDIM
24.34
29.05
31.12
24.42
30.29
28.83
CMDIM (p=l)
35.80
40.11
32.31
29.24
30.08
30.04
CMDIM (p=0.5)
35.12
37.02
35.27
29.67
35.15
31.06
CMDIM (p=0.T)
29.83
33.60
33.02
27.03
32.35
31.14
PN
37.59
26.29
FC
46.48
52.81
46.04
45.94
45.98
46.09
VAE
29.17
28.54
28.76
30.02
29.60
29.48
beta- VAE
29.98
30.11
29.22
29.00
29.47
29.94
AAE
32.07
29,46
30.93
31.94
29.31
31.85
DIM
38.73
45.54
43.89
39.63
44.23
44.32
AWA2
AMDIM
42.84
45.41
46.95
42.04
49.01
43.77
CMDIM (p=1)
45.80
46.56
42.14
46.87
45.00
39.70
CMDIM 0.5
46.87
48.06
48.45
46.87
47.92
45.63
CMDIM (p=0.1)
47.29
51.51
50.17
45.71
49.51
48.01
PN
46.53
45.23
FC
33.02
36.89
37.57
32.20
38.79
32.74
VAE
14.61
15.08
14.33
15.14
16.58
15.22
beta- VAE
13.80
14.20
13.79
15.08
15.29
16.58
AAE
17.93
16.78
17.86
18.55
18.41
18.13
DIM
31.73
39.06
37.64
33.69
41.44
38.52
SUN
AMDIM
38.04
41.44
39.67
37.64
42.26
38.19
CMDIM (p=l)
35.73
37.43
32.81
34.44
37.98
31.18
CMDIM (p=0.5)
37.43
40.15
36.62
35.39
39.74
34.10 10
CMDIM (p=0.1)
40.01
42.05
38.51
40.56
43.13
38.93
PN
32.00
29.82
19

---

Published as a conference paper at ICLR 2020
Table 9: ZSL. aceuracy, comparing the different local models.
Average before
Average after
Model
Dataset
pool
no pool
pool
no pool
FC
28.55
21.45
13.02
11.75
VAE
8.37
8.06
8.33
8.15
beta- VAE
8.04
8.77
7.91
8.02
AAE
8.20
7.37
7.28
6.84
DIM
CUB
18.48
14.39
15.07
11.81
AMDIM
21.45
17.68
16.67
14.04
CMDIM (p=1)
34.11
22.16
13.51
15.17
CMDIM (p=0.5)
33.48
22.88
19.07
16.35
CMDIM (p=0.1)
26.17
19.23
18.38
15.89
FC
48.57
46.95
38.90
41.98
VAE
26.36
30.04
25.47
26.37
beta- VAE
24.78
26.32
27.63
27.11
AAE
26.74
23.59
26.08
25.23
DIM
AWA2
35.37
34.77
33.54
33.99
AMDIM
41.37
41.34
40.57
37.12
CMDIM (p=1)
49.66
46.26
48.16
42.57
CMDIM (p=0.5)
49.17
45.99
45.90
43.28
CMDIM 01=0.1)
50.95
44.26
44.90
37.79
FC
33.97
34.31
32.13
32.15
VAE
11.48
12.84
10.73
10.14
beta- VAE
11.96
14.06
11.89
13.06
AAE
9.38
11.07
9.31
11.32
DIM
SUN
25.82
26.83
24.80
25.56
AMDIM
34.04
35.19
34.51
34.58
CMDIM (p=l)
37.02
36.75
33.70
36.53
CMDIM (p=0.5)
37.98
38.93
35.53
37.29
CMDIM (p=0.1)
35.73
35.60
34.58
36.39
L FULL PLOTS
20

---

Published as a conference paper at ICLR 2020
APE
54
70%
DRMM. CE
Figure 12: Comparison between averaging representations VS averaging scores for all models.
0UR
5A
CUB
68
Figure 13: Comparison between small and big receptive field for all models.
21


Reviews:
[{"review_id": "rkx4AoO0Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary \n- The paper tries to understand what learning principles can lead to representations that allow zero-shot learning/generalization. \n- It explores the impact of two properties -- locality and compositionality -- on ZSL performance. \n- To encourage locality (and also compositionality since they can overlap), the paper uses an auxiliary loss at an earlier layer (when the receptive field of features is small) to predict local attributes. \n- For interpreting local features, the paper computes MI between the global feature map of one image and local features from a different image of the same class and visualizes it using a heatmap.  \n- It uses normalized TRE to measure compositionality (Normalized to take into account that some methods are biased towards learning more compositional representations.)\n- It compares supervised, unsupervised, and self-supervised representation learning methods and studies the impact of locality and compositionality on ZSL performance for these methods. \n\n### Decision and reasons\nI vote for a weak accept. However, I think it's a good paper and does a lot of things right. I'm willing to increase my score if the authors can convincingly answer my questions.  \n\nPositives: \n1- The proposed zero-shot learning from scratch setting is a step in the right direction for focusing on uncovering general learning principles. \n\n2- Locality and compositionality are sensible goals for good representations. The paper defines both and explores their importance with clear and novel experiment-designs and metrics. The experiments are also well conducted.\n\nNegatives: \n\n1- The paper makes a lot of observations, but sometimes does not even try to explain some unexpected observations. \n2- CMDIM is presented as a self-supervised algorithm but seems to require class labels. \n\n### Supporting arguments for the reasons for the decision.\n\nPositives: \n\n1- I agree with the paper that ZSL setting is more about uncovering learning principles and less about constructing practical systems that can do well in zero-shot settings. In a practical setting, it doesn't make much sense to zero-shot learning anyway. I also agree that current ZSL work is focusing too much on pushing the state-of-the-art using pre-trained imagenet features and missing the actual goal of the problem setting (I wouldn't say we shouldn't do that at all, however. Some learning principles may require a significant amount of data to learn good representations for zero-shot learning, and imagenet pretraining is a good proxy for that). This paper acts as a good reminder that ZSL research should be done keeping in mind the goal of ZSL.\n\n2- The paper first defines locality and compositionality. It then uses interesting and meaningful metrics for empirically testing if these properties correlate with zero-shot performance. I found the experiment designs to be clever (such as computing parts F-1 score using boolean maps, visualizing MI between local and global features of images from the same class, etc).  I think this paper will act as an important reference for motivating future work on learning more modular representations. \n\nNegatives: \n\n1- The paper doesn't try to explain the possible reasons behind some observations. For example, why does locality not correlate with ZSL performance of generative models? Why does LC loss hurt performance for CMDIM? \n2- CMDIM draws positive samples from other images of the same class. As a result, it's not a truly unsupervised learning method. The direct comparison of CMDIM to DIM seems unfair. \n\n### Questions\n\nQ1-  Computation of the local classification and attribute auxiliary loss is a bit unclear. How is it assured that a certain attribute is present in the local feature when computing the auxiliary attribute-loss? \n\nQ2- Did the authors try a baseline in which AC is also trained on the global representation in the context of Figure 2? The extra information about the parts may be the reason behind better ZSL performance, and a global AC could improve ZSL performance without improving Parts F1 score (Although a more likely outcome is that a global AC would increase both the parts F1 score and the ZSL accuracy.)\n\nQ3- For VAEs, the local F-1 score does not correlate with ZSL performance. This seems to contradict the idea locality leads to better ZSL performance. Is there an explanation for this? The paper just glances over this by calling this 'interesting.' \n\nQ4- Can CMDIM be considered a self-supervised algorithm? Doesn't it need class labels to draw positive samples from the same class? \n\n\n### Other minor remarks \n\n- \"Formally, f(x) \\elem R is compositional if it can be expressed as a combination of the elements of ...\"\n\nThe definition would be more clear if the authors could mention some reasonable combination operators here (such as weighted average etc).\n\n- \"However, this choice in architecture does not guarantee locality, as CNN representations could only contain “global” information, such as the class or color of the object, despite having a limited receptive field. \"\n\nI'm not sure what this means. Why are CNN representations restricted to 'only' global information?\n\n### Update after author's response\n\nThe author's response fixes some minor clarity issues. I think this work is a good contribution and should be accepted. I've updated my score accordingly. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJeARAZTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: My recommendation has been borderline because the discussion of the paper about the nature of locality and compositionality seems to be less in-depth than I would have expected, but if the authors will revise the submission to shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, I would lean towards an accept as the empirical evaluation is extensive.\n\n----\nSummary: This paper aims to investigate the role of locality and compositionality in zero-shot learning. The authors propose a novel evaluation setup that differs from the original zero-shot learning framework in that the model is not allowed to be pretrained on another dataset. The authors propose several methods for learning and visualizing local representations.\n\nThe overall problem the paper tackles is quite ambitious which involve many parts: defining locality and compositionality, defining constraints for enforcing locality and compositionality, evaluating the existence of locality and compositionality, and evaluating model performance. Because of the extensive nature of the research problem, the coverage of each piece of this research problem in this paper could have been more thorough: for example, I would be really excited for a more in-depth analysis on methods for enforcing locality, but the authors consider only one method, which is the auxiliary loss, and this method does not seem to be applicable for datasets that do not have attribute labels. Given the formalism in section 2.1, I would furthermore be excited about a more in-depth analysis on methods for enforcing compositionality, but the authors only investigate compositionality from the perspective of a weighted average. As a result, the claims about whether enforcing locality and compositionality helps with generalization seem weak, as the authors only do not seem to consider different ways of achieving locality and compositionality -- it is not clear whether their method for enforcing locality (with auxiliary losses) or compositionality (by explicitly performing a weighted average) is representative of enforcing locality and compositionality in general. Furthermore, locality seems to be more of a statement about *ignoring* information (discussed below), which the authors do not explore. As a result, my recommendation is borderline. Perhaps the paper could benefit by reframing it with a more specific focus, with more emphasis on the evaluation, which the paper seems to do well. I would recommend that the authors either (1) perform a more extensive analysis of methods for enforcing locality and compositionality or (2) shift the focus of the paper to more focused on analysis and evaluation and weaken the claims that their model exhibits locality and compositionality, as there is not enough evidence to tell whether their method for enforcing locality and compositionality is the best way of doing so.\n\n\nStrengths:\n- The paper is well-motivated and tackles an important problem.\n- The empirical evaluation is extensive.\n- The framework of not relying on pretrained features is a novel contribution.\n\nWeaknesses:\n- There does not seem to be a comparison between a compositional model and a non-compositional model in section 5.3. Would the authors be able to provide such an analysis? Otherwise it is difficult to tell exactly how and to what extent compositionality helps.\n- I would have assumed that enforcing locality should actually be a problem of *ignoring* information, rather than *predicting* labeled information. However, the auxiliary loss introduction in section 4.1 only seems to enforce that local features (which by default are local from the CNN) are predictive of attributes/class, rather than enforcing that these features ignore information that is not local to the particular image patch they are modeling. Would the authors be able to comment on this point, as well as provide an empirical analysis of how *unpredictive* the features of their model are on image patches elsewhere?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgLUdTnYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an evaluation framework for Zero-Shot Learning (ZSL) methods called zero-shot learning from scratch (ZFS) where the model is not allowed to be pretrained on other datasets such as ImageNet. The main motivation of this approach is that it is difficult to understand what is useful for generalization in ZSL since most state-of-the-art methods use features pretrained on large datasets such as ImageNet.\nMost state-of-the-art approaches exploit models pretrained on ImageNet. Instead, the paper proposes to randomly initialize model parameters to have a better understanding of what's happening in ZSL.\nZFS adds one constraint: model parameters should not contain information about data outside that from the training split of the target dataset.\nTwo main criteria are studied to study neural networks:\n- compositionality (ability to be expressed as a combination of simpler parts)\n- locality (ability to encode only information specific to locations of interest)\nTo provide a better understanding of their claims, the authors use MTurk annotations to construct boolean map for each local part labelled in the CUB dataset.\n\n\nAlthough the paper is experimental, I vote for borderline accept for the following reasons:\n- The paper is well-written and the contributions are clear.\n- The evaluation metrics to evaluate how models generalize for both criteria are well motivated theoretically (e.g. Tree Reconstruction Error is used to evaluate compositionality), and different types of encoders (e.g. variations of DCGANS) are studied.\n- The analysis of models pretrained in different contexts (e.g. supervised classification, reconstruction etc...), with only global or local information, and their impact on generalization is clear (see conclusion for a summary of results). Some conclusions are very intuitive but useful to know (e.g. VAEs and reconstruction models are not well suited to learn representations that generalize in ZFS).\n\n\nThe weaknesses I found in the paper are the following:\n- The paper claims that the models used are smaller than the “standard” backbones common in state-of-the-art Imagenet-pretrained ZSL methods. However, few-shot learning method (e.g. ProtoNet) do not retrain the whole model: Protonet freezes most layers and fine-tunes only the last layers to avoid overfitting.\n- The paper introduces \"Class-Matching Deep Informax (CMDIM)\" which draws positive samples from other images from the same class to extract information that discriminative between categories instead of individual samples. However, I did not understand its exact formulation and how it is exactly different from other DIM approaches.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the role of locality (ability to encode only information specific to locations of interest) and compositionality (ability to be expressed as a combination of simpler parts) in Zero-Shot Learning (ZSL). Main contributions of the paper are (i) compared to previous ZSL frameworks, the proposed approach is that the model is not allowed to be pretrained on another dataset (ii) a thorough evaluation of existing methods.

Following discussions, weaknesses are (i) the proposed method (CMDIM) isn't sufficiently different or interesting compared to existing methods (ii) the paper does not do an in-depth discussion of locality and compositionality. The empirical evaluation being extensive, the accept decision is chosen.

Decision: Accept (Poster)