Class Normalization for (Continual)? Generalized Zero-Shot Learning
Authors: Ivan Skorokhodov
Mohamed Elhoseiny
Venue: ICLR 2021 Poster (2021)
Keywords: zero-shot learning
normalization
continual learning
initialization

Abstract: Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime. However, in the zero-shot learning (ZSL) world, these ideas have received only marginal attention. This work studies normalization in ZSL scenario from both theoretical and practical perspectives. First, we give a theoretical explanation to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass. Next, we demonstrate that they are insufficient to normalize a deep ZSL model and propose Class Normalization (CN): a normalization scheme, which alleviates this issue both provably and in practice. Third, we show that ZSL models typically have more irregular loss surface compared to traditional classifiers and that the proposed method partially remedies this problem. Then, we test our approach on 4 standard ZSL datasets and outperform sophisticated modern SotA with a simple MLP optimized without any bells and whistles and having ~50 times faster training speed. Finally, we generalize ZSL to a broader problem — continual ZSL, and introduce some principled metrics and rigorous baselines for this new setup. The source code is available at https://github.com/universome/class-norm.

Published as a conference paper at ICLR 2021
CLASS NORMALIZATION FOR (CONTINUAL)? GENERALIZED ZERO-SHOT LEARNING
Ivan Skorokhodoy :2 Thuwal, Saudi Arabia lakorokhooov@gmmi com
Mohamed Elhoseiny' Thuwal, Saudi Arabia
mohamed. e hoseinykkaust.
King Abdullah University of Science and Technology (KAUST), Saudi Arabia Moscow Institute of Physics and Technology (MIPT), Russia
ABSTRACT
Normalization techniques have proved to be crucial ingredient of successful training in traditional supervised learning regime. However, in the zero-shot learning (ZSL) world, these ideas have received only marginal attention. This work studies normalization in ZSL scenario from both theoretical and practical perspectives. First, we give a theoretical explanation to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass. Next, we demonstrate that they are insufficient to normalize deep ZSL model and propose Class Normalization (CN): a normalization scheme, which alleviates this issue both provably and in practice. Third, show that ZSL models typically have more irregular loss surface compared to traditional classifiers and that the proposed method partially remedies this problem. Then, test our approach on 4 standard ZSL. datasets and outperform sophisticated modern SotA with a simple MLP optimized without any bells and whistles and having <550 times faster training speed. Finally, we generalize ZSL to a broader problem - continual ZSL, and introduce some principled metrics and rigorous baselines for this new setup. The source code is available at
INTRODUCTION
Zero-shot learning (ZSL) aims understand new concepts based on their semantic descriptions instead of numerous input-output learning pairs. It is a key element of human intelligence and our best machines still struggle to master it (Ferrari & Zisserman, 2008; Lampert et al., 2009; Xian et al., 2018a). Normalization techniques like batch/layerr/roop normalization (loffe & Szegedy, 2015; Ba et al., 2016; Wu He, 2018) are now common and important practice of modern deep learning. But despite their popularity in traditional supervised training, not much is explored in the realm of zero-shot learning, which motivated us to study and investigate normalization in ZSL models.
We start by analyzing two ubiquitous tricks employed by ZSL and representation learning practitioners: normalize+scale (NS) and attributes normalization (AN) (Bell et al., 2016; Zhang et al., 2019; Guo et al., 2020; Chaudhry et al., 2019). Their dramatic influence on performance can be observed from Table 1. When these two tricks are employed, a vanilla MLP model, described Sec 3.1, can outperform some recent sophisticated ZSL methods.
Normalize+scale (NS) changes logits computation from usual dot-produce to scaled cosine similarity:
g. TT= Pe - gc
?
Pe |P-ll
1
where z is an image feature, Pe is c-th class prototype and 7 is a hyperparameter, usually picked from [5, 10] interval et al., 2019; Zhang et al., 2019). Scaling by y is equivalent to setting high temperature of 72 in softmax. In Sec. 3.2, we theoretically justify the need for this trick and explain why the value of must be high.

---

Published as a conference paper at ICLR 2021
Table I:
Effectiveness of Normalize+Scale, Attributes Normalization and Class Normalization. When NS and AN are integrated into basic ZSL model, its performance is boosted up level of some sophisticated SotA methods and additionally using CN allows to outperform them. =NS and + AN denote if normalize+scale attributes normalization are heing used. Bold/normal blue font denote best/second-Mes results. Extended results are in Table 2, 5 and 8.
SUN
CUB
AwA1
Aw A2 S H
Avg training time
H
U
H
U
S
DCN Liu et al. (2018)
25.5 37.0
30.2
28.4
60.7
38.7
25.5
84.2
39.1
minutes
SGAL Yu Lee (2019)
42.9 31.2
36.1
47.1
44.7
45,9
52.7 75.7 62.2
55.1
81.2 65.6
minutes
LsrGAN Vyas et al. (2020)
44.8 37.7
40.9
48.1
59.1
53.0
54.6 74.6
63.0
1.25 hours
Vanilla MLP -NS -AN
4.7 27.2 8.0
5.9
26.0
9.7
43.1
81.3
37.7
84.3 52.1
Vanilla MI.P -NS +AN
9.6 34.0
14.9
8.8
4.6
6.0
28.6
84.4
42.7
23.3
87.4 36.8
30 seconds
Vanilla MLP +NS -AN
34.7 38.5
36.5
46.9
42.8
44.9
57.0
69.9 62.8
49.7
76.4 60.2
Vanilla MLP +NS+AN
31.4
40.4
35.3
45.2
50.7
47.8
58.1
70.3
63.6
58.2
73.0 64.8
Vanilla MLP +NS +AN +CN 44.77 41.6 43.1
49.9
50.7
50.3
63.1
73.4 67.8
60.2
77.1 67.6 30 seconds
Attributes Normalization (AN) technique simply divides class attributes by their L2 norms:
- Gc/ /lel|all2
(2)
While this may look inconsiderable, is surprising to see it being preferred in practice (Li et al., 2019; Narayan et al., 2020; Chaudhry et al., 2019) instead of the traditional pero-mean and anit-variance
data standardization (Glorot & Bengio, 2010). In Sec 3, we show that it helps in normalizing signal's variance in and ablate its importance in Table and Appx D.
These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear (see Figure 1), but they fail when
a multi-layer architecture. To remedy this issue, we introduce Class Normalization (CN): a novel normalization scheme, which is based on a different initialization and a class-wise standardization transform. Modern ZSL methods either utilize sophisticated architectural design like training generative models (Narayan et al., 2020; Felix et al., 2018) or use heavy optimization schemes like episode- based training (Yu et al., 2020; Li al., 2019). In contrast, we show that simply adding Class Normalization on top of a vanilla MLP is enough to set new sfateoof-the-ar results on several standard 7SSL datasets (see Table 2). Moreover, since is optimized with plain gradient descent without any bells and whistles, training time for us takes 50-100 times less and runs in about minute. We also demonstrate that many ZSL models tend to have more irregular loss surface compared to traditional supervised learning classifiers and apply the results of Santurkar et al. (2018) to show that our CN partially remedies the issue. We discuss and empirically validate this in Sec 3.5 and Appx F.
Apart from the theoretical exposition and a new normalization scheme, we also propose a broader ZSL setup: continual zero-shot learning (CZSL). Continual learning (CL) is an ability to acquire new knowledge without forgetting (e.g. (Kirkpatrick et al., 2017)), which is scarcely investigated in ZSL. We develop the ideas of lifelong learning with class attributes, originally proposed by Chaudhry et al. (2019) and extended by Wei et al. (2020a), propose several principled metrics for it and test several classical CL methods in this new setup.
RELATED WORK
Zero-shot learning. Zero-shot learning (ZSL) aims at understand example of unseen classes from their language or semantic descriptions. Earlier ZSL methods directly predict attribute confidence from images to facilitate zero-shot recognition (e.g.. Lampert et al. (2009); Farhadi et al. (2009); Lampert et al. (2013b)). Recent ZSL methods for image classification can be categorized into two groups: generative- based and embedding- based. The main goal for generative-base
approaches is to build a conditional generative model (e.g., GANs Goodfellow et al. (2014) and VAES (Kingma & Welling, 2014)) to synthesize visual generations conditioned on class descriptors (e.g., Xian et al. (2018b); Zhu et al. (2018); Elhoseiny & Elfeki (2019); Guo et al. (2017); Guo et al. (2017); Kumar Verma et al. (2018)). At test time, the trained generator is expected to produce synthetic/fake data of unseen classes given its semantic descriptor. The fake data is then used to train traditional classifier or to perform a simple ZNN-classification on the test images. Embedding- based approaches learn mapping that projects semantic attributes and images into common space where the distance
2

---

Published as a conference paper at ICLR 2021
between class projection and the corresponding images is minimized (e.g, Romera-Paredes & Torr (2015); Frome et al. (2013); Lei Ba et al. (2015); Akata et al. (2016a); Zhang et al. (2017); Akata et al. (2015; 2016b)). One question that arises is what space to choose to project the attributes or images to. Previous works projected images to the semantic space (Elhoseiny et al., 2013; Frome et al., 2013; Lampert et al., 2013a) some common space (Zhang & Saligrama, 2015; Akata et al., 2015), but our approach follows the idea of Zhang et al. (2016); Li et al. (2019) that shows that projecting attributes to the image space reduces the bias towards seen data.
Normalize+scale and attributes normalization. It was observed both in ZSL (e.g. Li et al. (2019); Zhang et al. (2019); Bell al. (2016)) and representation learning (e.g., Sohn (2016): Guo et al. (2020); Ye et al. (2020)) fields that normalize+scale (i.e. (1)) and attributes normalization (i.e. (2)) tend to significantly improve the performance of learning system. In the literature, these two techniques lack rigorous motivation and are usually introduced as practical heuristics that aid training (Changpinyo et al., 2017; Zhang et al., 2019; 2021). One of the earliest works that employ attributes normalization was done by (Norouzi et al., 2013), and in (Changpinyo et al., 20) authors also ablate its importance. The main consumers of normalize+scale trick had been similarity learning algorithms, which employ it to refine the distance metric between the representations (Bellet et al., 2013; Guo et al., 2020; Shi et al., 2020). Luo et al. (2018) proposed to use cosine similarity in normalization procedure, but didn't incorporate any analysis
the final output projection matrix on how it affects the variance. They also didn't 't use the scaling which our experiments in Table show to be crucial. Gidaris & Komodakis (2018) demonstrated a greatly superior performance of an NS-enriched model compared to dot-product based one in their setup where the classifying matrix constructed dynamically. Li et al. (2019) motivated their usage of NS by variance reduction, but didn't t elaborate on this in their subsequent analysis. Chen et al. (2020) related the use of the normalized temperature-scat scalled cross entropy loss (NT-Xent) to different weighting of negative examples in contrastive learning framework. Overall, to the best of our knowledge, there is no precise understanding of the influence of these two tricks on the optimization process and benefits they provide.
Initialization schemes. In the seminal work, Xavier's init Glorot & Bengio (2010), the authors showed how to preserve the variance during a forward pass. He et al. (2015) applied similar analysis but taking ReLU nonlinearities into account. There is also a growing interest in two-step Jia et al. (2014), data-dependent Krähenbühl et al. (2015), and orthogonal Hu et al. (2020) initialization schemes. However, the importance of a good initialization for multi-modal embedding functions like attribute embedding is less studied and not well understood. We propose a proper initialization scheme based on a different initialization variance and a dynamic standardization layer. Our variance analyzis similar in nature to Chang al. (2020) since attribute embedder may be seen as a hypernetwork (Ha et al., 2016) that outputs a linear classifier. But the exact embedding transformation is different from a hypernetwork since it has matrix-wise input and in our derivations we have to use more loose assumptions about attributes distribution (see Sec 3 and Appx H).
Normalization techniques. closely related branch of research is the development of normalization layers for deep neural networks (loffe & Szegedy, 2015) since they also influence signal's 's variance. BatchNorm, being the most popular one, normalizes the location and scale of activations. is applied in batch-wise fashion and that's why its performance is highly dependent on batch size (Singh & Krishnan, 2020). That's why several normalization techniques have been proposed to climinate the batch-size dependecy (Wu & He, 2018; Ba et al., 2016; Singh & Krishnan, 2020). The proposed class normalization is very similar to a standardization procedure which underlies BatchNorm, but is applied class-wis in the attribute embedder. This also makes it independent from the batch size.
Continual zero-shot learning. We introduce continual zero-shot learning: a new benchmark for ZSL agents that is inspired by continual learning literature (e.g., Kirkpatrick et al. (2017)). It is development of the scenario proposed in Chaudhry et al. (2019), but authors there focused on ZSL. performance only single task ahead, while in our case we consider the performance on all seen (previous tasks) and all unseen data (future tasks). This also contrasts our work to the very recent work by Wei et al. (2020b), where a sequence of seen class splits of existing ZSL benchmsks is trained and the zero-shot performance is reported for every task individually at test time. In contrast, for our setup. the label space is not restricted and covers the spectrum of all previous tasks (seen tasks so far), and future tasks (unseen tasks so far). Due to this difference, we need to introduce set of new metrics and benchmarks to measure this continual generalized ZSL skill over time. From the lifelong learning perspective, the idea to consider all the processed data to evaluate the model is not
3

---

Published as a conference paper at ICLR 2021
new and was previously explored by Elhoseiny et al. (2018); van de Ven & Tolias (2019). It lies in contrast with the common practice of providing task identity at test time, which limits the prediction space for model, making the problem easier (Kirkpatrick et al., 2017; Aljundi et al., 2017). In Iscle et al. (2016); Lopez-Paz. & Ranzato (2017) authors motivate the use of task descriptors for zero-shot knowledge transfer, but in our work consider class descriptors instead. We defined CZSL continual version of generalized-Ze -SLL which allows us to naturally extend all the existing ZSL metrics Xian al. (2018a); Chao et al. (2016) our new continual setup.
3 NORMALIZATION IN ZERO-SHOP LEARNING
The goal of a good normalization scheme to preserve a signal inside model from severe fluctuations and to keep it in the regions that are appropriate for subsequent transformations. For example, for ReLU activations, we aim that its input activations to be zero-centered and not scaled too much: otherwise, we risk to find ourselves in all-zero or all-linear activation regimes, disrupting the model performance. For logits, we aim them to have a close-to-unit variance since too small variance leads to poor gradients of the subsequent cross-entropy loss and too large variance is indicator of poor scaling of the preceding weight matrix. For linear layers, we aim their inputs to be zero-centered: in the opposite case, they would produce too biased outputs, which is undesirable.
In traditional supervised learning, we have different normalization and initialization techniques to control the signal flow. In zero-shot learning (ZSL), however, the set of tools is extremely limited. In this section, we justify the popularity of Normalize+Scale (NS) and Attributes Normalization (AN) techniques by demonstrating that they just retain a signal variance. We demonstrate that they are not enough to normalize a deep ZSL. model and propose class normalization to regulate a signal inside a deep ZSL model. We empirically evaluate our study in Sec. 5 and appendices A, B, D and
3.1 NOTATION
ZSL setup considers access to datasets of seen and unseen images with the corresponding labels N, and D" = {x" , 3/" T respectively, Each class i described by its class attribute vector a E Rd All attribute vectors are partitioned into non-overlapping seen and unseen sets as well: A = {a:};",
D = =xx]. y/ h fi=1
K. and A" = {a,) K.. Here NN, Nu, K,, K,, are number of seen images, unseen images, seen classes, and unseen classes respectively. In modern ZSL, all images transformed via some standard feature extractor E x + 2 E Rdd (Xian et al., 2018a). Then, a typical ZSL method trains altribute embedder Pg a. Pe E Rd: which projects class attributes a onto feature space Rd= in such a way that it lies closer to exemplar features x" of its class c.
usually
This is done by solving a classification task, where logits computed using formula (1). In such way at test time we are able to classify unseen images by projecting unseen attribute vectors a into the feature space and computing similarity with the provided features z". Attribute embedder Po is usually very simple neural network (Li et al., 2019); in many cases even linear Coomera-Parede & Torr, 2015; Elhoseiny et al., 2013), so it is the training procedure and different regularization schemes that carry the load. We will denote the final projection matrix and the body of Po V and H. respectively, i.e. Po(a.) = (H During training, it receives matrix of class attributes A = [a1, ..., ax,] of size Ke x d, and outputs matrix W = (A) of size Ke dz. Then W is used to compute class logits with a batch of image feature vectors 2 NN.
3.2 UNDERSTANDING NORMALIZE SCALE TRICK
One of the most popular tricks in ZSL and deep learning is using the scaled cosine similarity instead of a simple dot product in logits computation (Li et al., 2019; Zhang et al., 2019; Ye et al., 2020):
z Pc
ge x2 Pe ge = =, |
(3)
lpell
where hyperparameter y is usually picked from [5, 10] interval. Both using the cosine similarity and scaling it afterwards by large value is critical to obtain good performance; see Appendix D. To our knowledge, it has not been clear why exactly it has such big influence and why the value of y must be so large. The following statement provides answer to these questions.
4

---

Published as a conference paper at ICLR 2021
Statement 1 (informal). Normalize+scale trick forces the variance for De to be approximately:
dg
Var [e] ~ 72 (dg 2)22
(4)
where dy is the dimensionality of the feature space. See Appendix A for the assumptions, derivation and the empirical study. Formula (4) demonstrates two things:
1. When we use cosine similarity, the variance of ÿe becomes independent from the variance of W (A). leading to better stability.
2. If one uses Eq. (3) without scaling (i.e. 7 y= 1), then the Var [ño] will be extremely low (especially for large d2) and our model will always output uniform distribution and the training would stale. That's why need very large values for 7.
Usually, the optimal value of y is found via hyperparameter search (Li et al., 2019), but our formula suggests another strategy: one obtain any desired variance Var [ūe] by setting ? to:
(- -v-1d=22)
v (d= 2)2 de
7
(5)
For example, for Var [de] = == and d₂ = 2048 we obtain y x 6.78, which falls right in the middle of [5, 10] -0]-aa usual search region for y used by ZSL and representation learning practitioners Li et al. (2019); Zhang et al. (2019); Guo et al. (2020). The above consideration not only gives a theoretical understanding the trick, which we believe is important on own right, but also allows to speed up the search by either picking the predicted "optimal" value for y or by searching in its vicinity.
3.3 UNDERSTANDING ATTRIBUTES NORMALIZATION TRICK
We showed in the previous subsection that "normalize+scale" trick makes the variances of de independent from variance of weights, features and attributes. This may create impression that it does not matter how we initialize the weights normalization would undo any fluctuations. However it is not true, because it is still important how the signal flows under the hood, i.e. for unnormalized and unscaled logit value Ue = z Pe. Another common trick in ZSL is the normalization of attribute vectors unit norm a,
a We provide some theoretical underpinnings of its importance.
Let's first consider linear case for Pe. i.e. Hep is an identity. thus ÿ. = x = =z Va.. Then, the way we initialize V crucial since Var [ū]. depends on To derive an initialization scheme people use 3 strong assumptions for the inputs Glorot & Bengio (2010); He et al. (2015); Chang et al. (2020): 1) they are zero-centered 2) independent from each other; and 3) have the covariance matrix of the form o² I. But in ZSL setting, we have two sources of inputs: image features z and class attributes a.. And these assumptions are safe to assume only for but not for a., because they do not hold for the standard datasets (see Appendix H). To account for this, we derive the variance Var [ūc] without relying on these assumptions for aa (see Appendix B):
Var [Te] = d2 Var [z:] Var [V.g] E [
From equation (6) one can see that after giving up invalid assumptions for a,., pre-logits variance Var [ūc] now became dependent on ||a.]|2. which is not captured by traditional Glorot & Bengio (2010) and He et al. (2015) initialization schemes and thus leads to poor variance control. Attributes normalization trick rectifies this limitation, which is summarized in the following statement.
Statement 2 (informal). Attributes normalization trick leads to the same pre-logits variance as we have with Xavier
fan-out fan-out initialization. (see Appendix B for the formal statement and the derivation).
Xavier fan-out initialization selects such a scale for a linear layer that the variance of backward pass representations is preserved across the model (in the absence of non- non-linearities). The fact that attributes normalization results in the scaling of Po equivalent to Xavier fan-out scaling and not some other one is a coincidence and shows what underlying meaning this procedure has.
5

---

Published as a conference paper at ICLR 2021
Linerr model dl
Traditoral MLP
Linerr model -AN:
ZIL, HLP
Ner-inear mide 22:0--6A -N5
25L NJP +0X
Ser Ner-Inear mode +CN -AN -005
500
1000 1550
2000
25:00
302 2000 15000
2000
2502
5.00
1200
1500
2200
2200
teration
1000.01
Raroddnn
Figure 1: Logits variances (two left plots) and the approximate loss landscape smoothness (right plot) measured during training for different models. From variances plots, one can observe the following picture: linear model without NS (normalize+scaale and AN (attributes normalization) has diverging variance, but adding NS+AN fixes this pushing the variance to 1. For a non-linear model, using NS and AN on their own is not enough and the variance deteriorates, but class normalization (CN) rectifies it back to 1; see additional analysis in Appx E. On the right plot, the maximum gradient magnitude over batches at given iteration for different classification models is presented. As stated in 3.5, 7SL. attribute embedders have more irregular loss surface than traditional models: large gradient norms indicate abrupt changes in the landscape. Class normalization makes more smooth; more analysis in Appx F.
3.4 CLASS NORMALIZATION
What happens when Pe is not linear? Let h. = Ha((a)) be the output of H.p The analysis of this case is equivalent to the previous one but with plugging in h. everywhere instead of a.. This lead to:
Var [ūc] = dz Var [=:]
Var [Vi)]
E [1
(7)
As result, to obtain Var [Tc] Var [zy] property, we need to initialize Var [Vy] the following way:
Var [Ry] = d, -. hh
This makes the initialization dependent on the magnitude of |/hl instead of ||aell. normalizing attributes to a unit norm would not be sufficient to preserve the variance. To initialize the weights of V using this formula, two-step data-dependent initialization is required: first initializing Hęp then computing average | and then initializing V. However, this is not reliable since ||hell changes on each iteration, so we propose more elegant solution to standardize he
S(he) (he /
As one can note, this is similar to BatchNorm standardization without the subsequent affine transform. but we apply it class-wise on top attribute embeddings hg. We plug it in right before V. i.c. (a.a VVS (Hp())) This does not add any parameters and has imperceptible computational overhead. At test time, we use statistics accumulated during training similar to batch norm. Standard- ization (9) makes inputs V have constant norm, which now makes trivial pick proper value to initialize V:
[IIS = d, Var [Vy]] h
(10)
dzdr
We coin the simultaneous use of (9) and (10) class normalization and highlight its influence in the following statement. See Fig. 3 for the model diagram, Fig. 1 for empirical study of its impact, and Appendix C for the assumptions, proof and additional details.
Statement 3 (informal). Standardization procedure (9) together with the proper variance formanc (10), preserves the variance between and ÿ for mutli-layer attribute embedder Po-
3.5 IMPROVED SMOOTHNESS
We also analyze the loss surface smoothness for Pg. There are many ways to measure this notion (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Dinh et al., 2017; Skorokhodov & Burtsev,

---

Published as a conference paper at ICLR 2021
2019), but following Santurkar et al. (2018), we define it in "per-layer" fashion via Lipschitzness:
gf
max |X EX*-1122 I
| Ww
(11)
where is the layer index and X is its input data matrix. This definition is intuitive: larger gradient magnitudes indicate that the loss surface is prone to abrupt changes. We demonstrate two things:
1. For each example in a batch, parameters of a 7SL. attribute embedder receive K more updates than typical non-ZSL classifier, where K is the number of classes. This suggests hypothesis that it has larger overall gradient magnitude, hence more irregular loss surface.
2. Our standardization procedure (9) makes the surface more smooth. We demonstrate it by simply applying Theorem 4.4 from (Santurkar et al., 2018).
Due to the space constraints, we defer the exposition on this to Appendix
4 CONTINUAL ZERO-SHOT LEARNING
4.1 PROBLEM FORMULATION
In continual learning (CL), model is being trained on a sequence of tasks that arrive one by one. Each task is defined by a dataset D = {xr, yf h--1 of size N,. The goal of the model is to learn all the tasks sequentially in such way that each task t it has good performance both on the current task and all the previously observed ones. In this section we develop the ideas of Chaudhry et al. (2019) and formulate Continual Zero-Shot Learning (CZSL) problem. Like in CL, CZSL also assumes a sequence of tasks, but now each task is a generalized zero-shot learning problem. This means that apart from D' we also receive set of corresponding class descriptions A for each task t. In this way, traditional zero-shot learning can be seen as a special case of CZSL with just two tasks. In Chaudhry al. (2019), authors evaluate their zero-shot models on each task individually, without considering the classification space across tasks; looking only one step ahead, which gives limited picture of the model's quality. Instead, we borrow ideas from Generalized ZSL (Chao et al., 2016; Xian et al., 201 88), and propose to measure the performance on all the seen and all the unseen data for each task. More formally, for timestep t we have the datasets:
T
D DD' r=1
D D D
AS = AS'-UA'
AT
U A r=t+1
(12)
r=1+1
r=1
which are the datasets of all seen data (learned tasks), all unseen data (future tasks), seen class attributes, and unseen class attributes respectively. For our proposed CZSL,, the model at timestep t has access only data D and attributes A', but its goal is have good performance on all seen data DS and all unseen data Da with the corresponding attributes sets A and A²t, For T = 2, this would be equivalent to traditional generalized zero-shot learning, But for T > 2, a novel and a much more challenging problem.
4.2 PROPOSED EVALUATION METRICS
Our metries for CZSL use GZSL metrics under the hood and are based on generalized accurucy (GA) (Chao et al., 2016; Xian et al., 2018a). "Traditional" seen (unseen) accuracy computation discards unseen (seen) classes from the prediction space, thus making the problem casier, since the model has fewer classes to be distracted with. For generalized accuracy, we always consider the joint space of both seen and unseen and this is how GZSLL- and GZSL-U are constructed. We use this notion to construct mean seen (mS), mean unseen (mU) and mean harmonic (mH) accuracies. We do that just SELL-S/GSSL-U/GGSSS
by measuring all the future data as unseen. Another set of CZSL metrics are mean joint accuracy (mJA) which measures the performance across all the classes and mean area under seen/unseen curve (mAUC) which is adaptation of AUSUC measure by Xian al. (2018a). A more rigorous formulation of these metrics is presented in Appendix G.2. Apart from them, we also employ a popular forgetting measure (Lopez-Paz & Ranzato, 2017).
at each timestep, considering all the past data as seen and
7

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
5.1 ZSL EXPERIMENTS
Experiment details. We use 4 standard datasets: SUN (Patterson et al., 2014), CUB (Welinder et al., 2010), AwAl and AwA2 and seen/unseen splits from Xian et al. (2018a). They have 645/72, 150/50, 40/10 and 40/10 seen/unseen classes respectively with da being equal to 102, 312, 85 and 85 respectively. Following standard practice, we use ResNet101 image features (with d; = 2048) from Xian et al. (2018a). Our attribute embedder Pe is a vanilla 3-layer MLP augmented with standardization procedure and corrected output matrix initialization 10. For all the datasets, we train the model with Adam optimizer for 50 epochs and evaluate it at the end of training. We also employ NS and AN techniques with ? for NS. Additional hyperparameters are reported in Appx D. To perform cross-validation, we first allocate 10% of seen classes for validation unseen data (for AwA1 and AwA2 we allocated 15% since there are only 40 seen classes). Then we allocate 10% out of the remaining 85% of the data for validation seen data. This means that in total we allocate 30% of all the seen data to perform validation. It is known (Xian et al., 2018a; Min et al., 2020), that GZSL-H score be improved slightly by reducing the weight of seen class logits during accuracy computation since this would partially relieve the bias towards seen classes. We also employ this trick by multiplying seen class logits by value s during evaluation and find its optimal value using cross-validation together with the other hyperparameters. On Figure 4 in Appendix D.4, we provide validation/test accuracy curves of how it influences the performance.
Evaluation and discussion. We evaluate the model on the corresponding test sets using 3 metrics as proposed by Xian et al. (2018a): seen generalized unseen accuracy (GZSL-U), generalized seen accuracy (GZSL-S) and GZSL-S/GZZL-U harmonic mean (GZSL-H), which is considered to the main metric for ZSL. Table shows that our model has the state-of-the-an in 3 out of 4 datasets.
Training speed results. We conducted survey and rerun several recent SotA methods from their official implementations to check their training speed, which details we report Appx D. Table shows the average training time for each of the methods. Since our model is just vanilla MLP and does not use any sophisticated training scheme, it trains from 30 to 500 times faster compared to other methods, while outperforming them in the final performance.
5.2 CZSL EXPERIMENTS
Datasets. We test our approach in CZSL scenario on two datasets: CUB Welinder al. (2010) and SUN Patterson et al. (2014). CUB contains 200 classes and is randomly split into 10 tasks with 20
Table 2: Generalized Zero-Shot Learning results. S, U denote generalized seen/unseen accuracy and H is their harmonic mean. Bold/normal blue font denotes the best/second- result.
SUN S
CUB
AwAl
AwA2 S
Avg training
H
S H
U
H
U
H
time
DCN (Liu et al., 2018)
25.5 37.0 30.2
28.4
60.7
38.7
25.5 84.2
39.1
50 min
RN (Sung et al,, 2018)
38.1 61.4
47.0
31.4
91.3 46.7
30.9
93.4
45.3
35 min
F-CI (Xian al,, 201
42.6 36.6
39.4
57.7
43.7
49.7
$7.9
61.4
59.6
CIZSL (Elhoseiny & Elfekii, 2019)
27.8
24.6
hours
CVC-ZSL (Li et al, 2019)
36.3 42.8
39.3
47.4 47.6 47.5
62.7
77.0 69.1
56.4
81.4 66.7
hours
SGMA (Zhu et al, 2019)
36.7
71.3 48.5
37.6 87.1 52.5
SGAL (Yu Lec, 2019)
42,9 31.2
36.1
47.1 44.7 45.9
52.7
75.7
62.2
55.1 81.2
65.6
50 min
DASCN (Ni et al., 2019)
42.4 38.5
40.3
45.9
51.6
59.3
68.0 63.4
R.VABGAN-D2 (Xian et al., 2019)
45.1 38.0
41.3
48.4 60.1 53.6
$7.6 70.6 63.5
TE-VAGAAN (Narayan et al., 2020)
45.6 40.7
43.0
52.8 64.7 58.1
59.8 75.1 66.6
1.75 hours
EPGN (Yu et al. 2020)
52.0 61.1 56.2
62.1 83.4 71.2
52.6 83.5 64.6
DVBE (Min et al., 2020)
45.0 37.2
40.7
53.2 60.2 56.5
63.6
70.8 67.0
LsrGAN (Vyas ct al., 2020)
44.8 37.7
40.9
48.1 59.1 53.0
54.6
74.6 63.0
1.25 hours
ZSML (Verma et 2020)
60.0 52.1 55.7
57.4
71.1 63.5
58.9
74.6 65.8
3-layer MLP
31.4 40.4 35.3
45.2
48.4
46.7
57.0
69.9
62.8
54.5
72.2 62.1
3 l-ayerr MLP Eq. (9)
41.5 41.3
41.4
49.4
48.6
49.0
60.1
73.0
65.9
60.3
75.6 67.1
30 seconds
laayrr MLP Eq. (10)
24.1 37.9
29.5
45.3
44.5
44.9
58.4
70.7
64.0
52.1
72.0 60.5
3 3-layrr MLP CN (i.c. (9) + (10))
44.7 41.6
43.1
49.9
50.7
50.3
63.1
73.4
67.8
60.2
77.1 67.6
8

---

Published as a conference paper at ICLR 2021
Table 3: Continual Zero-Shot Learning results with and without CN. Best scores are in bold blue.
CUB
SUN
mAUC f
mH*
mJA f
Forgetting +
mAUCt
mHf
mJAt
Forgetting
I
EWC-online (Schwarz et al., 2018)
11.6
18.0
25.4
0.08
2.7
9.6
11.4
0.02
EWC-enline + ClassNorm
14.1 (228
23.3 202
28.6 136
0.04 506
4.8" 198
14.3
(4.
15.8 196
0.03 308
MAS-online (Aljundi et al., 2017)
11.4
17.7
25.1
0.08
2.5
9,4
11.0
0.02
MAS online + ClassNom
14.0
23.8
28.5
0.05
4.8
14.2
15.8
0.03
A-GEM (Chaudhry et al., 2019)
10.4
17.3
23.6
0.16
2.4
9.6
10.8
0.05
A-GEM + ClassNorm
13.8
23.8
28.2
0.06 4.06
4.6 4.6
14,2
15,4
0.04 20%
Sequential
9.7
17.2
22.6
0.17
2.3
9.3
10.4
0.05
Sequential + ClassNorm
13,5-
23,0*
27.9-
0,05 Tr
4.6*
14.0*1
15.3-1%
0.03 tis
Multi-task
23.4
24.3
39.6
0.00
4.2
12.5
14.9
0.00
Multi-task + ClassNorm
26.5 2335
30.0
42.6
0.01
6.2
14.8
18.5
0.01
classes per task. SUN contains 717 classes which is randomly split into 15 tasks, the first 3 tasks have 47 classes and the rest of them have 48 classes each (717 classes are difficult to separate evenly). We use official train/test splits for training and testing the model.
Model and optimization. We follow the proposed prossvvalidation procedure from Chaudhry et al. (2019). Namely, for each run we allocate the first 3 tasks for hyperparameter search, validating on the test data. After that we reinitialize the model from scratch, discard the first 3 tasks and train it on the rest of the data. This reduces the effective number of tasks by 3, but provides a more fair way to perform cross- voos--vlliationn Chaudhry al. (2019). We use ImageNet-pretrained ResNet- 18 model as an image encoder E(x) which is optimized jointly with Pe. For CZSL experiments, Pe is a 2-layer MLP and we test the proposed CN procedure. All the details can be found in Appendix G.
We test our approach on 3 continual learning methods: EWC Kirkpatrick et al. (2017), MAS Aljundi et al. (2017) and A-GEM Chaudhry et al. (2019) and 2 benchmarks: Multi-Task model and Sequential model. EWC and MAS fight forgetting by regularizing the weight update for a new task in such a way that the important parameters are preserved. A-GEM maintains a memory bank of previously encountered examples and performs gradient step in such a manner that the loss does not increase on them. Multi-Tase is an "upper bound" baseline: a model which has an access to all the previously encountered data and trains on them jointly. Sequential is a "lower bound" baseline: model which does not employ any technique at all. We give each model an equal number of update iterations on each task. This makes the comparison of the Multi- Task baseline to other methods more fair: otherwise, since its dataset grows with time, it would make times more updates inside task than the other methods.
Evaluation and discussion. Results for the proposed metrics mU, mS, mH, mAUC, mJA and forgetting measure from Lopez-Paz & Ranzato (2017) are reported in Table 3 and Appendix G. As one can observe, class normalization boosts the performance of classical regularization-bat replay-based continual learning methods by up to 100% and leads to lesser forgetting. However, we are still far behind traditional supervised classifiers as one can infer from mJA metric. For example, some state-of-te of-the-r- approaches on CUB surpass 90% accuracy Ge et al. (2019) which is drastically larger compared to what the considered approaches achieve.
and
6 CONCLUSION
We investigated and developed normalization techniques for zero-shot learning. We provided theo- retical groundings for two popular tricks: normalize+scale and attributes normalization and showed both provably and in practice that they aid training by controlling a signal's 's variance during a forward pass. Next, we demonstrated that they are not enough to constrain a signal from fluctuations for a deep ZSL model. That motivated us to develop class normalization: new normalization scheme that fixes the problem and allows to obtain SotA performance on 4 standard ZSL datasets in terms of quantitative performance and training speed. Next, we showed that ZSL attribute embedders tend to have more irregular loss landscape than traditional classifiers and that class normalization partially remedies this issue. Finally, we generalized ZSL to broader setting of continual zero-shot learning and proposed a set of principled metrics and baselines for it. We believe that our work will spur the development of stronger sero-shot systems and motivate their deployment in real-world applications.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015.
Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bertt Schicle. Multi-cue zero-shot learning with strong supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 59-68, 2016a.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classifica- tion. PAMI, 38(7):1425-14388 2016b.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. CoRR, abs/1711.09601, 2017. URL httpt:tttaaaee /taetp://arxi...o. org/ abs /1 711. 696
Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffre Hinton. Layer normalization. arXiv preprint arXiv: 1607.0640, 2016.
Sean Bell, C. Lawrence Zitnick, Kavita Bala, and Ross Girshick. Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Patiern Recognition (CVPR), June 2016.
Aurélien Bellet, Amaury Habrard, and Mare Sebban. A survey metric learning for feature vectors and structured data, arXiv preprint 1rXiv:1306.6700 2013.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. Interna- tional Conference on Learning Representations, 2020. URL https: http:://////eeeeeeeewee httpeeewwwweeeeee/reeeeeeee net//orummii H1 m1l1ma24r PB.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2016a.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, pp. 5327-5336, 2016b.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Sha. An empirical study and analysis generalized zero-shot lcarning for object recognition in the wild, In ECCV (2), pp. 52-68, 2016. URL https:
//doi.org//// 10.00..... 11000100000000 978-9-888777 3- 31 9-46475-6644 4.
Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019. URL https:
// //oppnrevieeeeee net. wttee///orrmmiddid22222 _C5XX.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL http:/// /ttp///ppeeeengsseeeeeeee// ml press/v119/ c html.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv: /ttt:::773.043333 2017.
Mohamed Elhoseiny and Mohamed Elfeki. Creativity inspired zero-shot learning. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write a classifier: Zero-shot learning using purely textual descriptions. In The IEEE International Conference on Computer Vision (ICCV), December 2013.
Mohamed Elhoseiny, Francesca Babiloni, Rahaf Aljundi, Marcus Rohrbach, Manohar Paluri, and Tinne Tuytelaars, Exploring the challenges towards lifelong fact learning. CoRR, 10s/1 2018. URL attp:://axie www.viivrrggaooo wogg///gaaaoo. abs/ 111111111111 10524.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR 2009., pp. 1778-1785. IEEE, 2009.
10

---

Published as a conference paper at ICLR 2021
Rafael Felix, Vijay B. G. Kumar, Ian Reid, and Gustavo Cameiro. Multi-modal cycle-consistent generalized zero-shot learning. The European Conference on Computer Vision (ECCV), September 2018.
Vittorio Ferrari and Andrew 7isserman. Learning visual attributes. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis (eds.), Advances Neural Information Processing Systems 20, pp- 433-440. Curran Associates, Inc., 2008. URL http:/ / papers www.aappsspppppeee http://eeppppe////eaaaaaaoo
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio. Jeff Dean, Marc Aurelio Ranzato, and Tomas Mikolov. Devise: deep visual-semantic embedding model.
C. Burges, Bottou, M. Welling. Z. Ghahramani, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems 26, pp. 2121-2129. Curran Associates, Inc., 2013. URI. hwtttttppppppeeee peppprrsniss... wiipp.cc..... ccppaprr//////c 5204-devis se- a deep-vi vi sual 1 semant c- embedd ng- pdd..
Weifeng Ge, Xiangru Lin, and Yizhou Yu. Weakly supervised complementary parts models for fine-grained image classification from the bottom up. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4367-433777 0477-, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-
256. Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http:/ httpp://ppoceeee http:///ccccccec..ccoo. press/ vvv/ htt//glooooiaseom////oo//oooooot hhtml...
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, pp. 2672-2688 2014.
Jianzhu Guo, Xiangyu Zhu, Chenxu Zhao, Dong Cao, Zhen Lei, and Stan Li. Learning meta face recognition in unseen domains. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Y. Guo, G. Ding, J. Han, and Y. Gao. Zero-shot learning with transferred samples. IEEE Transactions on Image Processing, 26(7):3277-322 2017.
Yuchen Guo, Guiguang Ding, Jungong Han, and Yue Gao. Synthesizing samples for zero-shot learning. In Proceedings of the Twenty-Sixth International Joini Conference on Artificial Intelligenne, IJCAI-17. pp. 1774- 1780, 2017. doi: 10.24963/ijcai. htt.///aaaa..11724444.. 2017/2 URL https: //tss:///ooss/ org /0 24963/ /1/////33///////// 2017/248
David Ha, Andrew Dai, and Quoe Le. Hypernetoorks. arXiv preprint 1rXiv:1609.09100 2016.
K. He, Zhang, S. Ren, and Sun. Delving deep into rectifiers: Surpassing human- performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1::1-42, 1997.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Conference on Learning Representations, 2020. URL https: / //pppeeevii ew net/ wtw.roorurnrrrroer sYvr.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR. URL http:/ /ttp////cccceeeeeeeeeeee mlr www.s.prss//e...e / vv http///////////////////// 1offf15. html.
David Isele, Mohammad Rostami, and Eric Eaton. Using task features for zero-shot knowledge transfer in lifelong learning. In International Joint Conferences on Artificial Intelligence, 2016.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadar- rama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv: 5093, 2014.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arvi: 1600 1609.6609.800 2016.
11

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. stat, 1050:1, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Baroskka-aaiiskka,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-333 3526, 2017.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolu- tional neural networks. arXiv preprint urXiv: 1510085,, 2015.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 4281-4289 2018.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, pp. 951-958. IEEE, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Attribut- classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):453-465, 2013a.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Atribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):433446 465, 2013b.
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In ICCV, 2015.
Kai Li, Martin Rengiang Min, and Yun Fu. Rethinking zero-shot learning: A conditional visual elassification perspective. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Shichen Liu, Mingsheng Long, Jianmin Wang. and Michael Jordan. Generalized zero-shot learning with deep calibration network. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 2005-2015. Curran Associates, Inc., 2018. URL http:/ http:://papprs.ttt. .nw.rr.s..pss...mm........m wttp://ss..is....ppppee..p 7471 1771-general zed- wwwezeerrosseeeee wwwttsstttttttaee wewaannnnnwwiiieeeeee with deep- cal inprraiioonnetorr....o
David Lopez-Paz and Marc Aurelio Ranzato. Gradient episodic memory for continual Icarning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6467-6476. Curran Associates, Inc., 2017. URL http: /tattt::/ppppate /pppers. nips. cc httpppppppp2/5552 1e155-gradien ent episociccc cele--mmmorry ry- for cont inual -eerning .dff.
Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, and Qiang Yang. Cosine normalization: Using cosine similarity instead of dot product in neural networks. In International Conference on Artificial Neural Networks, pp. 382-391. Springer, 2018
Shaobo Min, Hantao Yao, Hongtao Xic, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang. Domain-aware visual bias eliminating for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snock, and I. Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. arXiv preprint a7Xiv2200..07833, 2020.
Jian Ni, Shanghang Zhang, and Haiyong Xie. Dual adversarial cemaatics-consisten network for generalized zcro-shot learning. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 6146-6157. Curran Associates, Inc., 2019.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint / 2013.
Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The sun attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision, (08(1-2):59) 81,, 2014.
Bernardino Romera-Pareee Paredes and Philip Torr An embarrassingly simple approach to zero-shot learning. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2152-2161, Lille, France, 07-09 Jul 20155 PM.R. URL http:/t::/ /ttp///rceeeeeeeee ngs. ml www.ss//aseoo............ v/7. rome www.ooeeaaaaeeeeeaaaeeeeee paredes 5.mml..
12

---

Published as a conference paper at ICLR 2021
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization?
S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Biandn Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 2483. Curran Associates, Inc., 2018. URL http:/ wwwpprs.nips.coomm 7515- how deee-doe---ee batehbaacc----
maziionninnn on- hellp-pptiiizateee
pdf.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinske Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: scalable framework for continual learning. In International Conference on Machine Learning, pp. 4528-4537, 45337, 2018
Yichun Shi, Xiang Yu, Kihyuk Sohn, Manmohan Chandraker, and Anil K. Jain. Towards universal representation learning for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Saurabh Singh and Shankar Krishman, Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Ivan Skorokhodov and Mikhail S. Burtsev. Loss landscape sightseeing with multi-point optimization. CoRR, abs/1910.03867, 2019. URL http:////// www.eiiiioooggaaooooo abs///////////////////////////// 1910. 03867.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1857-1865, 2016.
Flood Sung. Yongxin Yang. Li Zhang, Tao Xiang. Philip HS Tom, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. CoRR, abs/1904.07734, 2019. URL http::/tt///// //aaaiivvooro....ooo.o abaaabaal990 1904. 1.9400000000 07734.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learning for generalized zero-shot learning. In AAAI, 2020.
Maunil R Vyas, Hemanth Venkateswara, and Sethuraman Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In European Conference on Computer Vision (ECCV), 2020.
Kun Wei, Cheng Deng. and Xu Yang. Lifelong zero-shot learning. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, IJCAI-20, pp. 551-557. International Joint Conferences on Artificial Intelligence Organization, 2020a. URL https:::: ottp:////...../////////// Main track.
ijcai. 1icai.20200 2020/77.
Kun Wei, Cheng Deng, and Xu Yang. Lifelong zero-shot learning. IJCAI, 2020b.
Welinder, Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and Perona. Caltech-UCSD Birds 200.
Technical Report CNS-TR SAS-TR--21000 2010-001, California Institute of Technology, 2010.
Yuxin Wu and Kaiming He, Group normalization. The European Conference on Computer Vision (ECCV), September 2018.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata, Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. PAMI, 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 201 8b.
Yonggin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. F-vaegan-d2: A feature generating framework for any-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR), June 2019.
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. Deep leamingg for person re-identification: A survey and outlook. 2020.
Hyeonwoo and Beomhee Lee. Zero-shre learning simultaneous generating and learning. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett Advances in Neural Information Processing Systems 32, pp. 46-56. Curran Associates, Inc., 2019. URL http: / //tp:://aaaa pappers rs nips, cc/ paper/B8 88300 zero- shot - 1a rni ng- vi v a mu m taneous generatinneneeeeeeeeeeee and- 1ea marning. pd
13

---

Published as a conference paper at ICLR 2021
Yunlong Yu, Zhong Ji, Jungong Han, and Zhongfei Zhang. Episode-based prototype generating network for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
Han Zhang, Jing Koh, Jason Baldridge. Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. arXiv preprint arXiv:2 101.0420, 2021.
Zhang, Yannis Kalantidis, Mareus Rohrbach, Manohar Paluri, Ahmed Elgammal, and Mohamed Elhoseiny, Large-scale visual relationship understanding. In AAAI, 2019.
Zhang. Xiang, and Shaogang Gong. Learning deep embedding model for zero-shot learning. In CVPR. 2016.
Zhang, Tao Xiang, Shaogang Gong, Learning a deep embedding model zero-shot learning. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR). July 2017.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning semantic similarity embedding. In ICCV, pp. 4166 4174, 2015.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng. and Ahmed Elgammal. generative adversarial approach for zero-shot learning from noisy texts. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang. Xi Peng. and Ahmed Elgammal. Semantic guidded multi- attention localization for zero-shot learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox. and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 14943-14953. Curran Associates, Inc., 2019. URL http::/papers-.. ps cc/paper/ 9632-semanti c-gui ded mu ti ci-attention local zat for- learning pdf.
14

---

Published as a conference paper at ICLR 2021
A "NORMALIZE + SCALE" TRICK
As heing discussed in Section 3.2, "normalize+seale" trick changes the logits computation from usual dot: product to the scaled cosine similarity:
De Pc) De
Pe
(13)
|pell
where de the logit value for class c; is an image feature vector; Pe the attribute embedding for class c:
p. Pe(a.) V Hy((a))
(14)
and y is the scaling hyperparameter. Let's denote a penultimate hidden representation of Po as h. (ac). We note that in case of linear Po, we have h. Let's also denote the dimensionalities of and he by d, and dn.
A.1 ASSUMPTIONS
To derive the approximate variance formula d. we will use the following assumptions approximate identities:
All weights in matrix V:
are independent from each other and from an and he, (for all k, i);
[Vy] = 0 for all j;
Var [V,z] S. for j.
(ii) There exists s.t. (2 e)-th central moment exists for each of he.,y-.., he.d,, We require this technical condition to be able apply the central limit thcorem for variables with non-equal variances.
(iii) All he,j independent from each other for + j. This is the least realistic assumption from the list, because in case linear Ps would equivalent to independence coordinates in attribute vector a.. We are not going to use in other statements. As we show in Appendix A.3 it works well in practice.
(iv) All Pe.i. Po.j are independent between each other. This also a nasty assumption, but more safe to assume in practice (for example, it is easy to demonstrate that Cov [pe,i, Pe.sl = 0 for i & j). We are going use it only in normalize+scale approximate variance formula derivation.
(v) z ~ (00 8² I). This property safe to assume since is usually hidden representation of deep neural network and each coordinate is computed as a vector-vector product between independent vectors which results in the normal distribution (see the proof below for pe ~/e N(0, s0,ss) T)).
(vi) For E E (z, p&) we will use the approximations:
and EEE
EEG)) R
(15)
E
|
This approximation is safe to use if the dimensionality of & is large enough (for neural networks it is definitely the case) because the contribution of each individual & in the norm ||||| negligible.
becomes
Assumptions (i-v) are typical for such kind of analysis and can also be found in (Glorot & Bengio, 2010; He al., 2015; Chang et al., 2020). Assumption (vi), as noted, holds only for large-dimensional inputs, but this is exactly and validate that using leads decent approximation on Figure 2.
A.2 FORMAL STATEMENT AND THE PROOF
Statement (Normalize+scale trick). If conditions (i)-(vi) hold, then:
Var [ic] = Var K1
Pa
2
(16)
p
(d, 2)
Proof. First of all, we need to show that Pe,s (0, for some constant s". Since
De.
Viyha.
(17)
15

---

Published as a conference paper at ICLR 2021
from assumption (i) we can easily compute its mean:
dp
E
E
Vishes
(18)
10 ha.ll
(19)
E [V..]= E [he.s]
(20)
dp
0 E [hesl
(21)
O.
(22)
and the variance:
Var [P..]
[p
(B (e..l)*
(23)
[p
(24)
27
Vinlhos
(25)
du
E
Vas Vilethattione
(26)
Using B [V., V.,]] = 0 for A * J. we have:
dg Vie No.
(27)
dx
[Vit]]
[b2
(28)
Since 8. = [Vs]] = TE. [v]]] - B [Vo.)² = E. [v:3] we have:
dn
E [AC.p]
(29)
F.E.
(30)
F.
[1]har
(31)
S
(32)
(33)
Now, from the assumptions (ii) and (iii) we apply Lyapunov's Central Limit theorem Pc,s. which gives us:
Vap Pe.i NN(0 1)
(34)
For finite da, this allows us say that:
Do ~ NO( s²)
(35)
16

---

Published as a conference paper at ICLR 2021
Now note that from (vi) have:
B
[e
E
Pe
(36)
1
PP
E
Pc
(37)
PPP
IE
(E
[z]
(p.))
(38)
||
Pe
.I
(0.0)
(39)
|pp.]
(40)
Since & ~ (N, sf) for E (z,p.),
dg
follows scaled inverse chi-squared distribution with inverse variance
= 1/85, which has known expression for expectation:
de
Tde
de
E
(41)
de
2
2)
Now we are left with using approximation (vi) and plugging in the above expression into the variance formula:
Var [da] = Var
Pc
(42)
|
|eel
E
Pe
TR.
Pe
(43)
PP
|P-|
(1 |
Pc
F.
(44)
P.P
(z
(45)
|| p
7
[)]]
d2
-E
1
d2
(46)
d=
||
d,
|p.113
2"
d
d2
de
E
Pe
E
z=
Pc
8 (d.
2)
sP
s(dd - 2)
(47)
E Pe
pe
Ia, Po
(48)
s 2)2
d=
'E
G
sP (d+ - 2)
(49)
de
(50)
(d. - 2)
7dd
(51)
(d. 2)
(52)
A.3 EMPIRICAL VALIDATION
In this subsection, we validate the derived approximation empirically. For empirical validation of the variance analyzis, see Appendix F., For this, we perform two experiments:
Synthetic data. An experiment on a synthetic data. We sample æ ~ ON - (O Ia) different dimensionalities d = 32, 64, 128, ..., 8192 and compute the cosine similarity:
for
7V
(53)
|
After that, we compute Var [a] and average it out across different samples. The result is presented on figure 2a.
17

---

Published as a conference paper at ICLR 2021
Predicted sid
0.02
0.175
Estimated sid
Stadard devitton
0.125
0.100
0.003
0.075
0.232
0.050
0.232
32
64
128
256
512
1024
2048 4096
8192
Dimansioraliy
Darsnes
(a) Empirical validation of variance formula (4) (b) Empirical validation of variance formula (4) on real-world data and for a wallworld model
synthetic data
Figure 2: Empirical validation of the derived approximation for variance (4)
Real data. We ImageNet--rettrined ResNe1101 features and real class attributes (unnormalized) for SUN, CUB, AwAl, AwA2 and aPY datasets. Then, we initialize a random 2-layer MLP with 512 hidden units, and generate real logits (without scaling). Then we compute mean empirical variance
and the corresponding standard deviation over different batches of size 4096. The resulted boxplots are presented figure 2b.
In both experiments, we computed the logits with y = 1. As one can see, even despite our demanding assumptions, our predicted variance formula is accurate for both synthetic and real-world data.
B ATTRIBUTES NORMALIZATION
We will use the same notation as in Appendix A. Attributes normalization trick normalizes attributes to the unit Ly-norm:
(54)
|a ll
We will show that helps to preserve the variance for pre-logit computation when attribute embedder He is linear:
Je F(ae) x
(55)
non-linear attribute embedder it not true, that's why we need proposed initialization scheme.
B. ASSUMPTIONS
We will need the following assumptions:
Feature vector z has the properties:
E 0:
Var [z.] = s² for all i = 1..... d2.
All z; are independent from each other and from all Pa.-
(ii) Weight matrix V is initialized with Xavier fan-out mode, i.c. Var [ig] 1/d, and are independent from each other.
Note here, that we do not have any assumptions on a.. This is the core difference from Chang et al. (2020) and is an essential condition for ZSL (see Appendix H).
B.2 FORMAL STATEMENT AND THE PROOF
Statement 2 (Attributes normalization for a linear embedder). If ffssumpptioss (i)-(ii) are satisfied and ||a.|2 = 1,
then:
Var [Tc]
Var [2s] =²
(56)
Proof. Now, note that:
1 [ = E
[Va]
E Via
z Va.
=
Va =0
(57)
18

---

Published as a conference paper at ICLR 2021
Then the variance for Be has the following form:
Var [Wc] [
[vc]
(58)
[
(59)
I
(s Va.)*
(60)
a.
VT
x2
Va
(61)
E
E
E
a
V
#=
Va.
(62)
E
[o.
V
E
==
V
(63)
s E
a
v
a.
(64)
5 s'd
ac
a.
(65)
since 8" = 1/d,, then:
'EEE ||a.1
(66)
since attributes normalized, i.e. ||a.||2 then:
(67)
C NORMALIZATION FOR DEEP ATTRIBUTE EMBEDDER
C. FORMAL STATEMENT AND THE PROOF
Using the same derivation as in B, one can show that for a deep attribute emhedder:
Po(a.) = o (a.)
(68)
normalizing attributes not enough to preserve the variance of Var [ūc], because
Var [+] FFF [1h
(69)
and h. (a. is normalized to unit issue, we are going to use two mechanisms:
1. A different initialization scheme:
Var [Vop] 1
(70)
d,dn
Using standardization layer before the final projection matrix:
S(x) (x fxx) 2 o.;
(71)
Pzz Ty are the sample mean and variance and Ø is element-wise division.
C.2 ASSUMPTIONS
We'll need the assumption:
Feature vector has the properties:
[z] O; Var [zx] s* for all i 1,..., d2.
All 2; independent from each other and from all pe.j-
19

---

Published as a conference paper at ICLR 2021
h Pr
ag
Oh.
S(h)
Hig (ac)
V(8.)
Figure 3: Our architecture: plain MLP with the standardization procedure (9) inserted before the final projection and output matrix V being initialized using (10).
C.3 FORMAL STATEMENT AND THE PROOF
Statement 3. If the assumption (i) is satisfied, an attribute embedder has the form Pe V S Hip and we initialize output matrix V s.t. Var [Vy] dzdh' then the variance for De is preserved:
Var [ūc] Ps [a][=:
(72)
Proof. With some abuse notation, h = Shee) (in practice, S receives batch he instead a single vector). This leads to:
E [hc] 0 and Var [he] Rs
(73)
Using the reasoning in Appendix
show that:
Var [ de s Var [Vyg] [1 [1] di [1
(74)
So we are left demonstrate that IE
[hc da:
dp
do
E
he > [1a.]] >
Var [hr.s] dn
(75)
C.4 ADDITIONAL EMPIRICAL STUDIES
D ZSL DETAILS
D. EXPERIMENTS DETAILS
this section, we cover hyperparameter and training details our experiments and also provide extended training speed comparisons with other methods.
We depict the architecture of our model on figure 3. As being said, Po is just simple multi-layer MLP with standardization procedure (9) and adjusted output layer initialization (10).
Besides, also found it useful to entropy regularizer (the methods for exploration) for some datasets:
which is often used in policy gradient
(em())= = - Hp) Σ Pe. log
(76)
We train the model with Adam optimizer with default 31 and 32 hyperparams. The list of hyperparameters is presented in Table 4. In those ablation experiments where we do not attributes normalization (2), we apply simple standardization convert attributes to zero-mest aero-mean and unit-variance.
D.2 ADDITIONAL EXPERIMENTS AND ABLATIONS
In this section we present additional experiments and ablation studies for our approach (results are presented in Table 5). We also validate:
20

---

Published as a conference paper at ICLR 2021
Table 4: Hyperparameters for ZSL experiments
SUN
CUB
AwA1
AwA2
Batch size
128
512
128
128
Learning rate
0.0005
0.005
0.005
0.002
Number of epochs
50
50
50
50
Len weight
0.001
0,001
0.001
0.001
Number of hidden layers
2
2
2
2
Hidden dimension
2048
2048
1024
512
7
5
5
5
5
Table 5: Additional GZSL ablation studies. From this table one can observe the sensitivity of normalizetscale to y value. We also highlight y importance Section 3.
SUN
CUB
AwAl
AwA2
U
H
S
Linear +NS+AN 7
11.7
35.1
17.6
5.1
20.3
66.5
31.1
20.6
71.1
31.9
Linear +NS+AN 7
11.2
37.1
17.2
10.3
53.7
17.3
13.6
72.8
23.0
21.0
50.8
29.7
Linear +NS+AN 7 =
13.8
41.0
20.6
16.8
62.0
26.4
16.8
74.2
27.4
18.9
73.2
30.0
Linear +NS+AN Y 10
17.1 40.9
24.1
14.9
61.7
24.0
36.4
46.2
40.7
27.9
86.8
42.2
Linear +NS+AN ? 20
13.9
35.5
20.0
13.8
52.7
21.9
46.4
43.3
44.8
47.9
59.4
53.1
2- layer
34.0
36.4
35.1
38.7
37.4
38.0
49.6
61.9
55.1
49.7
65.9
56.7
2- layer M 7
32.0
37.4 34.5
42.0
43.1 42.6
53.6
68.9 60.3
53.7
72.3 61.6
2- layer MLP +NS+ANN 7 = 34.4
39.6
36.8
46.9
45.0
45.9
57.3
73.8 64.5
55.4
77.1 64.5
2 layer MLP +NS+AN 7 = 10
31.7
37.5
34.4
47.0
43.3
45.1
54.1
65.5 59.3
56.0
72.4 63.2
2 layer MLP +NS+AN
20
56.4
11.0
18.4
44.4
35.9
39.7
51.4
69.3
59.1
46.4
73.7 56.9
layer MLP +NS+AN = 18.6
37.9
25.0
23.0
42.3
29.8
50.1
60.9
55.0
48.4 64.0 55.1
3-yer +NN+AN = =3 23.9
37.3
29.1
35.5
48.6 41.0
57.3
67.3 61.9
57.3
70.3 63.2
3-layer MLP +NN+ 7 31.4
40.4
35.3
45.2
50.7
47.8
58.1
70.3 63.6
58.2 73.0 64.8
3-layer MLP +NS+AN 7 = 29.7
37.8
33.3
40.7
40.5
40.6
55.4
63.0 58.9
53.8
69.6 60.7
3- layer MLP +NS+AN 20
15.8
39.5
22.6
22.2
54.0
31.4
53.8
63.8
58.3
49.2 69.4 57.6
Dynamic Normalization
31.9 39.8 35.5
22.7
56.6
32.4
58.5
68.4
63.1
55..5
70.3 62.0
Xavier
41.5 41.3 41.4
49.3
49.2
49.2
60.2
73.1 66.0
58.3
76.2 66.0
Kaiming fan-in (9)
42.0 41.4
41.7
51.1
49.2
50.1
59.8
74.3 66.2
55.4
75.6 63.9
Kaiming fan-out + (9)
42.8 41.2 42.0
51.0
49.0
50.0
60.3
73.2
66.1
56.8 76.9 65.4
Dynamic normalization. one can from formula (8), achieve the desired variance would enough to initialize V s.t. Var [Viz] 1/d, (equivalent to Xavier fan-out) and use a dynamic normalization:
DN h/E [|
(77)
(a.)). Expectation E || is computed over a batch on cach itcration. A downside of such an approach is that if the dimensionality is large, than a lot of dimensions will get suppressed Icading to bad signal propagation. Besides, one has to compute the them test time which is cumbersome.
between and Hip, i.c. Po(az) VDN( (a.))
running statistics
Traditional initializations + standardization procedure (9). These experiments ablate the necessity of using the corrected variance formula (10).
Performance of NS for different scaling values of y and different number of layers.
D.3 MEASURING TRAINING SPEED
We conduct survey and search for open- source implementations of classification ZSL papers that were recently published on top conferences. This is done by 1) checking the papers for code urls: 2) checking their supplementary: 3) searching for implementations on github.com and 4) searching authors by their names on github.com and checking their repositories list. As result, we found 8 open-source implementations of the recent methods, but them got discarded since the corresponding data not provided. We all these
21

---

Published as a conference paper at ICLR 2021
Table Training time for the recent ZSL methods that made their official implementations publicly available. We reran them on the corresponding datasets with the official hyperparameters and training setups. All the comparisons are done on the same machine and hardware: NVidia GeForee RTX 2080 Ti GPU, Intel Xeon Gold 6142 CPU and 64 GB RAM. N/C stands for "no code" meaning that authors didn't release the code for a particular dataset.
SUN
CUB
AwA1
AwA2
RelationNet Sung et al. (2018)
25 min
40 min
40 min
DCN Liu et al. (2018)
40 min
50 min
55 min
CIZSL Elhoseiny
Elfeki (2019)
hours
hours
3 hours
3 hours
CVC-ZSL Li et al. (2019)
hours
hours
1.5 hours
1.5 hours N/C
SGAL Yu & Lee (2019)
N/C
N/C
50 min
LsrGAN Vyas al. (2020)
1.1 hours
1.25 hours
1.5 hours
TF-VAEGAN Narayan et al. (2020)
1.5 hours
1.75 hours
2 hours
Ours
20 sec
20 sec
30 sec
30 sec
methods with the official hyperparameters on the corresponding datasets and report their training time in Table 6 in Appx D.
All runs made with the official hyperparameters and training setups and on the same hardware: NVidia GeForce RTX 2080 Ti GPU, x16 Intel Xeon Gold 6142 CPU and 128 GB RAM. The results are depicted on Table
As one can see, the proposed method trains 50- 50-100 faster than the recent SotA, This is due to not using any sophisticated architectures employing generative models (Xian et al., 2018b; Narayan et al., 2020); or optimization schemes like episode- based training (Li al., 2019; Yu et al. 2020).
D.4 CHOOSING SCALE FOR SEEN CLASSES
As mentioned in Section 5, reweigh class logits multiplying them scale value This is similar a strategy considered by Xian (2018a); Min (2020), but we found that multiplying by a value instead of adding it by summation more intuitive. We find the optimal scale value by cross- volssvaaiiotiion together with all other hyperparameters on the grid [1.0, 0.95, 0.9, 0.85, 0.8). On Figure 4, we depict the curves of how influences G7SL-JUG77SI-///7/1..4 for each dataset.
D.5 INCORPORATING CN FOR OTHER ATTRIBUTE EMBEDDERS
In this section, we employ our proposed class normalization for two other methods: RelationNet' (Sung al., 2018) and CVC-ZSL" (Sung al., 2018). We build upon the officially provided source code bases and the official hyperparameters
the setups. For RelationNet, the authors provided the running commands, CvC-ZSL, we used those hyperparameters for each dataset, that were specified in their paper. That included using different weight decay of 1e - 4, le - 3, le 3 and le-3 5 for AwAl, AwA2, CUB and SUN respectively. stated in Section 4.2 of the paper (Li et al., 2019). We incorporated these attribute embedders and launched them the corresponding datasets. The results are reported in Table 7. For some reason, we couldn't 't reproduce the official results for both these methods which we additionally report. As one can see from the presented results, our method gives +2.0 and +1.8 of GZSL-H improvement on average for these two methods respectively which emphasizes once again its favorable influence on the learned representations.
Class Normalization procedure
D.6 ADDITIONAL ABLATION ON AN AND NS TRICKS
In Table provide additional ablations attributes normalization and mormalize+scale tricks. As see, they greatly influence the performance of ZSL attribute embedders.
"RelationNet: http//gttthbco/////// http::/www.ooooooo///ea//onnnnnn
CCVC-ZSL: https: http:/ggghtbboooooooo i i go/ cvexSt.
22

---

Published as a conference paper at ICLR 2021
0.5
0.6
0.5
0.3
C251, [val)
C2514 [val]
GZSL [test)
GZSL (estt
9.2
G25L-8
0.1
GZ5L'S [tess]
CZSLUU [val]
0.1
CZSLU [val]
GZSL [sest)
C2514I [test]
025141
11
100
0.6
1.2
1.0
0.8
0.6
SUN (s" = 0.95)
CUB (s" 1.0)
Accurscy
Accurscy
GZSL-H [vll)
0.4
/val
GZ5LH [sest)
GZSL [eest)
C25L- [val]
G251-S [val]
6251-8 [beet)
0.2
G2SL-5 [teet)
[val]
[val]
GZSL-U [test]
GZSL-U [test]
2.2
1.0
0.9
0.8
0.7
0.6
0.5
1.2
1.
1.0
0.9
0.8
0.7
0.6
05
Scale value for seen classes
Scale value for seen classes
AwA1 (s* 0.95)
AwA2 (s* 0.95)
Figure 4: Optimal value of seen logits scale for different datasets. Multiplying seen logits by some scale 1 during evaluation leads to sacrificing GZSL-S for an increase in GZSLL-U which results in the increased GZSL--H value Xian et al. (2018a); Min et al. (2020). High gap between validation/test accuracy caused by having different number of classes in these two sets. Lower test GZSL-H than reported in Table 2 is caused by splitting the train set into train/validation sets for the presented run, iee using less data for training: we allocated 50, 30, 5 and seen classes for the validation unseen ones to construct these plots for SUN, CUB, AwAl and AwA2 respectively and an equal amount of data was devoted to being used as validation seen data, i.e. we "lost" x 25% train data in total. As one can see from these plots, the trick works for those datasets where the gap between G7SL-S and GZSL-U is large and does not give any benefit for CUB where seen and unseen logits are already well-balanced.
Table 7: Incorporating Class Normalization into RelationNet (Sung et al., 2018) and CVC-ZSL (Li et al., 2019) based the official source code and running hyperparameters. For some reason, our results differ considerably from the reported ones on AwA2 for RelationNet and on SUN for CVC-ZSL. Adding CN provides the improvement in all the setups.
SUN S
CUB S
AwA1 S
AwA2
H
H
H
S
RelationNet (official code)
38.3 62.4 47.5
28.5
87.8
43.1
10.2
88.1
18.3
RelationNet (official code) CN
40.1
62.8
48.9
29.8
88.4
44.6
12.7
88.8 22.3
CVC-ZSL (official code)
20.7 43.0 28.0
42.6
47.8 45.1
58.1
78.1 66.6
51.4
79.9 62.5
CVC-7SL (official code) + CN
24.6 42.5 31.1
44.6
48.7 46.6
58.8
79.7 67.7
53.2
80.4 64.0
RelationNet (reported)
38.1
61.4
47.0
91.3
46.7
30.9
93.4 45.3
CVC-ZSL (reported)
36.3 42.8 39.3
47.4
47.6 47.5
62.7
77.0
69.1
56.4
81.4 66.7
23

---

Published as a conference paper at ICLR 2021
Table Ablating other methods for AN and NS importance. For CVC-ZSL, used the officially provided code with the official hyperparameters, When we do not employ AN, we standardize them zero-mean and unit-variance otherwise training diverges due to too high attributes magnitudes.
SUN S
CUB S
AwAl S
AwA2
U
H
U
H
U
H
U
S
H
Linear
41.0
33.4
36.8
26.9
58.1
36.8
40.6
76.6
53.1
38.4
81.7 52.2
Linear -AN
13.8
41.0
20.6
16.8
62.0
26.4
16.8
74.2
27.4
18.9
73.2 30.0
Linear -NS
38.7
3.5
6.4
33.5
6.7
11.2
44.0
42.7
43.3
44.9
48.8
46.8
Linear -AN -NS
17.7
2.6
4.5
3.0
0.0
0.0
13.2
0.0
0.1
23.3
0.0 0.0
2-layer MLP
34,4 39.6
36.81
45.0
45.9
73.8 64.5
77.1 64.5
2-layer MLP-AN
33.8 40.1 36.7
44.9 42.9 43.9 72.5 66.8 59.1 74.1 65.8
2-layer MLP-NS
51.0 11.1 18.3 40.6 22.4 40.9 68.9 51.3 40.7 69.3 51.2
2-layer MLP-AN -NS
20.9 24.0 22.3
11.6 13.1 12.3
12.3 40.7 50.2 45.0 30.8 61.1 41.0
CVC-ZSL (official code)
20.7 43.0 28.0)22.6
42.6
47.8 45.1
58.1 78.1 66.6 51.4 79.9 62.5
CVC-ZSL (official code) -NS
18.4 40.5 25.3 23.7 56.6 33.4
15.9 65.3 25.6 14.9 49.7 22.9
CVC-ZSL (official code) -AN
31,4 30.8 31.1
24.8 57.1 34.6
44.4
82.8 57.8
17.6 89.9 29.5
CVC-ZSL (official code) -NS -AN 18.2 36.9 24.3
24.3
21.6 54.7 30.9
14.1 59.4 22.8
14.6 45.9 22.2
CVC-ZSL (reported)
36.3 42.8 39.3
39.3|47.4 47.4 47.6 67.5 77.0 69.1 56.4 81.4 66.7
Linear model [SUN]
Non-linear model (odel|[SU)) [SUN]
Non-linear model +CN [SUN]
Unner model -AN -AN-NS -MS
Non-inear model +AN +NS
Non-linear model -CN +AN -NS
Linear model -AN -NS
Not-inear model 4N4
Nor-lihear model CCN -AN +NS
Non-inear
Bon-lineer model
3
model -AN
vortr:
Nan-inear model -AN
Non-linear model -AN
verl
1500
2000
1500 Iberation
2000
1500
Ieeraion
Reration
Figure 5: Variances plots for different models for SUN dataset. See Appendix E for the experimental details.
E ADDITIONAL VARIANCE ANALYZIS
In this section, we provide the extended variance analyzis for different setups and datasets. The following models are used:
1. A linear ZSL model with/without normalize+scale (NS) and/or attributes normalization (AN).
A 3-layer ZSL model with/without NS and/or AN.
3. A 3-layer ZSL model with class normalization, with/without NS and/or AN.
These models are trained on 4 standard ZSL datasets: SUN, CUB, AwA1 and AwA2 and their logits variance calculated each iteration and reported. The same batch size, learning number of epochs, hidden dimensionalities were used. Results are presented on figures 5, 6. and 8, which illustrates same trend:
A traditional linear model without NS and AN has poor variance.
Adding NS with proper scaling of 5 and AN improves it and bounds to be close to 1.
After introducing new layers, NS and AN stop "working" and variance vanishes below unit.
Incorporating class normalization allows to push it back to 1.
Loss LANDSCAPE SMOOTHNESS ANALYSIS
F. OVERVIEW
As being said, demonstrate things:
24

---

Published as a conference paper at ICLR 2021
Linear model [CUB]
(CUB)
Non-linear +CN (CUB]
Uneer model -AN -NS
Non-inear model +AN +85
Non-linesr model -CN +AN -NS
Lincr model AN -5
Non Incar model AN +NS
6
Non linesr model -CN AN +NS
Liner -AN
Non-lies model -N5
NN
Verty
Vartec
2
0
500
1000
1500
2000
25:00
500
1000
1500
2000
2500
500
1000
1500
2000
2500
lteration
Iteration
Reration
Figure 6: Variances plots for different models for CUB dataset. See Appendix E for the experimental details.
Linear model [AwA1]
Nan-linear model [AwAl]
Non-lines linnlinarr model + CN [AwA1]
+NS
medel
model -AN -NS
Non-inear model JAN
Non-thheat model -AN 4NS
Lnner motel AAN .N5
Non-inear medel +AN .15
Non-tinear model =CN VAN NN
voryy33
Unear model -AN -tel
Not-inewr model -AN -V5
var.c.
Nor-linear model CCN -AN -V5
1500
2000
1500
2000
1000
1500
Ieeaatoo
Reration
Figure Variances plots for different models for AwAl dataset. See Appendix for the experimental details.
Linear model [AwA2]
Non-linear model [AwA2]
Non-linear model +CN (AwA2]
Unear model -AN +NS
Non linar model +AN +NS
Non linear model =CN +AN -NS
JAN -N-
Non-lnear model -AN
Nor-linear model -AN
Lnnerrmeddl
incar
model
Unner model AN I6
Non incar model -N
Non linear model -CN AN
Veryy
Party 2
1500
2000
1500
Iteration
Reration
Figure 8: Variances plots for different models for AwA2 dataset. See Appendix E for the experimental details.
25

---

Published as a conference paper at ICLR 2021
1. For each example in batch, parameters of a ZSL attribute embedder receive K more updates than typical non-ZSL classifier, where K is the number of classes. This suggests hypothesis that it has larger overall gradient magnitude, hence a more irregular loss surface.
2. Our standardization procedure (9) makes the surface more smooth. We demonstrate it by simply applying Theorem 4.4 from (Santurkar et al., 2018).
To see the first point, one just needs to compute the derivative with respect to weight Way for n-th data sample for loss surface L of a traditional model and loss surface L of a ZSL embedder:
OC
ac
OC OW Σ OC
(n)
By xa)))
(78)
Ow?
Oy:
Since the gradient has K more terms and these updates are not independent from each other (since the final representations are used to construct single logits vector after dot- dot-product with x... this may lead to an increased overall gradient magnitude.
verify this empirically by computing the gradient magnitudes for our model and its non-7.I. "equivalent"; model with the same number layers and hidden dimensionalities, hut trained to classify objects in non-ZSI. fashion.
To show that our class standardization procedure (9) smoothes the landscape, we apply Theorem 4.4 from Santurkar et al. (2018) that demonstrates that a model augmented with batch normalization (BN) has smaller Lipschitz constant. This easily done after noticing that (9) is equivalent to BN, but without scaling/shifting and is applied a class-wise instcad of the batch-wise fashion.
We empirically validate the above observations on Figures and 9.
F.2 FORMAL REASONING
ZSL. embedders are prone to have more irregular loss surface. We demonstrate that the loss surface of attribute embedder Po is more irregular compared to a traditional neural network. The reason is that its output vectors P:: Pa(a.) are not later used independently, but instead combined together in single matrix
W [PI, .... pk, compute the logits vector y Wz. Because of this, the gradient update for 0, receives K signals instead of just like for traditional model, where the number of classes.
Consider a classification neural network Fy (x) optimized with loss L and some intermediate transformation y Wf Then the gradient of L,, n-th training example with respect WGJ is computed
dCn JLn dy. (n)
Hhn
y W'
aw ay; (m) aw, 0y (m) 2)
(79)
While for attribute embedder Po (ac),, we have K times more terms in the above sum since we perform K forward passes for each individual class attribute vector a.. The gradient on n-th training example for its inner transformation y = W'x(a.) is computed as:
K
dcn Dy. æ (a.)
y ((.)) OW
Σ
(80)
From this, can that the average gradient for Po is K times larger which may lead to the increased overall gradient magnitude and hence more irregular loss surface as defined in Section 3.5.
CN smoothes the loss landscape. In contrast to the previous point, we can prove this rigorously by applying The- orem 4.4 by Santurkar et al. (2018), who showed that performing standardization across hidden representations smoothes the loss surface of neural networks. Namely Santurkar et al. (2018) 8) proved the following:
Theorem 4.4 from (Santurkar et al., 2018). For a network with BatchNorm with loss E and network without BatchNorm with loss L if:
= max SIXISS
de = max I
V
(81)
Ihem:
ge < o7
(9) mp2 (V C, 0.)²)
(82)
where yr. gi are hidden representations at the f-th layer, mn their dimensionality, ae their standard deviation. Pg 1 m (1,0(/2z) for z, 7Bt + B, is the average gradient norm, 7 is the BN scaling parameter, X is the input data matrix at layer L.
Now, it casy easy to see that our class standardization (9) is "equivalent" to BN (and thus the above theorem he applicd to our model):
26

---

Published as a conference paper at ICLR 2021
Trackiionll NL. ZSL vur
Tradkona MLP Z5. KL
Traditonal MLP
wCh
Approom abc mattppcaa)
Aguessor cac 10g- (AA.A1A
2%. H.P
23L RuP
500
10200
1520
2000
2530
500
3003
3503
2003
2503
530
1600
1500
2000
2500
Iberation
Itenation
Reradon
Figure 9: Empirical validation of the more irregular loss surface of ZSL models and smoothing effect of class normalization on other datasets. Like in figure I, we observe that the gradient norms for traditional MLPs are much lower compared to basic ZSL model, but class normalization partially remedies this problem.
First, set 1 (i.c. remove scaling) and / = 0 (i.c. remove bias addition).
Second, apply this modified BN inside attribute emhedder Po on top of attributes representations H/ [hr,.... hx] across K-axis (class dimension) instead of objects representations X [x]
across B-axis (batch dimension).
It is important to note here that there are no restricting assumptions on the loss function or what data X is heing used. Thus Theorem 4.4 of Santurkar et al. (2018) is applicable to our model which means that CN smoothes its loss surface.
F.3 EMPIRICAL VALIDATION
To validate the above claim empirically, we approximate the quantity 11, but computed for all the parameters of the model instead of single layer each iteration. We do this by taking 10 random batches of size 256 from the dataset, adding parameters, then computing its norm scaled by 1/n factor to account for a small difference number of parameters (#z 0.9) between a ZSL model and non non-ZSi ZSL one. This approximates the quantity 11, but instead of approximating it around O, we approximate it around real data points since it is more practically relevant. We run the described experiment for three models:
N(0, I) noise this batch, computing the gradient of the loss with respect to
A vanilla MLP classifier, i.e. without any class attributes. For each dataset, it receives feature vector and produces logits.
2. A vanilla MLP zero-shot classifier, as described in section 3.
An MI.P zero-shot classifier with class normalization.
All three models were trained with cross-entropy loss with the optimization hyperparameters: learning of 0.0001, batch size of 256, number of iterations of 2500. They had the same numbers layers, which was equal to The results are illustrated on figures (left) and 9. As one can see, traditional MLP models indeed have more flat loss surface which is observed by small gradient norm. But class normalization helps to reduce the gap.
G CONTINUAL ZERO-SHOT LEARNING DETAILS
G.1 CZSL EXPERIMENT DETAILS
As being said, use validation sequence approach from Chaudhry (2019) find the best hyperpa- rameters for each method. We allocate the first 3 tasks to perform grid search over a fixed range. After the best hyperparameters have been found, we train the model from scratch for the rest of the tasks. The hyperparameter range for CZSL experiments are presented in Table 9 (we use the same range for all the experiments).
We train the model for 5 epochs each task with SGD optimizer. We also found it beneficial decrease learning rate after each task by a factor 0.9. This equivalent to using step-wise learning rate schedule with the number of epochs equal to the number of epochs per task. As being said, for CZSL experiments, we use an ImageNet pretranee-eetraiine ResNet- 18 model as image encoder. In contrast with ZSL, we do not keep it fixed during training. The results for our mS, mU, mH, mJA, mAUC metries, well as the forgetting measure (Lopez-Paz Paz Ranzato, 2017) presented figures 10 and 11.
27

---

Published as a conference paper at ICLR 2021
Table 9: Hyperparameters range for CZSL experiments
Sampling distribution
uniform, normal
Gradient clipping value
10, 100
Attribute embedder learning rate
0.001, 0.005
Attribute embedder momentum
0.9, 0.95
Image encoder learning rate
0.001, 0.005
46
DBC
SIB
5A5 OuP
A.008
A.DE La0
A.OON
X.CEM Dur
A-Go4
A-GEM
Seqrevix
Sequerta
Sequertial
31-
Rus-Tacc
GZSL-S
(b) GZSL-U
(c) GZSL-HI
15-
MAC
0.20
KAC
MA5
MAS
A-G2
Sequedta
Sequerix
0.15
A.004
Squrrt
Sequrttr
A-DEM
MalS-euh
MuR-Taic
Mas-Taic Day
0.10
M.ti-ax
Muti-Tast
0.20
0.:0
16
-0.30
15sk
(d) mAUSUC
(e) Joint Accuracy
(f) Forgetting Measure
Figure 10: Additional CZSL results for CUB dataset
EXC
EBC 932
MAS oum
RMG
253
6-24
6.EM
X.DEM surs
AGDN
A.DM
22.5
Seqarda
1.2
Seureii
Mat Mat-Tac
Mati-eul
Sequevis
MJt-Tait
15.
T24
(a) GZSL-S
(b) GZSL-U
(c) GZSL-H
ENC
EOC 1229
0.50
ENC
EIC
805
MAS Our
4.32N
Sequeroal
Seqandd
0.52
MuaiTss
0.32
-0.02
30
32
13
24
7pak
22
(d) mAUSUC
(e) Joint Accuracy
(1) Forgetting Measure
Figure 11: Additional CZSL. results for SUN dataset
28

---

Published as a conference paper at ICLR 2021
As clearly see, adding class normalization significandl improves the results, timesteps surpussing the multi-task bascline without ClassNorm.
G.2 ADDITIONAL CZSL METRICS
this subsection, we describe our proposed C7SL, metrics that are used to access model's performance. Subscripts "tr"/"ts" denote train/test data.
Mean Seen Accuracy (mSA). We compute GZSL-S after tasks = 1,.., I and take the average:
mSA(F) = E S(F D AS
(83)
Mean Unseen Accuracy (mUA). We compute GZSL-U after tasks = 1, I,.... T 1 (we do not compute it after task T since D= 2) and take the average:
mUA(F) T
GZSL-U(F,, DE', AA*)
(84)
Mean Harmonic Seen/Unseen Accuracy (mH). We compute GZSL-H after tasks I,..., T 1 and take the avcrage:
mH(F)
GZSL- D AA)
(85)
Mean Area Under Seen/Unseen Curve (mAUC). We compute AUSUC Chao et al. (2016) after tasks
1, 1,, T P-1 and take the average:
mAUC(F) -
Σ AUSUC(F, D A)
(86)
AUSUC is performance metric that allows to detect model's bias towards seen or unseen data and in our case it measures this in continual fashion.
Mean Joint Accuracy (mJA). On each task & compute the generalized accuracy on all the test data have for the entire prohlem:
mJA(F) 1 T Σ ACC(F, Du,
(87)
This evaluation measure allows us understand how far behind model is from the traditional supervised classifiers. A perfect model would be able to generalize on all the unseen classes from the very first task and maintain the performance on par with normal classifiers.
H WHY CANNOT WE HAVE INDEPENDENCE, ZERO-MEAN AND ASSUMPTIONS FOR ATTRIBUTES IN ZSL?
Usually, when deriving initialization scheme, people assume their random vectors have zero mean, same coordinate variance and the coordinates are independent from each other. In the paper, we stated that these are unrealistic assumptions for class attributes in ZSI. and in this section elaborate on it.
Attribute values for the common datasets need be standardized to satisfy zero-meat and unit-variance (or any other same-varte variane) assumption. But not a sensible thing do. your data does not follow normal distribution, because makes it likely to encounter skewed long-tail distribution Figure 14. In reality, this does not break our theoretical derivations, but this creates an additional optimizational issue which hampers training and that we illustrate in Table 8. This observation is also confirmed by Changpinyo al. (2016b).
the one illustrated on
we not use these assumptions but rather use zero-mean and unit-variance one (and enforce during training), than formula (6) will transform into:
Var [8.] d. Var [2.] Var [Vy]
||a|| d. Var [a] Var [Vy]= do
(88)
29

---

Published as a conference paper at ICLR 2021
Table 10: Checking how model performs when we replace AN with the standardization procedure and with the standardization procedure, accounted for 1/d factor from (88). In the latter case, the performance noticeably improved.
SUN S
CUB
AwAl
AwA2 S
U
H
U
H
U
S
H
U
H
Linear
41.0 33.4 36.8
26.9 58.1 36.8
40.6
76.6
53.1
38.4 81.7 52.2
Linear -AN
13.8 41.0
20.6
16.8
62.0
26,4
16.8
74.2 27.4
18.9 73.2 30.0
Linear -AN +//do
36.2 33.0 34.5
36.0 39.0 37.4
49.2
72.3 58.6
46.9 79.5 59.0
2-layer MLP
39.6 36.8146.9
45.0 45.9 57.3 73.8 64.5
77.1 64.5
2-layer MLP -AN
40.5 38.4 39.4 48.0 40.1 43.7 59.7 68.5 63.8
54.9 69.4 61.3
2-layer MLP -AN + 1/da
37.1 38.4 37.7 50.8 33.3 40.2 60.1 66.3 63.1 50.5 71.8 59.3
3-layer MI.P
31.4 40.4 35.3
35.3145.2
45.2 48.4 46.7 55.6 73.0 63.1
72.2 62.1
3- 3-layer MLP -AN
34.7 38.5 36.5 46.9
42.8 44.9 57.0 69.9 62.8
49.7 76.4 60.2
3-layer MLP -AN + 1/da
42.0 33.4 37.2
50.4 30.6
38.1
57.1 64.7 60.6
55.2 69.0 61.4
1000
800
0.03
0.02
0.01
AnA
Dacase:
(a) =tsstaiistics of the normality test.
(b) Corresponding p-values
Figure 12: Results of the normality test for class attributes for real-world datasets. Higher values mean that the distribution further away from a normal one. For a dataset of truly normal random variables, these values are usually in the range [0, 5]. As one of attributes does not follow a normal one, thus requires more tackling and cannot be easily converted to it.
from 12a, real-workd distribution
This means, that we need to adjust the initialization by a value 1/da to preserve the variance. This means, that we initialize the first projection matrix with the variance:
1 1
Var [ell
(89)
d= d.
Table 10, we show what happens we count this factor and we don't. one can see, just standardizing the attributes without accounting for 1/da factor leads to worse performance.
To show more rigorously that attributes do not follow normal distribution and are not independent from cach other, report statistical results:
Results a normality test based on Agostino Pearson's tests, which comes with scipy python stats library. We run it for each attribute dimension for each dataset and report the distribution of the resulted x -statisstics with the corresponding p-values on Figure 12.
Compute the distribution of absolute values of correlation coefficients between attribute dimensions. The results are presented on Figure which demonstrates that attributes dimensions are not indepen- dent between cach other in practice and thus we cannot use a common indepdence assumption when deriving the initialization sheme for ZSL. embedder.
We note, however, that attributes distribution uni-m mi-modal and, in theory, it possible to transform it into normal one (by hacking it with log/inverse/sqrtttte). but such an approach is far from being scalable. It is not scalable because transforming non non-normal distribution into normal one is tricky and is done either manually by finding proper transformation or by solving an optimization task. This is tedious to do for each dataset and thus scales poorly.
30

---

Published as a conference paper at ICLR 2021
0.6
Absolute correlation value
0.5
0.3
0.1
0.0
CUB
AwA1
AwAZ
aPY
Dataset
Figure 13: Distribution of mean absolute correlation values between different attribute dimensions. This figure shows that attributes are not independent that's why it would be unreasonable use such an assumption. If attributes would be independent from each other, that would mean that, for example, that "having black stripes" is independent from "being orange", which tigers would argue not to be natural assumption to make.
1.2
1.0
berwity
0.6
0.4
0.2
20
Attribute
Attribute
(a) Histogram of attributes for SUN dataset
(b) Histogram of attributes for AwA2 dataset
Figure 14: Histogram of standardized attribute values for SUN and AwA2. These figures demonstrate that the distribution typically long-tailed and skewed, so is far from being normal.
31


Reviews:
[{"review_id": "rJAddEroK-W", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "================\nSummary: \n\nThis paper provides a thorough analysis in the perspective of data variances on the widely used normalization tricks in the zero-shot learning research: normalize+scale and attribute normalization. It also demonstrates these tricks are not enough w.r.t. normalizing the variance in a non-linear model and propose a normalization trick to alleviate the issue. Both theoretical and empirical analysis are provided and results look convincing. Finally the authors propose a continual zero-shot learning problem scheme and illustrate some pioneering experimental results.\n\n================\nReason for my score:\n\nThere is rare work on the normalization trick in the context of zero-shot learning, although techniques like attribute normalization are widely used in practice. This paper investigates the normalization effect extensively for zero-shot learning, and provides many insightful thoughts for utilizing these tricks. The authors also evaluate the proposed class normalization with a simple implementation on benchmark datasets and show convincing results. Such work makes good contributions to the related community and hence I give my score.\n\n================\nPros:\n\n1. The paper provides both theoretical and empirical analysis on the effect of commonly used normalization tricks for zero-shot learning, in the perspective of data variances.\n2. The paper proposes a class normalization trick to alleviate the variance inflation/diminish in the non-linear model, and demonstrates its effectiveness on benchmark datasets.\n3. The empirical analysis in the paper are extensive and convincing.\n4. The paper also proposes a new framework of continual zero-shot learning.\n\n================\nCons:\n\n1. The paper didn't evaluate on another widely used benchmark dataset aPY, can author explain the reason?\n2. On CUB dataset, the proposed method has a considerably large margin to the state-of-the-art methods, in contrast to other datasets. Is there any explanation on why it is the case? Have you tried to explain this failure especially from the perspective of the proposed normalization trick?", "extra_scores": {}, "extra_text": {}}, {"review_id": "prTYPzai0EO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a theoretical justification for normalization in model training on how it affects model performance and training time. It proposes two normalization tricks: normalize + scale trick and attributes normalization trick and apply in the zero-shot image classification task. This paper also shows that two normalization tricks are not enough to variance control in a deep architecture. To address this problem, a new initialization scheme is introduced. Apart from theoretical analysis and a new initialization scheme for normalization, it extends the zero-short learning approach in a continual learning framework. This new framework is called continual zero-shot learning (CZSL) and provides corresponding evaluation metrics. The experiments for CZSL are performed in two datasets, CUB and SUN.  This paper experimentally shows the effectiveness of the initialization, normalization, and scaling trick.\n\nStrong Points:  1-  Paper is well organized and easy to follow.\n\n2-  This paper took an interesting problem and developed a compelling investigation for how normalization affects performance. The theoretical justification for the normalization tricks sounds interesting and makes sense. \n\n3-  It introduced two new techniques for normalization and shows by the theoretical justification that only normalization techniques are not sufficient for proper model training. For good model training apart from normalization tricks, it introduced an initialization scheme.\n\n4- Using normalization tricks and a new initialization scheme reduces a significant model training time compared to previous approaches. It presents training speed results for several baseline approaches.\n \n5- Innovative attempts in introducing a new ZSL problem, and several evaluation metrics are proposed for continual ZSL.\n\n\nWeaknesses:  1- This paper presents a robust analysis of normalization, initialization, and scaling trick for ZSL. It also extends ZSL in continual learning.  I appreciate the author's effort for this solid analysis.  I expect a  new proposed model from authors to make the paper more strong. \n2-  Missing comparison:  I recommend including paper [a] in the comparison table for CZSL.  Approach [a] is the first proposed baseline for continual zero-shot learning. Therefore it must be included in the comparison table. \n3- Some recent state-of-the approaches are missing in the comparison table for ZSL. Please compare it with [b],[c] models.\n\n4- Why have aPY dataset is not included in the experiments? Does this model not perform well in aPY dataset?\n5- Is it possible for this normalization and scaling tricks for other applications such as object detection, action recognition, and image retrieval? \n6-  I wonder by the training time you reported. I understand you have used quite a small neural network. Still, to have a clear view or fair comparison, you should have compared the timings with other initialization and default normalization and scaling tricks as well.\n\n[a]- Lifelong Zero-Shot Learning, by Kun Wei et al. IJCAI 2020.\n[b]- Episode-Based Prototype Generating Network for Zero-Shot Learning,  by  Yu et al. CVPR2020.\n[c]- Meta-Learning for Generalized Zero-Shot Learning, by Verma et al. AAAI 2020.\n\nRating Reason: This paper has included a well-detailed analysis and mathematical formulations for normalization, initialization about model training. But it does not propose a novel model, which limits the novelty of the model.  CZSL formulation is also already explored in [a].", "extra_scores": {}, "extra_text": {}}, {"review_id": "gcIGABdSmhV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows that normalization is critical for zero-shot learning (ZSL). In the ZSL randomization is coming from the two sources, attribute and feature.  Normalization of the two source helps to reduce the variance. The paper uses an embedding based model where normalize visual feature are projected to the attribute space and in the attribute space, cosine similarity is measured to predict the class label. Paper also extend ZSL framework to the continual learning ZSL (CZSL) setup where whole data are not present at a time; instead, data comes in the form of a task, sequentially. The author shows that normalization helps to improve the CZSL result.\n\nPositive:\n1: The normalization is important in the NN/CNN model, in the ZSL, there are two sources of information, and proper normalization is important on both source for the better result. The paper gives a theoretical justification of why normalization is important and how we can do the same for the performance gain.\n\n2: The proposed normalization shows the significant performance gain on the standard dataset for the GZSL setup (provided evaluation and code is correct). \n\n3: Recently generative model shows the SOTA result for the GZSL setting since they can synthesize the unseen class samples and easily can handle the data biasness. It is nice to see that the non-generative model shows a significant improvement using the simple method, and it is much faster to train. \n\nComment:\n1: The main concern is the result, I am unable to understand from where the exact gain is coming. Many previous works use Normalize+scale or normalization in the supervised learning or meta-learning scenario; it helps the generalization ability and smooth training and resulting in a performance gain. The performance gain using eq:[9] and [10] is expected, but [9]+[10] (CN) shows the much better result, I am unable to understand why this happens? I request the author; please explain the same.\n\n2: The proposed approach is an embedding based model, it does not generate the samples from the unseen classes, then how model overcome the data biasness towards the seen class? Generally, it observes that seen-class shows the high accuracy and unseen-class show the low accuracy; hence H-mean is very poor. The generative model can handle this scenario since they can generate data from the unseen class. The normalization technique does not help to overcome the data biasness towards the seen-class then how model handle the data biases?\n\n3: I appreciate the theoretical justification and identifying the problem in the deep model and providing the solution for that (section 3.4)\n\n4: The comparison with the few recent meta-learning based approach [a] [b] [c] for the ZSL are missing, can you show the result compared with these approaches? Also, I request the author please provide the result if the same normalization is applied with the approach [c] (it is also embedding based model in the meta-learning framework). \n\n[a] Episode-based Prototype Generating Network for Zero-Shot Learning, CVPR-20\n[b] Meta-Learning for Generalized Zero-Shot Learning\n[c] Learning to Compare: Relation Network for Few-Shot Learning, CVPR=18\n\nOverall I like the idea and contribution, but I suspect the provided result for the GZSL result in table-2, I request the author please provide the code for the AWA2 and AWA1 dataset. I will further increase the score on the successful verification of the result.\n\n4: If you don't use cosine similarity, then what is the dependency relation between the variance and weight W. I mean in the statement-1 if we don't use normalization then how variance depends on weight W? Also Var[\\hat(y)_c] is independent of W, but learning is not independent of the initialization of W, in this case, variance does not matter, proper learning and generalization are more important.\n\n5: I agree with the author that the provided evaluation metric for the CZSL is more generic and realistic. Here the model and CL procedure is not clear. What is Multi-task? What is sequential? How you ensure to overcome the catastrophic forgetting over the previous task while training the current task, The proper description is not provided, I request the author please provide the same.\n\n6: In continual learning scenario with the increase of task, the model performance is degraded. It is shown in the figure-9, and 10 (supplementary) with the increase of the task model's performance is increasing, it means that model does not forget anything and also you have backward transfer how this is possible? Maybe I misunderstood something please explain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XsYE7DtikRt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper claims to have 3 main contributions. \n\nC1: Understanding/Theory. It explains why the two tricks work in zero-shot learning (ZSL): (i) normalization + scaling in the compatibility function of the class features and the attributes, and (ii) attribute unit normalization.\n\nC2: Method. It proposes a “class normalization” scheme (Eq. 9 and 10) and Fig. 3. \nC2.1 From a “theoretical explanation” of C1 (ii), this fixes (ii) in a “deep” ZSL model.\nC2.2 It improves “smoothness” of a “irregular” loss landscape in ZSL.\n\nC3: Experiments. It demonstrates strong accuracy and training speed of the proposed approach in standard generalized ZSL. It also considers continual ZSL (Sect. 4), in which the proposed method is evaluated via mean accuracy (over timesteps) accuracy metrics and a forgetting metric.\n\n###\n\n# Strengths\n\nS1. Simple method. This is a simple feature-attribute scoring function via scaled cosine similarity (with normalization).\n\nS2. Strong empirical results (on both accuracy and training speed). See Table 2.\n\n# Weaknesses\n\nW1. Clarity\n\nThe organization of the paper is such that the reader has to refer to the appendix a lot. My biggest concern on clarity is on the “theoretical” results which are not rigorous and at times unsupported. Further, some statements/claims are not precise or clear enough for me to be convinced that the method is well-motivated and is doing what it is claimed to be doing. \n\nW2. Soundness\n\nI have a lot of concerns and questions here as I read through Sect. 3. At a high-level, I don’t see a clear connection between “improved variance control of prediction y^ or the smoothness of loss landscape” and “zero-shot learning effectiveness.” Details below. This is in part due to poor clarity.\n\nW3. Experiments\n\nIMO, if the main claim is really about the effectiveness of the two tricks and the proposed class normalization, then the experiments should go beyond one zero-shot learning starting point --- 3-layer MLP (Table 2). \n\n- If baseline methods already adopt some of these tricks, it should be made clear and see if removing these tricks lead to inferior performance.\n- If baseline methods do not adopt some of these tricks, these tricks, especially class normalization, could be applied to show improved performance. If it is difficult to apply these tricks, further explanation should be given (generally, also mention applicability of these tricks.) \n\nThis is done to some degree in the continual setting.\n\nW4. Related work\n\nAs I mentioned in W3, it is unclear which methods are linear/deep, and which methods have already benefited from existing/proposed tricks.\n\n###\n\n# Detailed comments (mainly to clarify my points about weaknesses)\n\n\n## Statement 1\n\nThe main claim for this part is that this statement provides “a theoretical understanding of the trick” and “allows to speed up the search [of the optimal value fo \\gamma].”\n\nHowever, I feel that we need further justifications on the correlation between Statement 1 (variance of y^_c, “better stability” and “the training would not stale”) and the zero-shot learning accuracy for this to be the “why normalization + scaling works.” My understanding is that the Appendix simply validates that Eq. (4) seems to hold in practice.\n\nMoreover, is the usual search region [5,10] actually effective? Do we have stronger supporting empirical evidence than the three groups of practitioners (Li et al 2019, Zhang et al. 2019, Guo et al. 2020), who may have influenced each other, used it? \n\nFinally, can the authors comment on the validity of multiple assumptions in Appendix A? To which degrees does each of them hold in practice?\n\n\n## Statement 2 and 3\n\nWhy wouldn’t the following statement in Sect. 3.3 invalidate Statement 1? \n“This may create an impression that it does not matter how we initialize the weights — normalization would undo any fluctuations. However it is not true, because it is still important how the signal flows, i.e. for an unnormalized and unscaled logit value”\n\nIt is unclear (at least not from the beginning) why understanding attribute normalization has to do with initialization of the weights.\n\nSimilar to my comments to Statement 1, why should we believe that the explanation in Sect. 3.3 and Sect. 3.4 is the reason for zero-shot learning effectiveness? In particular, the authors again claim that the main bottleneck in improving zero-shot learning is “variance control” (the end of Sect. 3.3).\n\nI also have a hard time understanding some statements in Appendix H, which is needed to motivate the following statement in Sect. 3.3: “And these assumptions are safe to assume only for z but not for a_c, because they do not hold for the standard datasets (see Appendix H).”\nH1: Would this statement still be true after we transform a_c with an MLP?\nH2: Why is it not “a sensible thing to do” if we just want zero mean and unit variance? \nH3: Why is “such an approach far from being scalable”? \nH4: What if these are things like word embeddings?\nH5: Fig. 12 and Fig. 13 are not explained.\nH6: Histograms in Fig. 13 look quite normal.\n\nHow useful is Statement 2? Why is the connection with Xavier initialization important?\n\nWhy is “preserving the variance between z and y~” in Statement 3  important for zero-shot learning?\n\n\n## Improved smoothness\n\nThe claim “improved smoothness” at the end of Sect. 3 and Appendix F is really hard to understand.\nF1: How do the authors define “irregular loss surface”?\nF2: “Santurkar et al. (2018) showed that batch-wise standardization procedure decreases the Lipschitz constant of a model, which suggests that our class-wise standardization will provide the same impact.” This is not very precise and seems unsupported. Please make it clear how. If this is a hypothesis, please make it clear.\n\nSimilarly to my comments to Statement 1-3, how is improved smoothness related to zero-shot learning effectiveness?  \n\n\n## Other more minor comments\n1. Abstract: Are the authors the one to “generalize ZSL to a broader problem”? Please tone down the claim if not.\n2. After Eq. (2): Why does attribute normalization look “inconsiderable” (possibly this is not the right word?) or why is it “surprising” that this is preferred in practice? Don’t most zero-shot learning methods use this (see for example Table 4 in [A])?\n3. Suggestions for references for attribute normalization. This can be improved; I can trace this back to much earlier work such as [A] and [B] (though I think this fact is stated more explicitly in [A]).\n4. Under Table 1 “These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear (see Figure 1), but they fail when we use a multi-layer architecture.”: Could the authors provide a reference to evidence to support this? I think it is also important to provide a clear statement of what separates a “linear” or “multi-layer” model.\n5. The first paragraph of Sect. 3: Could you provide references for motivations for different activation functions? Further, It is unclear that all of them perform normalization.\n6. The second paragraph of Sect. 3: What exactly limits “the tools” for zero-shot learning vs. supervised learning? Further, it would also be nice to separate traditional supervised learning where classes are balanced and imbalanced; see, e.g., [C].\n7. What is the closest existing zero-shot model to the one the authors describe in Sect. 3.1? Why is the described model considered/selected? \n\n[A] Synthesized Classifiers for Zero-Shot Learning\n\n[B] Zero-Shot Learning by Convex Combination of Semantic Embeddings\n\n[C] Class-Balanced Loss Based on Effective Number of Samples\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper got mixed reviews. One for reject and three for acceptance. The reviewers and authors have extensive discussion. Authors also provided additional experiments for further clarifying some questions from the reviewers. The paper has some clarify issue in the theoretical justification part as pointed out by AR1. Authors should extensively improve this part or revise the statement. However, the method proposed in this paper is simple and the results are indeed good. This paper is valuable and should be shared within the community to advance research on ZSL. Therefore, AC recommends acceptance. 
Decision: Accept (Poster)