Class Normalization for (Continual)? Generalized Zero-Shot Learning

1 INTRODUCTION . Zero-shot learning ( ZSL ) aims to understand new concepts based on their semantic descriptions instead of numerous input-output learning pairs . It is a key element of human intelligence and our best machines still struggle to master it ( Ferrari & Zisserman , 2008 ; Lampert et al. , 2009 ; Xian et al. , 2018a ) . Normalization techniques like batch/layer/group normalization ( Ioffe & Szegedy , 2015 ; Ba et al. , 2016 ; Wu & He , 2018 ) are now a common and important practice of modern deep learning . But despite their popularity in traditional supervised training , not much is explored in the realm of zero-shot learning , which motivated us to study and investigate normalization in ZSL models . We start by analyzing two ubiquitous tricks employed by ZSL and representation learning practitioners : normalize+scale ( NS ) and attributes normalization ( AN ) ( Bell et al. , 2016 ; Zhang et al. , 2019 ; Guo et al. , 2020 ; Chaudhry et al. , 2019 ) . Their dramatic influence on performance can be observed from Table 1 . When these two tricks are employed , a vanilla MLP model , described in Sec 3.1 , can outperform some recent sophisticated ZSL methods . Normalize+scale ( NS ) changes logits computation from usual dot-product to scaled cosine similarity : ŷc = z > pc =⇒ ŷc = ( γ · z ‖z‖2 ) > ( γ · pc ‖pc‖ ) ( 1 ) where z is an image feature , pc is c-th class prototype and γ is a hyperparameter , usually picked from [ 5 , 10 ] interval ( Li et al. , 2019 ; Zhang et al. , 2019 ) . Scaling by γ is equivalent to setting a high temperature of γ2 in softmax . In Sec . 3.2 , we theoretically justify the need for this trick and explain why the value of γ must be so high . Attributes Normalization ( AN ) technique simply divides class attributes by their L2 norms : ac 7−→ ac/‖ac‖2 ( 2 ) While this may look inconsiderable , it is surprising to see it being preferred in practice ( Li et al. , 2019 ; Narayan et al. , 2020 ; Chaudhry et al. , 2019 ) instead of the traditional zero-mean and unit-variance data standardization ( Glorot & Bengio , 2010 ) . In Sec 3 , we show that it helps in normalizing signal ’ s variance in and ablate its importance in Table 1 and Appx D. These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear ( see Figure 1 ) , but they fail when we use a multi-layer architecture . To remedy this issue , we introduce Class Normalization ( CN ) : a novel normalization scheme , which is based on a different initialization and a class-wise standardization transform . Modern ZSL methods either utilize sophisticated architectural design like training generative models ( Narayan et al. , 2020 ; Felix et al. , 2018 ) or use heavy optimization schemes like episode-based training ( Yu et al. , 2020 ; Li et al. , 2019 ) . In contrast , we show that simply adding Class Normalization on top of a vanilla MLP is enough to set new state-of-the-art results on several standard ZSL datasets ( see Table 2 ) . Moreover , since it is optimized with plain gradient descent without any bells and whistles , training time for us takes 50-100 times less and runs in about 1 minute . We also demonstrate that many ZSL models tend to have more irregular loss surface compared to traditional supervised learning classifiers and apply the results of Santurkar et al . ( 2018 ) to show that our CN partially remedies the issue . We discuss and empirically validate this in Sec 3.5 and Appx F. Apart from the theoretical exposition and a new normalization scheme , we also propose a broader ZSL setup : continual zero-shot learning ( CZSL ) . Continual learning ( CL ) is an ability to acquire new knowledge without forgetting ( e.g . ( Kirkpatrick et al. , 2017 ) ) , which is scarcely investigated in ZSL . We develop the ideas of lifelong learning with class attributes , originally proposed by Chaudhry et al . ( 2019 ) and extended by Wei et al . ( 2020a ) , propose several principled metrics for it and test several classical CL methods in this new setup . 2 RELATED WORK . Zero-shot learning . Zero-shot learning ( ZSL ) aims at understand example of unseen classes from their language or semantic descriptions . Earlier ZSL methods directly predict attribute confidence from images to facilitate zero-shot recognition ( e.g. , Lampert et al . ( 2009 ) ; Farhadi et al . ( 2009 ) ; Lampert et al . ( 2013b ) ) . Recent ZSL methods for image classification can be categorized into two groups : generative-based and embedding-based . The main goal for generative-based approaches is to build a conditional generative model ( e.g. , GANs Goodfellow et al . ( 2014 ) and VAEs ( Kingma & Welling , 2014 ) ) to synthesize visual generations conditioned on class descriptors ( e.g. , Xian et al . ( 2018b ) ; Zhu et al . ( 2018 ) ; Elhoseiny & Elfeki ( 2019 ) ; Guo et al . ( 2017 ) ; Guo et al . ( 2017 ) ; Kumar Verma et al . ( 2018 ) ) . At test time , the trained generator is expected to produce synthetic/fake data of unseen classes given its semantic descriptor . The fake data is then used to train a traditional classifier or to perform a simple kNN-classification on the test images . Embedding-based approaches learn a mapping that projects semantic attributes and images into a common space where the distance between a class projection and the corresponding images is minimized ( e.g , Romera-Paredes & Torr ( 2015 ) ; Frome et al . ( 2013 ) ; Lei Ba et al . ( 2015 ) ; Akata et al . ( 2016a ) ; Zhang et al . ( 2017 ) ; Akata et al . ( 2015 ; 2016b ) ) . One question that arises is what space to choose to project the attributes or images to . Previous works projected images to the semantic space ( Elhoseiny et al. , 2013 ; Frome et al. , 2013 ; Lampert et al. , 2013a ) or some common space ( Zhang & Saligrama , 2015 ; Akata et al. , 2015 ) , but our approach follows the idea of Zhang et al . ( 2016 ) ; Li et al . ( 2019 ) that shows that projecting attributes to the image space reduces the bias towards seen data . Normalize+scale and attributes normalization . It was observed both in ZSL ( e.g. , Li et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Bell et al . ( 2016 ) ) and representation learning ( e.g. , Sohn ( 2016 ) ; Guo et al . ( 2020 ) ; Ye et al . ( 2020 ) ) fields that normalize+scale ( i.e . ( 1 ) ) and attributes normalization ( i.e . ( 2 ) ) tend to significantly improve the performance of a learning system . In the literature , these two techniques lack rigorous motivation and are usually introduced as practical heuristics that aid training ( Changpinyo et al. , 2017 ; Zhang et al. , 2019 ; 2021 ) . One of the earliest works that employ attributes normalization was done by ( Norouzi et al. , 2013 ) , and in ( Changpinyo et al. , 2016a ) authors also ablate its importance . The main consumers of normalize+scale trick had been similarity learning algorithms , which employ it to refine the distance metric between the representations ( Bellet et al. , 2013 ; Guo et al. , 2020 ; Shi et al. , 2020 ) . Luo et al . ( 2018 ) proposed to use cosine similarity in the final output projection matrix as a normalization procedure , but didn ’ t incorporate any analysis on how it affects the variance . They also didn ’ t use the scaling which our experiments in Table 5 show to be crucial . Gidaris & Komodakis ( 2018 ) demonstrated a greatly superior performance of an NS-enriched model compared to a dot-product based one in their setup where the classifying matrix is constructed dynamically . Li et al . ( 2019 ) motivated their usage of NS by variance reduction , but didn ’ t elaborate on this in their subsequent analysis . Chen et al . ( 2020 ) related the use of the normalized temperature-scaled cross entropy loss ( NT-Xent ) to different weighting of negative examples in contrastive learning framework . Overall , to the best of our knowledge , there is no precise understanding of the influence of these two tricks on the optimization process and benefits they provide . Initialization schemes . In the seminal work , Xavier ’ s init Glorot & Bengio ( 2010 ) , the authors showed how to preserve the variance during a forward pass . He et al . ( 2015 ) applied a similar analysis but taking ReLU nonlinearities into account . There is also a growing interest in two-step Jia et al . ( 2014 ) , data-dependent Krähenbühl et al . ( 2015 ) , and orthogonal Hu et al . ( 2020 ) initialization schemes . However , the importance of a good initialization for multi-modal embedding functions like attribute embedding is less studied and not well understood . We propose a proper initialization scheme based on a different initialization variance and a dynamic standardization layer . Our variance analyzis is similar in nature to Chang et al . ( 2020 ) since attribute embedder may be seen as a hypernetwork ( Ha et al. , 2016 ) that outputs a linear classifier . But the exact embedding transformation is different from a hypernetwork since it has matrix-wise input and in our derivations we have to use more loose assumptions about attributes distribution ( see Sec 3 and Appx H ) . Normalization techniques . A closely related branch of research is the development of normalization layers for deep neural networks ( Ioffe & Szegedy , 2015 ) since they also influence a signal ’ s variance . BatchNorm , being the most popular one , normalizes the location and scale of activations . It is applied in a batch-wise fashion and that ’ s why its performance is highly dependent on batch size ( Singh & Krishnan , 2020 ) . That ’ s why several normalization techniques have been proposed to eliminate the batch-size dependecy ( Wu & He , 2018 ; Ba et al. , 2016 ; Singh & Krishnan , 2020 ) . The proposed class normalization is very similar to a standardization procedure which underlies BatchNorm , but it is applied class-wise in the attribute embedder . This also makes it independent from the batch size . Continual zero-shot learning . We introduce continual zero-shot learning : a new benchmark for ZSL agents that is inspired by continual learning literature ( e.g. , Kirkpatrick et al . ( 2017 ) ) . It is a development of the scenario proposed in Chaudhry et al . ( 2019 ) , but authors there focused on ZSL performance only a single task ahead , while in our case we consider the performance on all seen ( previous tasks ) and all unseen data ( future tasks ) . This also contrasts our work to the very recent work by Wei et al . ( 2020b ) , where a sequence of seen class splits of existing ZSL benchmsks is trained and the zero-shot performance is reported for every task individually at test time . In contrast , for our setup , the label space is not restricted and covers the spectrum of all previous tasks ( seen tasks so far ) , and future tasks ( unseen tasks so far ) . Due to this difference , we need to introduce a set of new metrics and benchmarks to measure this continual generalized ZSL skill over time . From the lifelong learning perspective , the idea to consider all the processed data to evaluate the model is not new and was previously explored by Elhoseiny et al . ( 2018 ) ; van de Ven & Tolias ( 2019 ) . It lies in contrast with the common practice of providing task identity at test time , which limits the prediction space for a model , making the problem easier ( Kirkpatrick et al. , 2017 ; Aljundi et al. , 2017 ) . In Isele et al . ( 2016 ) ; Lopez-Paz & Ranzato ( 2017 ) authors motivate the use of task descriptors for zero-shot knowledge transfer , but in our work we consider class descriptors instead . We defined CZSL as a continual version of generalized-ZSL which allows us to naturally extend all the existing ZSL metrics Xian et al . ( 2018a ) ; Chao et al . ( 2016 ) to our new continual setup .

Summary: This paper provides a thorough analysis in the perspective of data variances on the widely used normalization tricks in the zero-shot learning research: normalize+scale and attribute normalization. It also demonstrates these tricks are not enough w.r.t. normalizing the variance in a non-linear model and propose a normalization trick to alleviate the issue. Both theoretical and empirical analysis are provided and results look convincing. Finally the authors propose a continual zero-shot learning problem scheme and illustrate some pioneering experimental results.