Class Normalization for (Continual)? Generalized Zero-Shot Learning

1 INTRODUCTION . Zero-shot learning ( ZSL ) aims to understand new concepts based on their semantic descriptions instead of numerous input-output learning pairs . It is a key element of human intelligence and our best machines still struggle to master it ( Ferrari & Zisserman , 2008 ; Lampert et al. , 2009 ; Xian et al. , 2018a ) . Normalization techniques like batch/layer/group normalization ( Ioffe & Szegedy , 2015 ; Ba et al. , 2016 ; Wu & He , 2018 ) are now a common and important practice of modern deep learning . But despite their popularity in traditional supervised training , not much is explored in the realm of zero-shot learning , which motivated us to study and investigate normalization in ZSL models . We start by analyzing two ubiquitous tricks employed by ZSL and representation learning practitioners : normalize+scale ( NS ) and attributes normalization ( AN ) ( Bell et al. , 2016 ; Zhang et al. , 2019 ; Guo et al. , 2020 ; Chaudhry et al. , 2019 ) . Their dramatic influence on performance can be observed from Table 1 . When these two tricks are employed , a vanilla MLP model , described in Sec 3.1 , can outperform some recent sophisticated ZSL methods . Normalize+scale ( NS ) changes logits computation from usual dot-product to scaled cosine similarity : ŷc = z > pc =⇒ ŷc = ( γ · z ‖z‖2 ) > ( γ · pc ‖pc‖ ) ( 1 ) where z is an image feature , pc is c-th class prototype and γ is a hyperparameter , usually picked from [ 5 , 10 ] interval ( Li et al. , 2019 ; Zhang et al. , 2019 ) . Scaling by γ is equivalent to setting a high temperature of γ2 in softmax . In Sec . 3.2 , we theoretically justify the need for this trick and explain why the value of γ must be so high . Attributes Normalization ( AN ) technique simply divides class attributes by their L2 norms : ac 7−→ ac/‖ac‖2 ( 2 ) While this may look inconsiderable , it is surprising to see it being preferred in practice ( Li et al. , 2019 ; Narayan et al. , 2020 ; Chaudhry et al. , 2019 ) instead of the traditional zero-mean and unit-variance data standardization ( Glorot & Bengio , 2010 ) . In Sec 3 , we show that it helps in normalizing signal ’ s variance in and ablate its importance in Table 1 and Appx D. These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear ( see Figure 1 ) , but they fail when we use a multi-layer architecture . To remedy this issue , we introduce Class Normalization ( CN ) : a novel normalization scheme , which is based on a different initialization and a class-wise standardization transform . Modern ZSL methods either utilize sophisticated architectural design like training generative models ( Narayan et al. , 2020 ; Felix et al. , 2018 ) or use heavy optimization schemes like episode-based training ( Yu et al. , 2020 ; Li et al. , 2019 ) . In contrast , we show that simply adding Class Normalization on top of a vanilla MLP is enough to set new state-of-the-art results on several standard ZSL datasets ( see Table 2 ) . Moreover , since it is optimized with plain gradient descent without any bells and whistles , training time for us takes 50-100 times less and runs in about 1 minute . We also demonstrate that many ZSL models tend to have more irregular loss surface compared to traditional supervised learning classifiers and apply the results of Santurkar et al . ( 2018 ) to show that our CN partially remedies the issue . We discuss and empirically validate this in Sec 3.5 and Appx F. Apart from the theoretical exposition and a new normalization scheme , we also propose a broader ZSL setup : continual zero-shot learning ( CZSL ) . Continual learning ( CL ) is an ability to acquire new knowledge without forgetting ( e.g . ( Kirkpatrick et al. , 2017 ) ) , which is scarcely investigated in ZSL . We develop the ideas of lifelong learning with class attributes , originally proposed by Chaudhry et al . ( 2019 ) and extended by Wei et al . ( 2020a ) , propose several principled metrics for it and test several classical CL methods in this new setup . 2 RELATED WORK . Zero-shot learning . Zero-shot learning ( ZSL ) aims at understand example of unseen classes from their language or semantic descriptions . Earlier ZSL methods directly predict attribute confidence from images to facilitate zero-shot recognition ( e.g. , Lampert et al . ( 2009 ) ; Farhadi et al . ( 2009 ) ; Lampert et al . ( 2013b ) ) . Recent ZSL methods for image classification can be categorized into two groups : generative-based and embedding-based . The main goal for generative-based approaches is to build a conditional generative model ( e.g. , GANs Goodfellow et al . ( 2014 ) and VAEs ( Kingma & Welling , 2014 ) ) to synthesize visual generations conditioned on class descriptors ( e.g. , Xian et al . ( 2018b ) ; Zhu et al . ( 2018 ) ; Elhoseiny & Elfeki ( 2019 ) ; Guo et al . ( 2017 ) ; Guo et al . ( 2017 ) ; Kumar Verma et al . ( 2018 ) ) . At test time , the trained generator is expected to produce synthetic/fake data of unseen classes given its semantic descriptor . The fake data is then used to train a traditional classifier or to perform a simple kNN-classification on the test images . Embedding-based approaches learn a mapping that projects semantic attributes and images into a common space where the distance between a class projection and the corresponding images is minimized ( e.g , Romera-Paredes & Torr ( 2015 ) ; Frome et al . ( 2013 ) ; Lei Ba et al . ( 2015 ) ; Akata et al . ( 2016a ) ; Zhang et al . ( 2017 ) ; Akata et al . ( 2015 ; 2016b ) ) . One question that arises is what space to choose to project the attributes or images to . Previous works projected images to the semantic space ( Elhoseiny et al. , 2013 ; Frome et al. , 2013 ; Lampert et al. , 2013a ) or some common space ( Zhang & Saligrama , 2015 ; Akata et al. , 2015 ) , but our approach follows the idea of Zhang et al . ( 2016 ) ; Li et al . ( 2019 ) that shows that projecting attributes to the image space reduces the bias towards seen data . Normalize+scale and attributes normalization . It was observed both in ZSL ( e.g. , Li et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Bell et al . ( 2016 ) ) and representation learning ( e.g. , Sohn ( 2016 ) ; Guo et al . ( 2020 ) ; Ye et al . ( 2020 ) ) fields that normalize+scale ( i.e . ( 1 ) ) and attributes normalization ( i.e . ( 2 ) ) tend to significantly improve the performance of a learning system . In the literature , these two techniques lack rigorous motivation and are usually introduced as practical heuristics that aid training ( Changpinyo et al. , 2017 ; Zhang et al. , 2019 ; 2021 ) . One of the earliest works that employ attributes normalization was done by ( Norouzi et al. , 2013 ) , and in ( Changpinyo et al. , 2016a ) authors also ablate its importance . The main consumers of normalize+scale trick had been similarity learning algorithms , which employ it to refine the distance metric between the representations ( Bellet et al. , 2013 ; Guo et al. , 2020 ; Shi et al. , 2020 ) . Luo et al . ( 2018 ) proposed to use cosine similarity in the final output projection matrix as a normalization procedure , but didn ’ t incorporate any analysis on how it affects the variance . They also didn ’ t use the scaling which our experiments in Table 5 show to be crucial . Gidaris & Komodakis ( 2018 ) demonstrated a greatly superior performance of an NS-enriched model compared to a dot-product based one in their setup where the classifying matrix is constructed dynamically . Li et al . ( 2019 ) motivated their usage of NS by variance reduction , but didn ’ t elaborate on this in their subsequent analysis . Chen et al . ( 2020 ) related the use of the normalized temperature-scaled cross entropy loss ( NT-Xent ) to different weighting of negative examples in contrastive learning framework . Overall , to the best of our knowledge , there is no precise understanding of the influence of these two tricks on the optimization process and benefits they provide . Initialization schemes . In the seminal work , Xavier ’ s init Glorot & Bengio ( 2010 ) , the authors showed how to preserve the variance during a forward pass . He et al . ( 2015 ) applied a similar analysis but taking ReLU nonlinearities into account . There is also a growing interest in two-step Jia et al . ( 2014 ) , data-dependent Krähenbühl et al . ( 2015 ) , and orthogonal Hu et al . ( 2020 ) initialization schemes . However , the importance of a good initialization for multi-modal embedding functions like attribute embedding is less studied and not well understood . We propose a proper initialization scheme based on a different initialization variance and a dynamic standardization layer . Our variance analyzis is similar in nature to Chang et al . ( 2020 ) since attribute embedder may be seen as a hypernetwork ( Ha et al. , 2016 ) that outputs a linear classifier . But the exact embedding transformation is different from a hypernetwork since it has matrix-wise input and in our derivations we have to use more loose assumptions about attributes distribution ( see Sec 3 and Appx H ) . Normalization techniques . A closely related branch of research is the development of normalization layers for deep neural networks ( Ioffe & Szegedy , 2015 ) since they also influence a signal ’ s variance . BatchNorm , being the most popular one , normalizes the location and scale of activations . It is applied in a batch-wise fashion and that ’ s why its performance is highly dependent on batch size ( Singh & Krishnan , 2020 ) . That ’ s why several normalization techniques have been proposed to eliminate the batch-size dependecy ( Wu & He , 2018 ; Ba et al. , 2016 ; Singh & Krishnan , 2020 ) . The proposed class normalization is very similar to a standardization procedure which underlies BatchNorm , but it is applied class-wise in the attribute embedder . This also makes it independent from the batch size . Continual zero-shot learning . We introduce continual zero-shot learning : a new benchmark for ZSL agents that is inspired by continual learning literature ( e.g. , Kirkpatrick et al . ( 2017 ) ) . It is a development of the scenario proposed in Chaudhry et al . ( 2019 ) , but authors there focused on ZSL performance only a single task ahead , while in our case we consider the performance on all seen ( previous tasks ) and all unseen data ( future tasks ) . This also contrasts our work to the very recent work by Wei et al . ( 2020b ) , where a sequence of seen class splits of existing ZSL benchmsks is trained and the zero-shot performance is reported for every task individually at test time . In contrast , for our setup , the label space is not restricted and covers the spectrum of all previous tasks ( seen tasks so far ) , and future tasks ( unseen tasks so far ) . Due to this difference , we need to introduce a set of new metrics and benchmarks to measure this continual generalized ZSL skill over time . From the lifelong learning perspective , the idea to consider all the processed data to evaluate the model is not new and was previously explored by Elhoseiny et al . ( 2018 ) ; van de Ven & Tolias ( 2019 ) . It lies in contrast with the common practice of providing task identity at test time , which limits the prediction space for a model , making the problem easier ( Kirkpatrick et al. , 2017 ; Aljundi et al. , 2017 ) . In Isele et al . ( 2016 ) ; Lopez-Paz & Ranzato ( 2017 ) authors motivate the use of task descriptors for zero-shot knowledge transfer , but in our work we consider class descriptors instead . We defined CZSL as a continual version of generalized-ZSL which allows us to naturally extend all the existing ZSL metrics Xian et al . ( 2018a ) ; Chao et al . ( 2016 ) to our new continual setup .

Summary: This paper presents a theoretical justification for normalization in model training on how it affects model performance and training time. It proposes two normalization tricks: normalize + scale trick and attributes normalization trick and apply in the zero-shot image classification task. This paper also shows that two normalization tricks are not enough to variance control in a deep architecture. To address this problem, a new initialization scheme is introduced. Apart from theoretical analysis and a new initialization scheme for normalization, it extends the zero-short learning approach in a continual learning framework. This new framework is called continual zero-shot learning (CZSL) and provides corresponding evaluation metrics. The experiments for CZSL are performed in two datasets, CUB and SUN.  This paper experimentally shows the effectiveness of the initialization, normalization, and scaling trick.