Alpha Net: Adaptation with Composition in Classifier Space

Deep learning classification models typically train poorly on classes with small numbers of examples . Motivated by the human ability to solve this task , models have been developed that transfer knowledge from classes with many examples to learn classes with few examples . Critically , the majority of these models transfer knowledge within model feature space . In this work , we demonstrate that transferring knowledge within classifier space is more effective and efficient . Specifically , by linearly combining strong nearest neighbor classifiers along with a weak classifier , we are able to compose a stronger classifier . Uniquely , our model can be implemented on top of any existing classification model that includes a classifier layer . We showcase the success of our approach in the task of long-tailed recognition , whereby the classes with few examples , otherwise known as the “ tail ” classes , suffer the most in performance and are the most challenging classes to learn . Using classifier-level knowledge transfer , we are able to drastically improve - by a margin as high as 10.5 % - the state-of-the-art performance on the “ tail ” categories . 1 INTRODUCTION . The computer vision field has made rapid progress in the area of object recognition due to several factors : complex architectures , larger compute power , more data , and better learning strategies . However , the standard method to train recognition models on new classes still relies on training using large sets of examples . This dependence on large scale data has made learning from few samples a natural challenge . Highlighting this point , new tasks such as low-shot learning and longtailed learning , have recently become common within computer vision . Many approaches to learning from small numbers of examples are inspired by human learning . In particular , humans are able to learn new concepts quickly and efficiently over only a few samples . The overarching theory is that humans are able to transfer their knowledge from previous experiences to bootstrap their new learning task ( Lake et al. , 2017 ; 2015 ; Gopnik & Sobel , 2000 ) . Inherent in these remarkable capabilities are two related questions : what knowledge is being transferred and how is this knowledge being transferred ? Within computer vision , recent low-shot learning and long-tailed recognition models answer these questions by treating visual “ representations ” as the knowledge structures that are being transferred . As such , the knowledge transfer methods implemented in these models transfer learned features from known classes learned from large data to the learning of new classes with low data ( Liu et al. , 2019 ; Yin et al. , 2019 ) . These models exemplify the broader assumption that , in both human and computer vision , knowledge transfer occurs within model representation and feature space ( Lake et al. , 2015 ) . In contrast , we claim that previously learned information is more concisely captured in classifier space . This inference is based on the fact that sample representation is unique to that sample , but classifiers are fitted for all the samples in a given class . The success of working within classifier space to improve certain classifiers has been established in several papers ( Elhoseiny et al. , 2013 ; Qi et al. , 2018 ) , where the models are able to directly predict classifiers from features or create new models entirely by learning other models . Other non-deep learning models use classifiers learnt with abundant data to generate novel classifiers ( Aytar & Zisserman , 2011 ; 2012 ) . ΘW a Figure 1 : A classifier space depicting how Alpha Net adaptively adjusts weak classifiers through nearest neighbor compositions . Despite these successes , the concept of learning within classifier space is not as common in deep learning models . We suggest that transfer learning can , likewise , be implemented in the classifier space of deep learning models . Specifically , we combine known , strong classifiers ( i.e. , learned with large datasets ) with weak classifiers ( i.e. , learned with small datasets ) to improve our weak classifiers . Our classifier space method is illustrated in Figure 1 . In this toy example demonstrating our method , we are given n classifiers Ci trained with large data and a weak classifier a , which was trained for a class with very few samples . Our goal is to combine the most relevant strong classifiers to adaptively adjust and improve a . Our method implements this approach in the simplest way possible . To select the most effective strong classifiers to combine , we take the target ’ s a nearest neighbor classifiers . Given the appropriate nearest neighbor classifiers , the challenge becomes how to combine these strong classifiers with the target weak classifier so as to improve its performance given no further samples . We address this challenge using what we view as the most natural approach - creating a new classifier by linearly combining the nearest neighbor classifiers and the original weak classifier . Embedded in this solution is the further challenge of choosing a strategy for computing the combination coefficients . We propose learning the coefficients of the linear combination through another neural network , which we refer to as “ Alpha Net ” . As compared to many other approaches to learning from small numbers of examples , our methodology has three characteristics : 1 . Our approach can be implemented on top of any architecture . This is because the Alpha Net does not need to re-learn representations and only operates within classifier space to improve weak classifiers . As a result , our approach is agnostic to the type of architecture used to learn the classifiers ; it merely provides a systematic method for combining classifiers . 2 . Our approach demonstrates the importance of combining not only the most relevant classifiers but also the original classifier . In the absence of the original classifier , any combination of classifiers becomes a possible solution , without being constrained by the initial classifier . 3 . Our approach creates for every target class a completely different set of linear coefficients for our new classifier composition . In this manner , we are learning our coefficients in a more adaptive way , which is extremely difficult to achieve through classical methods . To illustrate the efficacy of our method , we apply it to the task of long-tailed recognition . “ Longtailed ” refers to a data distribution in which there is a realistic distribution of classes with many examples ( head classes ) and classes with few examples ( tail classes ) . We compare our Alpha Net method to recent state-of-the-art models Kang et al . ( 2020 ) on two long-tailed datasets : ImageNetLT and Places-LT . Critically , we are able to improve the tail classifiers accuracy by as much as 10.5 % . 2 RELATED WORK . Creating , modifying , and learning model weights are concepts that are seen in many earlier models . In particular , these concepts appear frequently in transfer learning , meta-learning , low-shot learning , and long-tailed learning . Classifier Creation The process of creating new classifiers is captured within meta-learning concepts such as learning-to-learn , transfer learning , and multi-task learning ( Thrun & Pratt , 2012 ; Schmidhuber et al. , 1997 ; Pan & Yang , 2009 ; Caruana , 1997 ; Santoro et al. , 2016 ) . These approaches generalize to novel tasks by learning shared information from a set of related tasks . Many studies find that shared information is embedded within model weights , and , thus , aim to learn structure within learned models to directly modify another network ’ s weights ( Schmidhuber , 1987 ; 1992 ; 1993 ; Bertinetto et al. , 2016 ; Ha et al. , 2016 ; Finn et al. , 2017 ; Rebuffi et al. , 2017 ; Sinha et al. , 2017 ; Munkhdalai & Yu , 2017 ) . Other studies go even further and instead of modifying networks , they create entirely new networks from training samples only ( Socher et al. , 2013 ; Lei Ba et al. , 2015 ; Noh et al. , 2016 ) . However , our method only combines existing classifiers , without having to create new classifiers or networks from scratch . Classifier or Feature Composition In various classical approaches , there has been work that learns better embedding spaces for image annotation ( Weston et al. , 2010 ) or uses classification scores as useful features ( Wang et al. , 2009 ) . However , they do not attempt to compose classifiers nor do they address the long-tail problem . Within non-deep methods in classic transfer learning , there have been attempts of using and combining SVMs . In Tsochantaridis et al . ( 2005 ) , SVMs are trained per object instance and a hierarchical structure is required for combination in the datasets of interest . Such a structure is typically not guaranteed nor provided in our long-tail datasets . Additional SVM work uses regularized minimization to learn the coefficients necessary to combine patches from other classifiers ( Aytar & Zisserman , 2012 ) . While conceptually similar to Alpha Net , our method has an additional advantage of learning the compositional coefficients in a more adaptive way . Specifically , different novel classes will have their own sets of alphas , and similar novel classes will naturally have similar sets of alphas . Learning such varying sets of alphas is difficult in previous classical approaches . They either learn a fixed set of alphas for all of the novel classes or are forced to introduce more complex group sparsity-like constraints . Finally , in zero-shot learning there exist methods which compose classifiers of known visual concepts to learn a completely new classifier ( Elhoseiny et al. , 2013 ; Misra et al. , 2017 ; Lei Ba et al. , 2015 ; Changpinyo et al. , 2016 ) . However , such composition is often guided by additional attribute supervision or textual description , which Alpha Net does not depend on . Learning Transformations Between Models and Classes Other studies have demonstrated different ways of learning transformations to modify model weights in an attempt to learn these transformations during stochastic gradient descent ( SGD ) optimization ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ) . Additionally , Wang & Hebert ( 2016 ) empirically show the existence of a generic nonlinear transformation from small-sample to large-sample models for different types of feature spaces and classifier models . Finally , Du et al . ( 2017 ) provide theoretical guarantees on performance when one learns the transformation going from the source function to a related target function . In our work , we are also inferring that our target classifier is a transformation from a set of source classifiers . Low-shot Learning Meta-learning , transfer learning , and learning-to-learn are frequently applied to the domain of low-shot learning ( Fei-Fei et al. , 2006 ; Koch et al. , 2015 ; Lake et al. , 2015 ; Santoro et al. , 2016 ; Wang & Hebert , 2016 ; Li & Hoiem , 2017 ; Hariharan & Girshick , 2017 ; Bromley et al. , 1994 ; Koch et al. , 2015 ; Snell et al. , 2017 ; George et al. , 2017 ; Wang et al. , 2017 ; Akata et al. , 2015 ) . Many prior studies have attempted to transfer knowledge from tasks with abundant data to completely novel tasks ( Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ; Bertinetto et al. , 2016 ) . However , the explicit nature of low-shot learning consisting of tasks with small fixed samples means that these approaches do not generalize well beyond the arbitrary few tasks . This is indeed a significant problem as the visual world very clearly involves a wide set of tasks with continuously varying amounts of information . Long-tailed Learning The restrictions on low-shot learning have directly led to a new paradigm referred to as “ long-tailed ” learning , where data samples are continuously decreasing and the data distribution closely models that of the visual world . Recent work has approached long-tailed recognition by re-balancing the samples in different stages of model training ( Cao et al. , 2019 ) . One other study attempts to transfer features from common classes to rare classes ( Liu et al. , 2019 ) or transfer intra-class variance ( Yin et al. , 2019 ) . However , both these approaches to knowledge transfer require complex architectures , such as a specialized attention mechanism and memory models as in Liu et al . ( 2019 ) . While most studies have largely focused on representation space transferability , recent work explores the potential of operating in classifier space ( Kang et al. , 2020 ) . Results suggest that decoupling model representation learning and classifier learning is actually a more efficient way to approach long-tailed learning . Specifically , normalizing classifiers and adjusting classifiers only with re-sampling strategies are able to achieve good performance . Their success in working only with classifiers supports our general concept that combining strong classifiers is a natural and direct way to improve upon weak classifiers .

Summary: This paper addresses the well-known long-tail classification problem. The argument made here is that most of the existing methods attempt to transfer knowledge in the feature space, which is true. Based on this motivation, the paper proposes a method to do the knowledge transfer in the model space instead. The idea is to apply K-NN method in the model space to pick up a group of strong classifiers trained on the head classes with sufficient training samples available that are closest to a weak classifier in the model space and then a linear combination of the group of strong classifiers and the weak classifier to form a stronger classifier to the tail classes where only few samples are available for training; the linear combination weights are learned from a simple neural network, called Alpha Net. Two datasets artificially truncated from ImageNet and Places, respectively, that were also used in the peer work in the literature, were used to report the evaluations.