Cross-Domain Few-Shot Learning by Representation Fusion

1 INTRODUCTION . Currently , deep learning is criticized because it is data hungry , has limited capacity for transfer , insufficiently integrates prior knowledge , and presumes a largely stable world ( Marcus , 2018 ) . In particular , these problems appear after a domain shift , that is , a change of the input-target distribution . A domain shift forces deep learning models to adapt . The goal is to exploit models that were trained on the typically rich original data for solving tasks from the new domain with much less data . Examples for domain shifts are new users or customers , new products and product lines , new diseases ( e.g . adapting from SARS to COVID19 ) , new images from another field ( e.g . from cats to dogs or from cats to bicycles ) , new social behaviors after societal change ( e.g . introduction of cell phones , pandemic ) , self-driving cars in new cities or countries ( e.g . from European countries to Arabic countries ) , and robot manipulation of new objects . Domain shifts are often tackled by meta-learning ( Schmidhuber , 1987 ; Bengio et al. , 1990 ; Hochreiter et al. , 2001 ) , since it exploits already acquired knowledge to adapt to new data . One prominent application of meta-learning dealing with domain shifts is few-shot learning , since , typically , from the new domain much less data is available than from the original domain . Meta-learning methods perform well on small domain shifts like new target classes with similar inputs . However , larger domain shifts are still challenging for current approaches . Large domain shifts lead to inputs , which are considerably different from the original inputs and possess different high-level concepts . Nonetheless , low-level concepts are often still shared between the inputs of the original domain and the inputs of the new domain . For images , such shared low-level concepts can be edges , textures , small shapes , etc . One way of obtaining low level concepts is to train a new deep learning model from scratch , where the new data is merged with the original data . However , although models of the original domain are often available , the original data , which the models were trained on , often are not . This might have several reasons , e.g . the data owner does no longer grant access to the data , General Data Protection Regulation ( GDPR ) does no longer allow access to the data , IP restrictions prevent access to the data , sensitive data items must not be touched anymore ( e.g . phase III drug candidates ) , or data is difficult to extract again . We therefore suggest to effectively exploit original data models directly by accessing not only high level but also low level abstractions . In this context , we propose a cross-domain few-shot learning method extracting information from different levels of abstraction in a deep neural network . Representation fusion . Deep Learning constructs neural network models that represent the data at multiple levels of abstraction ( LeCun et al. , 2015 ) . We introduce representation fusion , which is the concept of unifying and merging information from different levels of abstraction . Representation fusion uses a fast and adaptive system for detecting relevant information at different abstraction levels of a deep neural network , which we will show allows solving versatile and complex cross-domain tasks . CHEF . We propose cross-domain ensemble few-shot learning ( CHEF ) that achieves representation fusion by an ensemble of Hebbian learners , which are built upon a trained network . CHEF naturally addresses the problem of domain shifts which occur in a wide range of real-world applications . Furthermore , since CHEF only builds on representation fusion , it can adapt to new characteristics of tasks like unbalanced data sets , classes with few examples , change of the measurement method , new measurements in unseen ranges , new kind of labeling errors , and more . The usage of simple Hebbian learners allows the application of CHEF without needing to backpropagate information through the backbone network . The main contributions of this paper are : • We introduce representation fusion as the concept of unifying and merging information from different layers of abstraction . • We introduce CHEF1 as our new cross-domain few-shot learning method that builds on representation fusion . We show that using different layers of abstraction allows one to successfully tackle various few-shot learning tasks across a wide range of different domains . CHEF does not need to backpropagate information through the backbone network . • We apply CHEF to various cross-domain few-shot tasks and obtain several state-of-theart results . We further apply CHEF to cross-domain real-world applications from drug discovery , where we outperform all competitors . Related work . Representation fusion builds on learning a meaningful representation ( Bengio et al. , 2013 ; Girshick et al. , 2014 ) at multiple levels of abstraction ( LeCun et al. , 2015 ; Schmidhuber , 2015 ) . The concept of using representations from different layers of abstraction has been used in CNN architectures ( LeCun et al. , 1998 ) such as Huang et al . ( 2017 ) ; Rumetshofer et al . ( 2018 ) ; Hofmarcher et al . ( 2019 ) , in CNNs for semantic segmentation in the form of multi-scale context pooling ( Yu & Koltun , 2015 ; Chen et al. , 2018 ) , and in the form of context capturing and symmetric upsampling ( Ronneberger et al. , 2015 ) . Learning representations from different domains has been explored by Federici et al . ( 2020 ) ; Tschannen et al . ( 2020 ) under the viewpoint of mutual information optimization . Work on domain shifts discusses the problem that new inputs are considerably different from the original inputs ( Kouw & Loog , 2019 ; Wouter , 2018 ; Webb et al. , 2018 ; Gama et al. , 2014 ; Widmer & Kubat , 1996 ) . Domain adaptation ( Pan & Yang , 2009 ; Ben-David et al. , 2010 ) overcomes this problem by e.g . reweighting the original samples ( Jiayuan et al. , 2007 ) , learning features that are invariant to a domain shift ( Ganin et al. , 2016 ; Xu et al. , 2019 ) or learning a classifier in the new domain . Domain adaptation where only few data is available in the new domain ( Ben-David et al. , 2010 ; Lu et al. , 2020 ) is called cross-domain few-shot learning ( Guo et al. , 2019 ; Lu et al. , 2020 ; Tseng et al. , 2020 ) , which is an instance of the general few-shot learning setting ( Fei-Fei et al. , 2006 ) . Few-shot learning can be roughly divided into three approaches ( Lu et al. , 2020 ; Hospedales et al. , 2020 ) : ( i ) augmentation , ( ii ) metric learning , and ( iii ) meta-learning . For ( i ) , where the idea is to learn an augmentation to produce more than the few samples available , supervised ( Dixit et al. , 2017 ; Kwitt et al. , 2016 ) and unsupervised ( Hariharan & Girshick , 2017 ; Pahde et al. , 2019 ; Gao et al. , 2018 ) methods are considered . For ( ii ) , approaches aim to learn a pairwise similarity metric under which similar samples obtain high similarity scores ( Koch et al. , 2015 ; Ye & Guo , 2018 ; Hertz et al. , 2006 ) . For ( iii ) , methods comprise embedding and nearest-neighbor approaches ( Snell et al. , 2017b ; 1Our implementation is available at github.com/tomte812/chef . Sung et al. , 2018 ; Vinyals et al. , 2016 ) , finetuning approaches ( Finn et al. , 2017 ; Rajeswaran et al. , 2019 ; Ravi & Larochelle , 2017 ; Andrychowicz et al. , 2016 ) , and parametrized approaches ( Gidaris & Komodakis , 2018 ; Ye et al. , 2020 ; Lee et al. , 2019 ; Yoon et al. , 2019 ; Mishra et al. , 2018 ; Hou et al. , 2019 ; Rusu et al. , 2018 ) . Few-shot classification under domain shifts for metric-based methods has been discussed in Tseng et al . ( 2020 ) . Ensemble methods for few-shot learning have been applied in Dvornik et al . ( 2019 ) , where an ensemble of distance-based classifiers is designed from different networks . In contrast , our method builds an ensemble of different layers from the same network . Hebbian learning as part of a few-shot learning method has been implemented in Munkhdalai & Trischler ( 2018 ) , where fast weights that are used for binding labels to representations are generated by a Hebbian learning rule . 2 CROSS-DOMAIN FEW-SHOT LEARNING . Domain shifts . We assume to have data ( x , y ) , where x ∈ X is the input data and y ∈ Y is the target data . A domain is a distribution p over X ×Y assigning each pair ( x , y ) a probability p ( x , y ) . A domain shift is a change from p ( x , y ) to p̃ ( x , y ) . We measure the magnitude of the domain shift by a distance d ( p , p̃ ) between the distributions p and p̃ . We consider four types of domain shifts ( Kouw & Loog , 2019 ; Wouter , 2018 ; Webb et al. , 2018 ; Gama et al. , 2014 ; Widmer & Kubat , 1996 ) : • Prior shift ( small domain shift ) : p ( y ) is changed to p̃ ( y ) , while p ( x | y ) stays the same . For example , when new classes are considered ( typical case in few-shot learning ) : p ( x , y ) = p ( y ) p ( x | y ) and p̃ ( x , y ) = p̃ ( y ) p ( x | y ) . • Covariate shift ( large domain shift ) : p ( x ) is changed to p̃ ( x ) , while p ( y | x ) stays the same . For example , when new inputs are considered , which occurs when going from color to grayscale images , using a new measurement device , or looking at traffic data from different continents : p ( x , y ) = p ( x ) p ( y | x ) and p̃ ( x , y ) = p̃ ( x ) p ( y | x ) . • Concept shift : p ( y | x ) is changed to p̃ ( y | x ) , while p ( x ) stays the same . For example , when including new aspects changes the decision boundaries : p ( x , y ) = p ( x ) p ( y | x ) and p̃ ( x , y ) = p ( x ) p̃ ( y | x ) . • General domain shift : domain shift between p ( x , y ) to p̃ ( x , y ) . For example , going from Imagenet data to grayscale X-ray images ( typical case in cross-domain datasets ) . Domain shift for images . We consider the special case that the input x is an image . In general , domain shifts can be measured on the raw image distributions e.g . by using theH-divergence ( BenDavid et al. , 2010 ) . However , distances between raw image distributions were shown to be less meaningful in computer vision tasks than abstract representations of deep neural networks ( Heusel et al. , 2017 ; Salimans et al. , 2016 ) . We approximate the distance between the joint distributions d ( p ( x , y ) , p̃ ( x , y ) ) by the distance between the marginals d ( p ( x ) , p̃ ( x ) ) , which is exact in the case of the covariate shift for certain choices of d ( · , · ) , like e.g . the Jensen-Shannon divergence . To measure the distance between the marginals d ( p ( x ) , p̃ ( x ) ) we use the Fréchet Inception Distance ( FID ; Heusel et al. , 2017 ) , i.e . the Wasserstein-2 distance of the features of the respective images activating an Inception v3 network ( Szegedy et al. , 2016 ) under a Gaussian assumption . The FID has proven reliable for measuring performance of Generative Adversarial Networks ( Goodfellow et al. , 2014 ) . Cross-domain few-shot learning . Large domain shifts lead to inputs , which are considerably different from the original inputs . As a result , the model trained on the original domain will not work anymore on the new domain . To overcome this problem , domain adaptation techniques are applied ( Pan & Yang , 2009 ; Ben-David et al. , 2010 ) . Domain adaption can be achieved in several ways , e.g . by reweighting the original samples ( Jiayuan et al. , 2007 ) . Another possibility is to learn a classifier in the new domain . Domain adaptation where in the new domain only few data is available ( Ben-David et al. , 2010 ) which can be used for learning is called cross-domain few-shot learning ( Guo et al. , 2019 ; Lu et al. , 2020 ; Tseng et al. , 2020 ) . In an N -shot K-way few-shot learning setting , the training set ( in meta learning also called one episode ) consists of N samples for each of the K classes .

Summary: This paper primarily deals with cross-domain few-shot learning. Under this setting, there is a large shift in domain going from the meta-train dataset to the few-shot datasets. Inspired by previous work, the authors argue that high-level concepts might not be useful in this setting but low-level concepts like edges, textures and shapes can be utilized. They propose a Cross-domain Hebbian Ensemble Few-shot (CHEF) learner, that learns an ensemble of classifiers at multiple levels of a deep neural network, thus making use of both low and high level concepts. Experimental results show that CHEF does better, in most cases, than learning a separate classifier at a given level. They show results under the cross-domain and the standard few-shot setting.