Cross-Domain Few-Shot Learning by Representation Fusion

1 INTRODUCTION . Currently , deep learning is criticized because it is data hungry , has limited capacity for transfer , insufficiently integrates prior knowledge , and presumes a largely stable world ( Marcus , 2018 ) . In particular , these problems appear after a domain shift , that is , a change of the input-target distribution . A domain shift forces deep learning models to adapt . The goal is to exploit models that were trained on the typically rich original data for solving tasks from the new domain with much less data . Examples for domain shifts are new users or customers , new products and product lines , new diseases ( e.g . adapting from SARS to COVID19 ) , new images from another field ( e.g . from cats to dogs or from cats to bicycles ) , new social behaviors after societal change ( e.g . introduction of cell phones , pandemic ) , self-driving cars in new cities or countries ( e.g . from European countries to Arabic countries ) , and robot manipulation of new objects . Domain shifts are often tackled by meta-learning ( Schmidhuber , 1987 ; Bengio et al. , 1990 ; Hochreiter et al. , 2001 ) , since it exploits already acquired knowledge to adapt to new data . One prominent application of meta-learning dealing with domain shifts is few-shot learning , since , typically , from the new domain much less data is available than from the original domain . Meta-learning methods perform well on small domain shifts like new target classes with similar inputs . However , larger domain shifts are still challenging for current approaches . Large domain shifts lead to inputs , which are considerably different from the original inputs and possess different high-level concepts . Nonetheless , low-level concepts are often still shared between the inputs of the original domain and the inputs of the new domain . For images , such shared low-level concepts can be edges , textures , small shapes , etc . One way of obtaining low level concepts is to train a new deep learning model from scratch , where the new data is merged with the original data . However , although models of the original domain are often available , the original data , which the models were trained on , often are not . This might have several reasons , e.g . the data owner does no longer grant access to the data , General Data Protection Regulation ( GDPR ) does no longer allow access to the data , IP restrictions prevent access to the data , sensitive data items must not be touched anymore ( e.g . phase III drug candidates ) , or data is difficult to extract again . We therefore suggest to effectively exploit original data models directly by accessing not only high level but also low level abstractions . In this context , we propose a cross-domain few-shot learning method extracting information from different levels of abstraction in a deep neural network . Representation fusion . Deep Learning constructs neural network models that represent the data at multiple levels of abstraction ( LeCun et al. , 2015 ) . We introduce representation fusion , which is the concept of unifying and merging information from different levels of abstraction . Representation fusion uses a fast and adaptive system for detecting relevant information at different abstraction levels of a deep neural network , which we will show allows solving versatile and complex cross-domain tasks . CHEF . We propose cross-domain ensemble few-shot learning ( CHEF ) that achieves representation fusion by an ensemble of Hebbian learners , which are built upon a trained network . CHEF naturally addresses the problem of domain shifts which occur in a wide range of real-world applications . Furthermore , since CHEF only builds on representation fusion , it can adapt to new characteristics of tasks like unbalanced data sets , classes with few examples , change of the measurement method , new measurements in unseen ranges , new kind of labeling errors , and more . The usage of simple Hebbian learners allows the application of CHEF without needing to backpropagate information through the backbone network . The main contributions of this paper are : • We introduce representation fusion as the concept of unifying and merging information from different layers of abstraction . • We introduce CHEF1 as our new cross-domain few-shot learning method that builds on representation fusion . We show that using different layers of abstraction allows one to successfully tackle various few-shot learning tasks across a wide range of different domains . CHEF does not need to backpropagate information through the backbone network . • We apply CHEF to various cross-domain few-shot tasks and obtain several state-of-theart results . We further apply CHEF to cross-domain real-world applications from drug discovery , where we outperform all competitors . Related work . Representation fusion builds on learning a meaningful representation ( Bengio et al. , 2013 ; Girshick et al. , 2014 ) at multiple levels of abstraction ( LeCun et al. , 2015 ; Schmidhuber , 2015 ) . The concept of using representations from different layers of abstraction has been used in CNN architectures ( LeCun et al. , 1998 ) such as Huang et al . ( 2017 ) ; Rumetshofer et al . ( 2018 ) ; Hofmarcher et al . ( 2019 ) , in CNNs for semantic segmentation in the form of multi-scale context pooling ( Yu & Koltun , 2015 ; Chen et al. , 2018 ) , and in the form of context capturing and symmetric upsampling ( Ronneberger et al. , 2015 ) . Learning representations from different domains has been explored by Federici et al . ( 2020 ) ; Tschannen et al . ( 2020 ) under the viewpoint of mutual information optimization . Work on domain shifts discusses the problem that new inputs are considerably different from the original inputs ( Kouw & Loog , 2019 ; Wouter , 2018 ; Webb et al. , 2018 ; Gama et al. , 2014 ; Widmer & Kubat , 1996 ) . Domain adaptation ( Pan & Yang , 2009 ; Ben-David et al. , 2010 ) overcomes this problem by e.g . reweighting the original samples ( Jiayuan et al. , 2007 ) , learning features that are invariant to a domain shift ( Ganin et al. , 2016 ; Xu et al. , 2019 ) or learning a classifier in the new domain . Domain adaptation where only few data is available in the new domain ( Ben-David et al. , 2010 ; Lu et al. , 2020 ) is called cross-domain few-shot learning ( Guo et al. , 2019 ; Lu et al. , 2020 ; Tseng et al. , 2020 ) , which is an instance of the general few-shot learning setting ( Fei-Fei et al. , 2006 ) . Few-shot learning can be roughly divided into three approaches ( Lu et al. , 2020 ; Hospedales et al. , 2020 ) : ( i ) augmentation , ( ii ) metric learning , and ( iii ) meta-learning . For ( i ) , where the idea is to learn an augmentation to produce more than the few samples available , supervised ( Dixit et al. , 2017 ; Kwitt et al. , 2016 ) and unsupervised ( Hariharan & Girshick , 2017 ; Pahde et al. , 2019 ; Gao et al. , 2018 ) methods are considered . For ( ii ) , approaches aim to learn a pairwise similarity metric under which similar samples obtain high similarity scores ( Koch et al. , 2015 ; Ye & Guo , 2018 ; Hertz et al. , 2006 ) . For ( iii ) , methods comprise embedding and nearest-neighbor approaches ( Snell et al. , 2017b ; 1Our implementation is available at github.com/tomte812/chef . Sung et al. , 2018 ; Vinyals et al. , 2016 ) , finetuning approaches ( Finn et al. , 2017 ; Rajeswaran et al. , 2019 ; Ravi & Larochelle , 2017 ; Andrychowicz et al. , 2016 ) , and parametrized approaches ( Gidaris & Komodakis , 2018 ; Ye et al. , 2020 ; Lee et al. , 2019 ; Yoon et al. , 2019 ; Mishra et al. , 2018 ; Hou et al. , 2019 ; Rusu et al. , 2018 ) . Few-shot classification under domain shifts for metric-based methods has been discussed in Tseng et al . ( 2020 ) . Ensemble methods for few-shot learning have been applied in Dvornik et al . ( 2019 ) , where an ensemble of distance-based classifiers is designed from different networks . In contrast , our method builds an ensemble of different layers from the same network . Hebbian learning as part of a few-shot learning method has been implemented in Munkhdalai & Trischler ( 2018 ) , where fast weights that are used for binding labels to representations are generated by a Hebbian learning rule . 2 CROSS-DOMAIN FEW-SHOT LEARNING . Domain shifts . We assume to have data ( x , y ) , where x ∈ X is the input data and y ∈ Y is the target data . A domain is a distribution p over X ×Y assigning each pair ( x , y ) a probability p ( x , y ) . A domain shift is a change from p ( x , y ) to p̃ ( x , y ) . We measure the magnitude of the domain shift by a distance d ( p , p̃ ) between the distributions p and p̃ . We consider four types of domain shifts ( Kouw & Loog , 2019 ; Wouter , 2018 ; Webb et al. , 2018 ; Gama et al. , 2014 ; Widmer & Kubat , 1996 ) : • Prior shift ( small domain shift ) : p ( y ) is changed to p̃ ( y ) , while p ( x | y ) stays the same . For example , when new classes are considered ( typical case in few-shot learning ) : p ( x , y ) = p ( y ) p ( x | y ) and p̃ ( x , y ) = p̃ ( y ) p ( x | y ) . • Covariate shift ( large domain shift ) : p ( x ) is changed to p̃ ( x ) , while p ( y | x ) stays the same . For example , when new inputs are considered , which occurs when going from color to grayscale images , using a new measurement device , or looking at traffic data from different continents : p ( x , y ) = p ( x ) p ( y | x ) and p̃ ( x , y ) = p̃ ( x ) p ( y | x ) . • Concept shift : p ( y | x ) is changed to p̃ ( y | x ) , while p ( x ) stays the same . For example , when including new aspects changes the decision boundaries : p ( x , y ) = p ( x ) p ( y | x ) and p̃ ( x , y ) = p ( x ) p̃ ( y | x ) . • General domain shift : domain shift between p ( x , y ) to p̃ ( x , y ) . For example , going from Imagenet data to grayscale X-ray images ( typical case in cross-domain datasets ) . Domain shift for images . We consider the special case that the input x is an image . In general , domain shifts can be measured on the raw image distributions e.g . by using theH-divergence ( BenDavid et al. , 2010 ) . However , distances between raw image distributions were shown to be less meaningful in computer vision tasks than abstract representations of deep neural networks ( Heusel et al. , 2017 ; Salimans et al. , 2016 ) . We approximate the distance between the joint distributions d ( p ( x , y ) , p̃ ( x , y ) ) by the distance between the marginals d ( p ( x ) , p̃ ( x ) ) , which is exact in the case of the covariate shift for certain choices of d ( · , · ) , like e.g . the Jensen-Shannon divergence . To measure the distance between the marginals d ( p ( x ) , p̃ ( x ) ) we use the Fréchet Inception Distance ( FID ; Heusel et al. , 2017 ) , i.e . the Wasserstein-2 distance of the features of the respective images activating an Inception v3 network ( Szegedy et al. , 2016 ) under a Gaussian assumption . The FID has proven reliable for measuring performance of Generative Adversarial Networks ( Goodfellow et al. , 2014 ) . Cross-domain few-shot learning . Large domain shifts lead to inputs , which are considerably different from the original inputs . As a result , the model trained on the original domain will not work anymore on the new domain . To overcome this problem , domain adaptation techniques are applied ( Pan & Yang , 2009 ; Ben-David et al. , 2010 ) . Domain adaption can be achieved in several ways , e.g . by reweighting the original samples ( Jiayuan et al. , 2007 ) . Another possibility is to learn a classifier in the new domain . Domain adaptation where in the new domain only few data is available ( Ben-David et al. , 2010 ) which can be used for learning is called cross-domain few-shot learning ( Guo et al. , 2019 ; Lu et al. , 2020 ; Tseng et al. , 2020 ) . In an N -shot K-way few-shot learning setting , the training set ( in meta learning also called one episode ) consists of N samples for each of the K classes .

Summary: In this paper, the authors focus on cross-domain few-shot learning in the case of large source-target domain shifts. In particular, a new Cross-domain Hebbian Ensemble Few-shot (CHEF) learning method is proposed that performs representation fusion using an ensemble of Hebbian learners on different layers of a DNN trained on the source domain. The proposed CHEF method is validated on classification benchmark datasets with smaller domain shifts (miniImagenet and tieredImagenet) and larger domain shifts (drug discovery, ChEMBL20), and it can outperform related SOTA methods, especially with larger shifts.