Multi-Domain Adversarial Learning

1 INTRODUCTION . Advances in technology have enabled large scale dataset generation by life sciences laboratories . These datasets contain information about overlapping but non-identical known and unknown experimental conditions . A challenge is how to best leverage information across multiple datasets on the same subject , and to make discoveries that could not have been obtained from any individual dataset alone . Transfer learning provides a formal framework for addressing this challenge , particularly crucial in cases where data acquisition is expensive and heavily impacted by experimental settings . One such field is automated microscopy , which can capture thousands of images of cultured cells after exposure to different experimental perturbations ( e.g from chemical or genetic sources ) . A goal is to classify mechanisms by which perturbations affect cellular processes based on the similarity of cell images . In principle , it should be possible to tackle microscopy image classification as yet another visual object recognition task . However , two major challenges arise compared to mainstream visual object recognition problems ( Russakovsky et al. , 2015 ) . First , biological images are heavily impacted by experimental choices , such as microscope settings and experimental reagents . Second , there is no standardized set of labeled perturbations , and datasets often contain labeled examples for a subset of possible classes only . This has limited microscopy image classification to single datasets and does not leverage the growing number of datasets collected by the life sciences community . These challenges make it desirable to learn models across many microscopy datasets , that achieve both good robustness w.r.t . experimental settings and good class coverage , all the while being robust to the fact that datasets contain samples from overlapping but distinct class sets . † Now at the French Ministry for the Economy and Finance , 75012 Paris . 1Code and data : github.com/AltschulerWu-Lab/MuLANN Multi-domain learning ( MDL ) aims to learn a model of minimal risk from datasets drawn from distinct underlying distributions ( Dredze et al. , 2010 ) , and is a particular case of transfer learning ( Pan & Yang , 2010 ) . As such , it contrasts with the so-called domain adaptation ( DA ) problem ( Bickel et al. , 2007 ; Ben-David et al. , 2010 ; Ganin et al. , 2016 ; Pan & Yang , 2010 ) . DA aims at learning a model with minimal risk on a distribution called `` target '' by leveraging other distributions called `` sources '' . Notably , most DA methods assume that target classes are identical to source classes , or a subset thereof in the case of partial DA ( Cao et al. , 2018 ; Zhang et al. , 2018 ) . The expected benefits of MDL , compared to training a separate model on each individual dataset , are two-fold . First , MDL leverages more ( labeled and unlabeled ) information , allowing better generalization while accommodating the specifics of each domain ( Dredze et al. , 2010 ; Xiao et al. , 2016 ) . Thus , MDL models have a higher chance of ab initio performing well on a new domain − a problem referred to as domain generalization ( Muandet et al. , 2013 ) or zero-shot domain adaptation ( Yang & Hospedales , 2015 ) . Second , MDL enables knowledge transfer between domains : in unsupervised and semi-supervised settings , concepts learned on one domain are applied to another , significantly reducing the need for labeled examples from the latter ( Pan & Yang , 2010 ) . Learning a single model from samples drawn from n distributions raises the question of available learning guarantees regarding the model error on each distribution . Kifer et al . ( 2004 ) introduced the notion ofH-divergence to measure the distance between source and target marginal distributions in DA . Ben-David et al . ( 2006 ; 2010 ) have shown that a finite sample estimate of this divergence can be used to bound the target risk of the learned model . The contributions of our work are threefold . First , we extend the DA guarantees to MDL ( Sec . 3.1 ) , showing that the risk of the learned model over all considered domains is upper bounded by the oracle risk and the sum of theH-divergences between any two domains . Furthermore , an upper bound on the classifier imbalance ( the difference between the individual domain risk , and the average risk over all domains ) is obtained , thus bounding the worst-domain risk . Second , we propose the approach Multi-domain Learning Adversarial Neural Network ( MULANN ) , which extends Domain Adversarial Neural Networks ( DANNs ) ( Ganin et al. , 2016 ) to semi-supervised DA and MDL . Relaxing the DA assumption , MULANN handles the so-called class asymmetry issue ( when each domain may contain varying numbers of labeled and unlabeled examples of a subset of all possible classes ) , through designing a new loss ( Sec . 3.2 ) . Finally , MULANN is empirically validated in both DA and MDL settings ( Sec . 4 ) , as it significantly outperforms the state of the art on three standard image benchmarks ( Saenko et al. , 2010 ; Le Cun et al. , 1998 ) , and a novel bioimage benchmark , CELL , where the state of the art involves extensive domain-dependent pre-processing . Notation . Let X denote an input space and Y = { 1 , . . . , L } a set of classes . For i = 1 , . . . , n , dataset Si is an iid sample drawn from distribution Di on X × Y . The marginal distribution of Di on X is denoted by DXi . Let H be a hypothesis space ; for each h inH ( h : X 7→ Y ) we define the risk under distribution Di as i ( h ) = Px , y∼Di ( h ( x ) 6= y ) . h ? i ( respectively h ? ) denotes the oracle hypothesis according to distribution Di ( resp . with minimal total risk over all domains ) : ? i = i ( h ? i ) = min h∈H i ( h ) ( 1 ) ̄ ( h ? ) =min h∈H ̄ ( h ) = min h∈H 1 n ∑ i i ( h ) ( 2 ) In the semi-supervised setting , the label associated with an instance might be missing . In the following , `` domain '' and `` distribution '' will be used interchangeably , and the `` classes of a domain '' denote the classes for which labeled or unlabeled examples are available in this domain . 2 STATE OF THE ART . Machine learning classically relies on the iid setting : when training and test samples are independently drawn from the same joint distribution P ( X , Y ) ( Vapnik , 1998 ) . Two other settings emerged in the 1990s , `` concept drift '' and `` covariate shift '' . They respectively occur when conditional data distributions P ( Y |X ) and marginal data distributions P ( X ) change , either continuously or abruptly , across training data or between train and test data ( Shimodaira , 2000 ) . Since then , transfer learning has come to designate methods to learn across drifting , shifting or distinct distributions , or even distinct tasks ( Pratt et al. , 1991 ; Pan & Yang , 2010 ) . Restricting ourselves to addressing a single task on a common input space , we distinguish two objectives : minimizing the learning risk over all considered distributions ( MDL ) , or over a single target distribution while exploiting samples from richer source ( s ) ( DA ) . MDL is thus distinct from multiple source DA by their respective focus on the average risk over all distributions , versus target accuracy only . Samples from the different domains can be all , partially , or not labeled ( supervised , semi-supervised and unsupervised settings ) . Finally , different domains can involve the same classes , or some domains can involve classes not included in other domains , referred to as class asymmetry . In MDL , the different domains can be taken into account by maintaining shared and domain-specific parameters ( Dredze et al. , 2010 ) , or through a domain-specific use of shared parameters . The domaindependent use of these parameters can be learned , e.g . using domain-guided dropout ( Xiao et al. , 2016 ) , or based on prior knowledge about domain semantic relationships ( Yang & Hospedales , 2015 ) . Early DA approaches leverage source examples to learn on the target domain in various ways , e.g . through reweighting source datapoints ( Mansour , 2009 ; Huang et al. , 2006 ; Gong et al. , 2013 ) , or defining an extended representation to learn from both source and target ( Daumé III & Marcu , 2006 ) . Other approaches proceed by aligning the source and target representations with PCA-based correlation alignment ( Sun et al. , 2016 ) , or subspace alignment ( Fernando et al. , 2015 ) . In the field of computer vision , a somewhat related way of mapping examples in one domain onto the other is image-to-image translation , possibly in combination with a generative adversarial network ( see references in Appendix A ) . Intuitively , the difficulty of DA crucially depends on the distance between source and target distribution . Accordingly , a large set of DA methods proceed by reducing this distance in the original input space X , e.g . via importance sampling ( Bickel et al. , 2007 ) or by modifying the source representation using optimal transport ( Courty et al. , 2017 ; Damodaran et al. , 2018 ) . Another option is to map source and target samples on a latent space where they will have minimal distance . Neural networks have been intensively exploited to build such latent spaces , either through generative adversarial mechanisms ( Tzeng et al. , 2017 ; Ghifary et al. , 2016 ) , or through combining task objective with an approximation of the distance between source ( s ) and target . Examples of used distances include the Maximum Mean Discrepancy due to Gretton et al . ( 2007 ) ( Tzeng et al. , 2014 ; Bousmalis et al. , 2016 ) , some of its variants ( Long et al. , 2015 ; 2016 ) , the L2 contrastive divergence ( Motiian et al. , 2017 ) , the Frobenius norm of the output feature correlation matrices ( Sun & Saenko , 2016 ) , or the H-divergence ( Ben-David et al. , 2006 ; 2010 ; Ganin et al. , 2016 ; Pei et al. , 2018 ; Long et al. , 2017 ) ( more in Sec . 3 ) . Most DA methods assume that source ( s ) and target contain examples from the same classes ; in particular , in standard benchmarks such as OFFICE ( Saenko et al. , 2010 ) , all domains contain examples from the same classes . Notable exceptions are partial DA methods , where target classes are expected to be a subset of source classes e.g . ( Zhang et al. , 2018 ; Cao et al. , 2018 ) . DA and partial DA methods share two drawbacks when applied to semi-supervised MDL with non-identical domain class sets . First , neither generic nor partial DA methods try to mitigate the impact of unlabeled samples from a class without any labeled counterparts . Second , as they focus on target performance , ( partial ) DA methods do not discuss the impact of extra labeled source classes on source accuracy . However , as shown in Sec . 4.3 , class asymmetry can heavily impact model performance if not accounted for . Bioinformatics is increasingly appreciating the need for domain adaptation methods ( Borgwardt et al. , 2006 ; Schweikert et al. , 2008 ; Xu & Yang , 2011 ; Vallania et al. , 2017 ) . Indeed , experimentalists regularly face the issues of concept drift and covariate shift . Most biological experiments that last more than a few days are subject to technical variations between groups of samples , referred to as batch effects . Batch effects in image-based screening data are usually tackled with specific normalization methods ( Birmingham et al. , 2009 ) . More recently , work by Ando et al . ( 2017 ) applied CorAl ( Sun et al. , 2016 ) for this purpose , aligning each batch with the entire experiment . DA has been applied to image-based datasets for improving or accelerating image segmentation tasks ( Becker et al. , 2015 ; van Opbroek et al. , 2015 ; Bermúdez-Chacón et al. , 2016 ; Kamnitsas et al. , 2017 ) . However , to our knowledge , MDL has not yet been used in Bioimage Informatics , and this work is the first to leverage distinct microscopy screening datasets using MDL .

Summary: The manuscript proposes a multi-domain adversarial learning (MDL) method called MULANN, to leverage multiple datasets with overlapping but distinct class sets, in a semi-supervised setting. The authors define a new discrimination task to discriminate, within each domain, labeled samples from unlabeled ones that most likely belong to extra classes (classes with no labeled or unlabeled samples in the domain). They also introduce a bound on the average- and worst-domain risk in MDL, obtained using the H-divergence.