Provable Adaptation across Multiway Domains via Representation Learning

This paper studies zero-shot domain adaptation where each domain is indexed on a multi-dimensional array , and we only have data from a small subset of domains . Our goal is to produce predictors that perform well on unseen domains . We propose a model which consists of a domain-invariant latent representation layer and a domain-specific linear prediction layer with a low-rank tensor structure . Theoretically , we present explicit sample complexity bounds to characterize the prediction error on unseen domains in terms of the number of domains with training data and the number of data per domain . In addition , we provide experiments on two-way MNIST and four-way fiber sensing datasets to demonstrate the effectiveness of our proposed model . 1 INTRODUCTION . In many applications , collected datasets are cross-classified by a number of categorical factors such as a set of experimental or environmental conditions that are either controllable or known . These applications can be formalized as a zero-shot domain adaptation ( ZSDA ) problem with each domain being indexed on a multi-dimensional array . Domain-shifts often exist among data collected under different conditions , and to minimize the bias , one would like to collect diverse data to cover all conditions . However , due to high cost of data collection or insufficient resource in the field , it is often impractical to cover all possible combinations of factors . In most cases , training data collection can only cover a few limited , accessible scenarios , and the data from target domain may not be available during the training phase at all . One motivating example is the distributed fiber optical sensing ( Rogers , 1988 ) technique started to be used in smart city applications ( Huang et al. , 2019 ) . Collecting fiber sensing data for machine learning is expensive and the characteristics of sensing data vary considerably across various heterogeneous factors , such as weather-ground conditions , signal source types , and sensing distances . Although conducting experiments to cover a small subset of combinations of different factors is possible , it is difficult to conduct experiments to cover all possible combinations . Specifically , in order to account for the effects of multiple factors ( e.g. , soil type × distance to sensor × weather ) , one has to visit multiple sites and set up the sensing equipment , and wait for the right time until the target type of data can be collected , and some combination of factors are not accessible due to the constraint of the physical world ( e.g . a testbed at 10km sensing distance with asphalt pavement surface ) . Besides , in the digital pathology application , domain shift can be caused by different clinical centers , scanners , and tumor types ( Fagerblom et al. , 2021 ) . It is impossible to cover every combination in training ( e.g. , one center maybe specialized in certain tumor class or own one particular type of scanners ) . Ideally , for these applications , one would want to train a model under a subset of scenarios , and adapt it to new unseen scenarios without collecting new data . Another motivating application to design data augmentation with composition of operations . For instance , in image recognition applications , there are often multiple operations can be combined into augmented datasets , including blurring , translating , viewpoint , flipping , scaling , etc , and each with several levels or values to choose from . It may require huge amount of augmented data to cover all combinations of operations . Ideally , one only wants to try a small subset of all possible combinations . Previous works have shown that composing operations in careful ways outperforms randomly using data augmentation schemes ( Ratner et al. , 2017 ; Cubuk et al. , 2019 ) . In the general ZSDA formulation , each domain is associated with a task descriptor , which is used to generate a predictor for the domain . This function that maps a task descriptor to a predictor , i.e. , a predictor generator , is trained on training domains where data is available ( Yang & Hospedales , 2014 ) . In our setting , we can view the categorical indices for each domain as a task descriptor and thus apply existing approaches . However , this ignores the inherent multiway structure that connects different domains . Furthermore , existing approaches lack theoretical guarantees , and thus it is unclear under what conditions , the learned predictor generator can provably output an accurate predictor on unseen domains . A natural question is Can we design a provably sample-efficient algorithm for zero-shot domain adaptation that exploits the multiway structure ? 1.1 OUR CONTRIBUTIONS . In this paper , we answer this question affirmatively . Our contributions are summarized below . • we consider a multilinear structure which naturally characterize the relatedness among different domains . Formally , we assume that there are totally D = dM domains that form a d×M -dimensional task tensor , where the predictor on domain t ∈ [ D ] is parameterized by w∗t ◦ ϕ∗ ∈ W ◦ Φ where ϕ∗ : X → Rp is a common representation for tasks and wt ∈ Rp is a domain specific linear predictor . We only consider the case when p is small . Note the linear predictors , w∗1 , . . . , w ∗ D ∈ Rp , together form a form a d× d× · · · × d︸ ︷︷ ︸ M times ×p-dimensional tensor , which we denote as T . To enable domain adaptation , we further impose a low-rank multi-linear on the linear predictor T . The structure we impose on the model is that the slice along the last axis is always rank-K with K ≪ dM . That is , for all j ∈ [ p ] , T· , j ∈ Rd×d×···×d has rank K. • We provide a finite-sample statistics analysis for this model . We show that if during training T ∈ [ D ] source domains are observed , and in each observed domain n samples are collected , then the expected excess risk across all D domains is of order Õ ( ( TC ( W ) +C ( Φ ) nT ) 1/4 + p ( KdM2 T ) 1/2 ) , where C ( F ) represents the complexity of function class F . The first term corresponds to the cost of learning the common representation and domain-specific linear classifiers , and the second term corresponds to the cost of generalizing to unseen domains . In the case where n is large , we can simply ignore the first term , and conclude that T should scale with Kd ( pM ) 2 . This theoretically justifies our method can adapt to unseen domains with limited training domains . • We test our proposed method on a two-way MNIST dataset and four-way fiber sensing datasets . The empirical results show our method matches the performance of general ZSDA algorithm by Yang & Hospedales ( 2014 ) which has a lot more parameters than our method . 2 RELATED WORK . For theoretical results of multi-task learning , Baxter ( 2000 ) first derived a covering number bound for sample complexities . Later Maurer et al . ( 2016 ) gave a uniform convergence bound for multitask representation learning . Their bound is of order O ( √ 1/m + √ 1/T ) , where m is the number of training samples in the target domain , so it is not reflecting how their learning process benefits from seeing multiple domains . Recently , Du et al . ( 2020 ) ; Tripuraneni et al . ( 2020 ) successfully derived results of type O ( √ 1/m + √ 1/ ( nT ) ) , which satisfactorily explained how learning algorithm benefits from seeing abundant training data in many source domains , even if training data in target domain is scarce . This setting is often referred to as transfer learning or few-shot learning . A key difference between this setting and ours is that we do not see any training data from the target domain . Blitzer et al . ( 2009 ) also studied ZSDA , but they studied a different approach which relies on multi-view learning and is restricted to linear models . Many attempts to solving multi-task learning via low-rank matrices/tensors have been proposed Romera-Paredes et al . ( 2013 ) ; Signoretto et al . ( 2013 ) ; Wimalawarne et al . ( 2014 ) , focusing on regularizing shallow models . Wimalawarne et al . ( 2014 ) specifically discussed task imputation and gave a sample complexity bound . However , their analysis assumed each training data is uniformly sampled from all source domains , so it is incomparable to our result . Yang & Hospedales ( 2017 ) ; Li et al . ( 2017 ) used low-rank tensor decomposition to share parameters between neural architectures for multiple tasks and empirically verified that both multi-task learning and domain generalization can benefit from the low-rank structure . Some different but closely related settings include unsupervised domain adaptation and zero-shot learning , where the test labels are either unseen during the training phase or different from what the model has been trained with . This is beyond the scope of this paper , as we only consider the case where all domains share same set of labels . The low-rank matrix completion problem has been thoroughly studied in the past decade Srebro & Shraibman ( 2005 ) ; Srebro et al . ( 2004 ) ; Candès & Recht ( 2009 ) ; Recht ( 2011 ) . Despite the combinatorial nature of rank , it has a nice convex relaxation , the trace norm , that can be optimized efficiently , in both noiseless and noisy settings Wainwright ( 2019 ) . Although these quantities ( trace norms , ranks , etc ) are easy to calculate for matrices , most quantities associated with tensors are NP-hard to compute , including the trace norm Hillar & Lim ( 2013 ) . There are many efforts in the theoretical computer science community to tackle tensor completion problems from the computational complexity perspective Barak & Moitra ( 2016 ) ; Liu & Moitra ( 2020 ) . In this paper , we use the statistical properties of tensor completion despite their computational inefficiency . Hence our sample complexity bound does not contradict the seemingly worse bounds in the theory literature . 3 PRELIMINARY AND OVERVIEW . Notation . For D ∈ Z , we let [ D ] = { 1 , 2 , . . . , D } . We write ∥ · ∥ to represent the ℓ2 norm or Frobenius norm . Let ⟨· , ·⟩ be the inner product on an inner product space . Throughout the paper , we assume for simplicity in total there are D = d× d× · · · × d︸ ︷︷ ︸ M data domains . One can easily generalize the results to the general case where D = ∏M i=1 di . We use ⊗ to denote the tensor product and ⊙ the Hadamard product . We use T ∈ [ D ] to denote the number of seen source domains , n to be the amount of sample in a single source domain . Since each t ∈ [ D ] also uniquely corresponds to a multi-index in I ∈ [ d ] ×M , if T ∈ Rd×···×d×p is a d×M dimension tensor composing of pdimensional linear functionals , we also write Tt , · ∈ Rp to represent the classifier at multi-index I , and we use t and I as indices interchangeably .

Summary: The paper addresses the problem of domain generalization in the setting where each domain is indexed on a multidimensional array. It assumes that only a limited number of domains is observed during training and presents a technique for learning a classifier with provable good generalization performance on the unseen domains. The method consists of learning a common latent feature representation for all domains and domain-specific linear functionals, which are obtained by imposing a low-rank constraint on the tensor built by aggregating the functionals of all domains. The authors present a thorough theoretical analysis of the algorithm, including an upper bound for the excess risk on the target domain.