Provable Adaptation across Multiway Domains via Representation Learning

This paper studies zero-shot domain adaptation where each domain is indexed on a multi-dimensional array , and we only have data from a small subset of domains . Our goal is to produce predictors that perform well on unseen domains . We propose a model which consists of a domain-invariant latent representation layer and a domain-specific linear prediction layer with a low-rank tensor structure . Theoretically , we present explicit sample complexity bounds to characterize the prediction error on unseen domains in terms of the number of domains with training data and the number of data per domain . In addition , we provide experiments on two-way MNIST and four-way fiber sensing datasets to demonstrate the effectiveness of our proposed model . 1 INTRODUCTION . In many applications , collected datasets are cross-classified by a number of categorical factors such as a set of experimental or environmental conditions that are either controllable or known . These applications can be formalized as a zero-shot domain adaptation ( ZSDA ) problem with each domain being indexed on a multi-dimensional array . Domain-shifts often exist among data collected under different conditions , and to minimize the bias , one would like to collect diverse data to cover all conditions . However , due to high cost of data collection or insufficient resource in the field , it is often impractical to cover all possible combinations of factors . In most cases , training data collection can only cover a few limited , accessible scenarios , and the data from target domain may not be available during the training phase at all . One motivating example is the distributed fiber optical sensing ( Rogers , 1988 ) technique started to be used in smart city applications ( Huang et al. , 2019 ) . Collecting fiber sensing data for machine learning is expensive and the characteristics of sensing data vary considerably across various heterogeneous factors , such as weather-ground conditions , signal source types , and sensing distances . Although conducting experiments to cover a small subset of combinations of different factors is possible , it is difficult to conduct experiments to cover all possible combinations . Specifically , in order to account for the effects of multiple factors ( e.g. , soil type × distance to sensor × weather ) , one has to visit multiple sites and set up the sensing equipment , and wait for the right time until the target type of data can be collected , and some combination of factors are not accessible due to the constraint of the physical world ( e.g . a testbed at 10km sensing distance with asphalt pavement surface ) . Besides , in the digital pathology application , domain shift can be caused by different clinical centers , scanners , and tumor types ( Fagerblom et al. , 2021 ) . It is impossible to cover every combination in training ( e.g. , one center maybe specialized in certain tumor class or own one particular type of scanners ) . Ideally , for these applications , one would want to train a model under a subset of scenarios , and adapt it to new unseen scenarios without collecting new data . Another motivating application to design data augmentation with composition of operations . For instance , in image recognition applications , there are often multiple operations can be combined into augmented datasets , including blurring , translating , viewpoint , flipping , scaling , etc , and each with several levels or values to choose from . It may require huge amount of augmented data to cover all combinations of operations . Ideally , one only wants to try a small subset of all possible combinations . Previous works have shown that composing operations in careful ways outperforms randomly using data augmentation schemes ( Ratner et al. , 2017 ; Cubuk et al. , 2019 ) . In the general ZSDA formulation , each domain is associated with a task descriptor , which is used to generate a predictor for the domain . This function that maps a task descriptor to a predictor , i.e. , a predictor generator , is trained on training domains where data is available ( Yang & Hospedales , 2014 ) . In our setting , we can view the categorical indices for each domain as a task descriptor and thus apply existing approaches . However , this ignores the inherent multiway structure that connects different domains . Furthermore , existing approaches lack theoretical guarantees , and thus it is unclear under what conditions , the learned predictor generator can provably output an accurate predictor on unseen domains . A natural question is Can we design a provably sample-efficient algorithm for zero-shot domain adaptation that exploits the multiway structure ? 1.1 OUR CONTRIBUTIONS . In this paper , we answer this question affirmatively . Our contributions are summarized below . • we consider a multilinear structure which naturally characterize the relatedness among different domains . Formally , we assume that there are totally D = dM domains that form a d×M -dimensional task tensor , where the predictor on domain t ∈ [ D ] is parameterized by w∗t ◦ ϕ∗ ∈ W ◦ Φ where ϕ∗ : X → Rp is a common representation for tasks and wt ∈ Rp is a domain specific linear predictor . We only consider the case when p is small . Note the linear predictors , w∗1 , . . . , w ∗ D ∈ Rp , together form a form a d× d× · · · × d︸ ︷︷ ︸ M times ×p-dimensional tensor , which we denote as T . To enable domain adaptation , we further impose a low-rank multi-linear on the linear predictor T . The structure we impose on the model is that the slice along the last axis is always rank-K with K ≪ dM . That is , for all j ∈ [ p ] , T· , j ∈ Rd×d×···×d has rank K. • We provide a finite-sample statistics analysis for this model . We show that if during training T ∈ [ D ] source domains are observed , and in each observed domain n samples are collected , then the expected excess risk across all D domains is of order Õ ( ( TC ( W ) +C ( Φ ) nT ) 1/4 + p ( KdM2 T ) 1/2 ) , where C ( F ) represents the complexity of function class F . The first term corresponds to the cost of learning the common representation and domain-specific linear classifiers , and the second term corresponds to the cost of generalizing to unseen domains . In the case where n is large , we can simply ignore the first term , and conclude that T should scale with Kd ( pM ) 2 . This theoretically justifies our method can adapt to unseen domains with limited training domains . • We test our proposed method on a two-way MNIST dataset and four-way fiber sensing datasets . The empirical results show our method matches the performance of general ZSDA algorithm by Yang & Hospedales ( 2014 ) which has a lot more parameters than our method . 2 RELATED WORK . For theoretical results of multi-task learning , Baxter ( 2000 ) first derived a covering number bound for sample complexities . Later Maurer et al . ( 2016 ) gave a uniform convergence bound for multitask representation learning . Their bound is of order O ( √ 1/m + √ 1/T ) , where m is the number of training samples in the target domain , so it is not reflecting how their learning process benefits from seeing multiple domains . Recently , Du et al . ( 2020 ) ; Tripuraneni et al . ( 2020 ) successfully derived results of type O ( √ 1/m + √ 1/ ( nT ) ) , which satisfactorily explained how learning algorithm benefits from seeing abundant training data in many source domains , even if training data in target domain is scarce . This setting is often referred to as transfer learning or few-shot learning . A key difference between this setting and ours is that we do not see any training data from the target domain . Blitzer et al . ( 2009 ) also studied ZSDA , but they studied a different approach which relies on multi-view learning and is restricted to linear models . Many attempts to solving multi-task learning via low-rank matrices/tensors have been proposed Romera-Paredes et al . ( 2013 ) ; Signoretto et al . ( 2013 ) ; Wimalawarne et al . ( 2014 ) , focusing on regularizing shallow models . Wimalawarne et al . ( 2014 ) specifically discussed task imputation and gave a sample complexity bound . However , their analysis assumed each training data is uniformly sampled from all source domains , so it is incomparable to our result . Yang & Hospedales ( 2017 ) ; Li et al . ( 2017 ) used low-rank tensor decomposition to share parameters between neural architectures for multiple tasks and empirically verified that both multi-task learning and domain generalization can benefit from the low-rank structure . Some different but closely related settings include unsupervised domain adaptation and zero-shot learning , where the test labels are either unseen during the training phase or different from what the model has been trained with . This is beyond the scope of this paper , as we only consider the case where all domains share same set of labels . The low-rank matrix completion problem has been thoroughly studied in the past decade Srebro & Shraibman ( 2005 ) ; Srebro et al . ( 2004 ) ; Candès & Recht ( 2009 ) ; Recht ( 2011 ) . Despite the combinatorial nature of rank , it has a nice convex relaxation , the trace norm , that can be optimized efficiently , in both noiseless and noisy settings Wainwright ( 2019 ) . Although these quantities ( trace norms , ranks , etc ) are easy to calculate for matrices , most quantities associated with tensors are NP-hard to compute , including the trace norm Hillar & Lim ( 2013 ) . There are many efforts in the theoretical computer science community to tackle tensor completion problems from the computational complexity perspective Barak & Moitra ( 2016 ) ; Liu & Moitra ( 2020 ) . In this paper , we use the statistical properties of tensor completion despite their computational inefficiency . Hence our sample complexity bound does not contradict the seemingly worse bounds in the theory literature . 3 PRELIMINARY AND OVERVIEW . Notation . For D ∈ Z , we let [ D ] = { 1 , 2 , . . . , D } . We write ∥ · ∥ to represent the ℓ2 norm or Frobenius norm . Let ⟨· , ·⟩ be the inner product on an inner product space . Throughout the paper , we assume for simplicity in total there are D = d× d× · · · × d︸ ︷︷ ︸ M data domains . One can easily generalize the results to the general case where D = ∏M i=1 di . We use ⊗ to denote the tensor product and ⊙ the Hadamard product . We use T ∈ [ D ] to denote the number of seen source domains , n to be the amount of sample in a single source domain . Since each t ∈ [ D ] also uniquely corresponds to a multi-index in I ∈ [ d ] ×M , if T ∈ Rd×···×d×p is a d×M dimension tensor composing of pdimensional linear functionals , we also write Tt , · ∈ Rp to represent the classifier at multi-index I , and we use t and I as indices interchangeably .

Summary: Short summary:   The authors propose a zero-shot domain adaptation method with the assumption of each domain having the same set of labels. They obtain finite sample complexity bounds that contain the number of samples(n) per class in the training, the number of seen(training) and unseen(test) tasks under regularity assumptions on the loss and mappings. They provide accuracy results for MNIST and MNIST with rotated digits.   Method:   First, a common linear mapping is used to obtain a common representation for tasks. Then each domain-specific classifer(linear) combines at each index as a rank K tensor. A tensor completion is applied at each index that would give classifier parameters for unseen data as well, that allows them to predict without any data from the target domain. They assume same number of samples and same input dimensions for each domain (the latter can directly be extended to different dimensions). For the experiments, they use LeNet and directly predicted decomposed tensors rather than training and applying tensor completion.  Contributions:   The authors extend Tripuraneni et al. (2020) method and proofs from transfer learning to zero-shot case. They show a uniform convergence bound that has polynomial complexity with respect to the # tasks, # dimensions and rank. They derive excess risk on the order of n^-1/4.