Title: Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-source domain adaptation; minimax optimization; learning theory
Abstract: We consider a problem of learning a model from multiple sources with the goal to perform
well on a new target distribution.  Such problem arises in
learning with data collected from multiple sources (e.g. crowdsourcing) or
learning in distributed systems, where the data can be highly heterogeneous. The
goal of learner is to mix these data sources in a target-distribution aware way and
simultaneously minimize the empirical risk on the mixed source.  The literature has made some tangible advancements in establishing
theory of learning on mixture domain.  However, there are still two unsolved problems. Firstly, how to estimate the optimal mixture of sources, given a target domain; Secondly, when there are numerous target domains, we have to solve empirical risk minimization for each target on possibly unique mixed source data , which is computationally expensive. In this paper we address both problems efficiently and with guarantees.
We cast the first problem, mixture weight estimation as convex-nonconcave compositional minimax, and propose an efficient stochastic
algorithm with provable stationarity guarantees.
Next, for the second problem, we identify that for certain regime,
solving ERM for each target domain individually can be avoided, and instead parameters for a target optimal
model can be viewed as a non-linear function on
a space of the mixture coefficients.
To this end, we show that in offline setting, a GD-trained overparameterized neural network can provably learn such function.
Finally, we also consider an online setting and propose an label efficient online algorithm, which predicts parameters for new models given arbitrary sequence of mixing coefficients, while enjoying optimal regret.

Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation

Abstract

We consider the problem of learning a model from multiple heterogeneous sources with the goal of performing well on a new target distribution. The goal of learner is to mix these data sources in a target-distribution aware way and simultaneously minimize the empirical risk on the mixed source. The literature has made some tangible advancements in establishing theory of learning on mixture domain. However, there are still two unsolved problems. Firstly, how to estimate the optimal mixture of sources, given a target domain; Secondly, when there are numerous target domains, how to solve empirical risk minimization (ERM) for each target using possibly unique mixture of data sources in a computationally efficient manner. In this paper we address both problems efficiently and with guarantees. We cast the first problem, mixture weight estimation, as a convex-nonconcave compositional minimax problem, and propose an efficient stochastic algorithm with provable stationarity guarantees. Next, for the second problem, we identify that for certain regimes, solving ERM for each target domain individually can be avoided, and instead parameters for a target optimal model can be viewed as a non-linear function on a space of the mixture coefficients. Building upon this, we show that in the offline setting, a GD-trained overparameterized neural network can provably learn such function to predict the model of target domain instead of solving a designated ERM problem. Finally, we also consider an online setting and propose a label efficient online algorithm, which predicts parameters for new targets given an arbitrary sequence of mixing coefficients, while enjoying regret guarantees.

1 Introduction

With a rapidly increasing amount of decentralized data, multiple source domain adaptation has been an important learning scheme in modern machine learning, e.g., in learning with data collected from multiple sources (e.g. crowdsourcing) or learning in distributed systems where the data can be highly heterogeneous such as federated learning. In this learning scenario, given an input space X and output space Y , we assume access to N sources of data, each with its own underlying distributions D j , j ∈ [ N ] over X × Y . Then, given i.i.d. training samples ̂ D 1 , . . . , ̂ D N , and a hypothesis space H , our goal is to learn a model on the combination of these sources, for instance through the Empirical Risk Minimization (ERM) procedure ̂ h α = arg min h ∈H ∑ N j =1 α ( j ) L ̂ D j ( h ) , where L ̂ D j ( h ) is the empirical loss of a model h ∈ H over data samples in ̂ D j , and α ∈ ∆ N is some mixing parameter, such that predictor ̂ h α entails a good generalization performance on a target domain characterized by a distribution T , i.e., yielding a small true risk L T ( ̂ h α ) = ∫ ℓ ( ̂ h α ( x ) , y ) d T ( x , y ) . It is natural to measure the quality of ̂ h α in terms of the excess risk - namely, the difference between the risk of optimal model for target domain h ∗ T = arg min h ∈H L T ( h ) , and that achieved by ̂ h α . Clearly, the performance of ̂ h α will be influenced by several factors, such as the choice of mixing coefficients

Mehrdad Mahdavi

α to aggregate the empirical losses, capacity of H , and discrepancy between target and source data distributions. So, in order to design a good procedure for learning ̂ h α we need to understand aforementioned trade-offs. Over the years the literature on the multiple source learning has dedicated a considerable attention to this problem [25, 5, 35, 17, 12]. To this end, we consider the following bound on the excess risk of ̂ h α :

Theorem 1 (Multi-source learning bound [17]) . Given N source data distributions D 1 , . . . , D N and a target data distribution T , let ̂ h α = arg min h ∈H ∑ N j =1 α ( j ) L ̂ D j ( h ) be the ERM solution with fixed mixture weights α ∈ ∆ N . Then for any ν ≥ 0 , with probability at least 1 -4 e -ν it holds that

where C is some constant, the complexity term is C ( H , α ) := ∑ N j =1 α ( j ) R j ( H ) with R j ( H ) being the Rademacher complexity of H w.r.t. data source j , and m j is the number of samples from source j .

The above bound indicates that, the generalization ability of a model learnt by ERM on an α -combined sources, depends on the α -weighted sum of target-source discrepancies, and the number of samples drawn from each source. To entail a good generalization on target domain, it naturally motivates us to minimize right hand side of the bound over α ∈ ∆ N to get a good mixture parameter. In this paper we can cast this idea as solving the following minimax optimization problem:

where we drop the complexity term as it becomes identical for all sources by fixing the hypothesis space H and bounding it with a computable distribution-independent quantity such as VC dimension [31], or it can be controlled by choice of H or through data-dependent regularization. [17] gave a simple algorithm to minimize the bound of theorem 1 for binary classifiers and 0-1 loss, however their algorithm does not extend to a more general setting. [24] also looked at minimization of a similar bound with the goal to find weights for an optimal mixture, but they did not give a practical algorithm, nor a provable convergence guarantee. However, none of these works aimed to solve (1) because of its complex structure, and so an efficient algorithm for solving (1) so far has not been proposed. In particular, the first difficulty with (1) is that it is a convex-nonconcave objective, which means all minimax algorithms that require inner concavity [23, 22, 27, 28] or PL-condition [30] will fail to converge to a stationary point. However, recently the literature on optimization of this type of objectives has recently made a tangible progress: The first provable convex-nonconcave algorithm was proposed by [34], where they consider alternating gradient descent ascent algorithm. Their algorithm is deterministic, but in practice, we favor a stochastic gradient method. The second difficulty in solving (1) is its compositional structure, which means that simply replacing gradient with stochastic gradient in [34] will not retain convergence guarantees. To tackle these two difficulties, we propose a stochastic corrected gradient descent ascent algorithm, with provable convergence guarantee for solving (1). Our method can be viewed as a variant of the Stochastic Gradient Descent-Ascent (SGDA) algorithm, and moreover here we give a positive answer to the question posed by [34], on whether an algorithm performing simultaneous updates can optimize convex-nonconcave problem? , which could be interesting by its own right.

The discussion above concerns learning with one target domain, but in practice, a more common scenario is that we have multiple target domains to adapt to. For example, in federated learning [26], millions of users might wish to learn a good model from multiple sources, which can have good performance on their own data distribution. Hence, we propose to study Multi-source Multi-target Domain Adaptation scenario (M 2 DA). Here we assume that we have M target domains, each of them characterized by its own distribution T i , i ∈ [ M ] over X × Y . Adapting to M different target domains requires different mixture weights α 1 , . . . , α M , either obtained by solving (1), or supplied by the user. Equipped with mixing parameters, next we have to solve M weighted ERM problems to tailor solutions for each target domain T i , that is

Notice that these M ERM objectives share the same component functions, so we name this problem as a co-component empirical risk minimization . A straightforward and naïve approach is to solve all M weighted ERMs individually which becomes computationally inefficient when dealing with a large number of data sources. Nevertheless, given the benign structure of these M ERM problems, we may inquire whether there is a computationally efficient method for discovering all solutions without the necessity of solving each one individually.

We give an affirmative answer to this question by replacing the learning of the target model by predicting the target model and propose two efficient strategies to learn such predictors (for instance, a neural network). Our algorithm designs are based on the following observation: if we assume that each empirical risk L ̂ D j ( h ) is strongly convex and smooth in parameters of a hypothesis h , then the optimal parameters are given by a Lipschitz function of mixture weights α . More formally, denoting w ∗ ( α ) as optimal parameters of a hypothesis h for α -weighted ERM, w ∗ ( α ) is Lipschitz in α . This means that we can learn function w ∗ ( · ) with, say, a neural network and gradient descent, with provably small generalization error. Moreover, analysis of generalization error allows us to understand when such target model prediction is more efficient than direct learning. In particular, we look at such a phase transition of efficiency , and conclude that when the number of targets M is much larger than number of sources N , i.e, M ≥ Ω((1 /ϵ ) N/ 2 ) , learning to predict solution is more efficient than optimizing to solve all M ERMs. When M is relatively smaller, optimizing to solve ERMs is more efficient than learning to predict.

Finally, as a second learning scenario we consider an online learning setting, where mixture weights α 1 , ..., α M are arriving sequentially, and may not even originate from the same distribution. We cast this problem as an online non-parametric regression problem with inexact labels and propose an label-efficient online algorithm to predict models.

Our contributions The main contributions of this paper is summerized as follows:

We study the multi-source domain adaptation problem where there are multiple source domains and we wish to learn a new model given the mixture of source domains, that can perform well on a given target domain. We build upon existing learning-theoretic results on multi-source domain adaptation and design a new algorithm for weighing of source domains, that casts this problem as a a convex-nonconcave minimax optimization problem. In section 2 we give the first stochastic optimization algorithm for this problem which provably converges to a stationary point. The proposed algorithm is the first provably convergent algorithm for a stochastic compositional convex-nonconcave minimax problem.

We further consider the above adaptation problem with multiple target domains, the Multisource Multi-target Domain Adaptation (M 2 DA). We observe that these multiple adaptation might share a common structure, which allows us to avoid solving adaptation problem for each target domain individually, and instead we can replace it by direct prediction of parameters for a new problem. We consider offline and online settings for prediction of target parameters, and propose computationally efficient algorithms for both. For the offline setting, in section 3.1 we propose to use a two-layer neural network to learn optimal parameters w ∗ ( α ) using bilevel gradient descent. We show that our algorithm can achieve O ( n -2 2+ N ) excess risk. We also identify the regime where our learning based approach is more efficient compared to directly solving each target problem individually.

Finally, in section 3.2 we focus on scenario where target problems arrive sequentially (and could be dependent) and extend our study of direct target parameter prediction to the online setting. We propose a label-efficient algorithm which enjoys O ( n -1 1+ N ) average regret.

Notation We introduce some basic definitions and notation that will be used throughout the paper. Let B d q ( r ) be the ball in q -metric centered at the origin and of radius r > 0 , and let S d -1 ( r ) = { x ∈ R d : ∥ x ∥ 2 = r } ⊂ R d be the ℓ 2 -norm unit sphere centered at the origin, and finally let S d -1 = S d -1 (1) . In addition, the probability simplex is defined as ∆ N = { α ∈ [0 , 1] N : ∥ α ∥ 1 = 1 } . Concatenation of vectors is denoted by parentheses, that is ( w 1 , . . . , w m ) = [ w ⊤ 1 , . . . , w ⊤ m ] ⊤ . A vector norm ∥ · ∥ is understood as Euclidean norm, while ∥ x ∥ ∞ = max i | x i | . For a matrix M , ∥ M ∥ op denotes its spectral norm while ∥ M ∥ F is its Frobenius norm. For some f : S d -1 → R the empirical semi-norm is defined as ∥ f ∥ 2 n = 1 n ( f ( x 1 ) 2 + · · · + f ( x n ) 2 ) and is always taken

w.r.t. the training sample S . In addition, for g : S d -1 → R , we define an empirical inner product ⟨ f, g ⟩ n = 1 n ( f ( x 1 ) g ( x 1 ) + · · · + f ( x n ) g ( x n )) . At the same time, ∥ f ∥ 2 = ∥ f ∥ 2 L 2 ( P X ) .

2 Mixture Weights Estimation via Convex-nonconcave Minimax Optimization

In this section we focus on a single target domain and present an Algorithm 1 designed to solve a minimax problem (1) to estimate the mixture weights. We assume that hypothesis h is parameterized by a vector space { w ∈ W ⊆ R d } , and use f j ( w ) = L ̂ D j ( h ) to denote the empirical risk over data source j . Similarly we define f ̂ T ( w ) = L ̂ T ( h ) . We do the following standard relaxations. First, for the sake of simplicity in computation, we relax the square root on the quadratic term w.r.t. α . Second, since the absolute value function is non-smooth, we shall use the smooth approximation function g to replace it, e.g., g ( x ) = √ x 2 + c where c is some small number (here g ( · ) is smooth approximation of | · | ). These relaxations lead to solving the following compositional convex-nonconcave minimax optimization problem:

where M = diag { 1 m 1 , . . . , 1 m N } . We are interested in developing a stochastic optimization algorithm to solve (2). It is a strongly-convex-nonconcave minimax problem, and it is one of most difficult type of minimax problem due to the absence of inner concavity. To the best of our knowledge, only Xu et

Algorithm 1: Mixture Weight Estimation

Input: Target domain T , Source domains D 1 , ..., D N , Initialization variable w 0 = w -1 , z 0 1 , ..., z 0 N , Positive hyper-parameters ( B,β,η,γ ) (see theorem 2). for t = 0 , ..., T -1 do

Sample a minibatch ξ t T of size B from target domain T , and ξ t 1 , . . . , ξ t N from source domains D 1 , . . . , D N

Compute gradient for

w

t

+1

=

P

W

(

w

t

+

γ

w

g

t

:

w

g

)

Make vector v ∈ R N whose j th coordinate is g ( z t j ) .

Compute gradient for

α

t

+1

=

P

∆

N

(

α

t

-

α

η

g

:

t

α

g

)

al. [34] proposed a deterministic algorithm to solve it, but it is still unknown whether a stochastic algorithm can solve it with provable guarantee. We give an affirmative answer to this question, by proposing an algorithm built on celebrated stochastic gradient descent ascent [23]. In addition to nonconcavity nature of 2, another difficulty that arises from the compositional structure of objective is that we cannot simply compute stochastic gradients, namely (with E [ · ] ≡ E [ · | data] ):

̸

E [ g ′ ( f ̂ T ( w ; ξ T ) -f j ( w ; ξ j ))( ∇ f ̂ T ( w ; ξ T ) -∇ f j ( w ; ξ j ))] = g ′ ( f ̂ T ( w ) -f j ( w ))( ∇ f ̂ T ( w ) -∇ f j ( w )) , where ξ 1 , ξ 2 , . . . are independent random elements in sample space Ξ = X × Y that capture stochasticity of the algorithm. To alleviate this issue, we borrow 'the stochastic corrected gradient' idea from [7], and maintain an auxiliary variable by introducing

In words, for each source j , we maintain a variable z t j serving as a correction term, to ensure that z t j is getting closer to inner function component f ̂ T ( w t ) -f j ( w t ) . Relying on the auxiliary variable, we then have gradient estimates

t

α

=

v

+2

C

M

α

t

.

t

w

=

∑

α

t

j

∇

g

(

z

)(

∇

f

(

w

t

;

ξ

t

T

)

-∇

f

j

(

w

t

;

ξ

t

j

))

N

j

=1

t

+1

j

T

̂

Then, denoting the projection operator onto convex set C by P C ( · ) , the update rule becomes

2.1 Convergence Analysis

In this section we are going to present the convergence guarantee for Algorithm 1. We make the following standard assumption on objective in (2).

Assumption 1. We make the following assumptions on g and f :

g ( z ) is G g Lipschitz and L g smooth. f j ( w ; ξ ) is G f Lipschitz and L f smooth, ∀ w ∈ W , j ∈ [ N ] , ξ ∈ Ξ .

E ∥∇ f j ( w ; ξ ) -∇ f j ( w ) ∥ 2 ≤ σ 2 , ∀ w ∈ W .

max α ∈ ∆ N , w ∈W F ( α , w ) ≤ F max , max w ∈W g ( f ̂ T ( w ) -f j ( w )) ≤ B g , ∀ j ∈ [ N ] .

Points 1 and 2 of Assumption 1 are standard in the literature on compositional optimization [7]. Point 3 guarantees boundedness of objective value, which can be ensured since we are working in the bounded parameter domain. Assumption 1 also implies the following property of F .

Proposition 1. Under Assumption 1, F ( α , w ) is L := max { 4 G 2 f L g +2 G g L f , 2 C m min } smooth, and µ = 2 C m max strongly convex in α .

Next, we consider the following convergence measure:

Definition 1 (Convergence Measure [34]) . Given two parameters, α and w , we define the following quantity as a stationary gap

Given the nonconcave nature of (2), we are only able to show the convergence to a stationary point. Definition 1 measures the stationarity given parameter pair ( α , w ) by examining how much the parameter will change if we run one step projected gradient descent-ascent on them. Alternatively, one could consider the widely employed primal function [23] as a convergence measure, ∥∇ Φ( α ) ∥ with Φ( α ) = max w ∈W F ( α , w ) , but it is ill-suited to express stationarity since F ( α , · ) is non-concave.

One of our main results, proved in appendix A, establishes the convergence rate of Algorithm 1:

Theorem 2. Consider Assumption 1 and let L and µ be defined in Proposition 1. Then, letting B = Θ ( max { G 2 g Nσ 2 ϵ 2 , κLσ 2 ϵ 2 }) , β = 0 . 1 , η = Θ ( µ L 2 ) , γ = Θ ( µ 3 NG 2 g G 2 f L 2 ) , the Algorithm 1 guarantees that

with the gradient complexity bounded by:

To the best of our knowledge, this is the first convergence proof for stochastic algorithm on solving strongly-convex-nonconcave problem. We achieve O ( ϵ -4 ) gradient complexity required to reach an ϵ stationary point. In contrast to the most relevant result of [34], they show the rate O ( ϵ -2 ) for a deterministic Alternating Gradient Projection (AGP) in a strongly-convex-nonconcave setting. Note that our result also positively answers the question posed by [34], on whether some algorithm performing simultaneous instead of alternative updates can optimize strongly-convex-nonconcave minimax problem. Finally, compared to O ( ϵ -4 ) rate of SGDA given a nonconvex-strongly-concave problem [23], we need roughly same stochastic gradient evaluations.

3 Multiple Target Domains: Learning to Solve Co-component ERM

Up till now, the main focus was on the problem of learning good mixture parameters given a single target domain. Now we turn to a more general setting where we have M target domains, each associated with a different data distribution which necessitates per target mixture weights α 1 , . . . , α M , either obtained by our algorithm, or provided by the user to guarantee good generalization on individual domains. Next, to get personalized models for these M domains, we have to solve M different ERM problems based on these mixture weights:

Anaïve way is to solve each of them, which will result in a computational complexity of M multiplied by the cost required to minimize each individual f α i to a desired precision. We note that such a solution does not exploit the benign structure of these ERM problems: they share the same component functions ( f j ) j , and the only difference is in the mixture weights. It naturally motivates us to ask, can we propose an efficient algorithm which avoids solving all these M co-component ERM problems from scratch? Consider the solution of min w ∈W f α ( w ) := ∑ N j =1 α ( j ) f j ( w ) as a function of α ,

Fortunately, if we assume that each source empirical risk f j is strongly convex and L f smooth in model parameters, we have the following Lipschitz property (shown in appendix B.1):

Lemma 1. If each f j is µ f strongly convex and L f smooth, then w ∗ ( · ) is κ ∗ = √ NG f /µ f Lipschitz.

Some basic algebra shows that in the above example w ∗ ( α ) is indeed Lipschitz in α with respect to ℓ 2 metric. The Lipschitz property allows us to learn w ∗ efficiently. In particular, learning arbitrary Lipschitz (and bounded) vector-valued function w ∗ : R N → R d is an instance of a well-studied nonparametric regression problem [11]. In the following we will consider algorithms for learning w ∗ in both offline and online setting and which are provably capable of estimating w ∗ at an almost optimal rate. In offline setting, we assume that we have access to a subset of M mixture weights, say α 1 , ..., α n , and we shall use a two layer neural network h θ ( · ) to learn w ∗ ( · ) . Our algorithm is GD based empirical risk minimization with adaptive label refining. In a nutshell, given an α , since we do not have access to w ∗ ( α ) , we will use gradient descent to jointly solve weighted ERM with α to get an approximation of w ∗ ( α ) as well as optimizing neural network parameters. With a mild distributional assumption on α , we show that our algorithm guarantees that the two layer network learns w ∗ ( α ) , that is, it achieves a small excess risk E α ∥ h θ ( α ) -w ∗ ( α ) ∥ 2 .

In online setting, we assume that we observe an arbitrary sequence α 1 , α 2 , ... on a simplex, and we wish to predict parameters close to w ∗ ( α 1 ) , w ∗ ( α 2 ) , ... . As baseline algorithm we will consider a well-known online nonparametric regression that greedily covers the simplex with local online learners and which enjoys almost-optimal regret [13]. However, in the considered online protocol, the algorithm will need access to labels, and revealing each label requires to solve (3) to some desired accuracy. Here we explore a possibility that in practice we might be satisfied with ϵ -average regret, while saving the labelling cost. To this end we propose a modification of the algorithm that randomly skips some labels, while incurring a slightly larger regret.

3.1 Offline Setting: Learning Lipschitz function with ReLU Neural Network

In this section we consider offline learning of w ⋆ . The Lipschitzness guarantees that the w ∗ ( α ) function can be efficiently learnt on finite α s, and generalizable to unseen α . Hence, we propose to use a vector-valued two layer ReLU neural network h θ to learn w ∗ ( α ) .

We consider a two layer vector-valued neural network h θ : R N → R d , h θ ( x ) = [ a ⊤ 1 ( U 1 x ) + , ..., a ⊤ d ( U d x ) + ] , where parameters of the hidden layer are matrices U i ∈ R m × N , collectively captured by the parameter vector θ = (vec( U 1 ) , . . . , vec( U d )) ∈ R dmN . Here a i ∈ {± 1 / √ m } m are parameters of the output layer . In the following the hidden layer is tuned by Algorithm 2, while parameters of the output layer are fixed throughout training. We assume that at initialization, for each U i , the first half of its rows are drawn i.i.d. from isotropic standard Gaussian and the remaining half is identical to the first half. Similarly, for each a i , half of the entries are set

Algorithm 2: Learning w ∗ function by a neural network

Input: Number of global iteration T , Number of Iteration for inner problem R .

for

t

= 1

, ..., T

do

Algorithm 3: GD( v , α , K )

to -1 / √ m and the rest to 1 / √ m (we assume that m is even). This initialization ensures that each output coordinate is 0 and so the empirical risk is bounded by a constant at initialization.

We assume that we observe mixture weights α 1 , . . . , α n ∈ ∆ N i.i.d. according to some underlying distribution U . Such mixture weights can be obtained by Algorithm 1 and their independence means that samples originating from target domains are independent from each other.

We learn the neural network by solving the following Bi-level ERM:

The parameters of a neural network θ should have the well-controlled excess risk on unseen α :

To solve (4), we use a nested loop procedure which performs a GD step on a neural network objective ̂ R , while in the inner loop we approximately find 'labels' w ∗ ( α 1 ) , ..., w ∗ ( α n ) using a K -step GD. The entire procedure for solving (4) is described in Algorithm 2. Then, the following theorem shows that the two layer neural net optimized by Algorithm 2 learns w ∗ ( α ) :

Theorem 3. Let λ 0 = N · polylog( N,n ) . Consider a neural network of with m ≥ Ω ( n 8+ 2 2+ N ) . Then, for the Algorithm 2 with η ≤ 1 2 , γ = 1 L f , T ≥ Ω ( n Nλ 2 0 η log( n ) ) , and K ≥ Ω ( κ log ( ηTnD λ 0 )) , the following excess risk bound holds with probability at least 0 . 99 :

The proof given in appendix B.5 is based on a more-or-less standard Neural Tangent Kernel (NTK) approximation argument [15, 4], namely we use the key fact that predictions made by a GD-trained overparameterized neural network are close to those made by a Kernelized Least-Squares (KLS) predictor (given that the width of the network is sufficiently large). Now, such a KLS GD-trained predictor can learn Lipschitz target functions: It is well known that by learning on a sufficiently large Reproducing kernel Hilbert space (RKHS) (with polynomial spectral decay), one can approximate Lipschitz functions well [8, 2]. Here our goal is to approximate a vector-valued function, however, by treating each output independently we follow existing proofs [14, 21] for scalar-valued Lipschitz regression by GD-trained neural networks and arrive at the same excess risk times d .

Optimality of our rate. Here we show that a two-layer neural network trained by a bi-level Gradient Descent (GD) can learn a vector-valued function with O ( dn -2 2+ N ) excess risk. If we ignore the dependency on d , our result matches the minimax rate of learning a scalar valued Lipschitz function [11].

▷ Neural network parameter update

▷ Label refining by K -step gradient descent

Efficiency of our learning-based approach. Given M mixture weights ( α i ) i M =1 , the baseline naïve approach is to solve all M weighted ERM with gradient descent, to accuracy level ϵ , which requires Θ( Mκ log(1 /ϵ )) time complexity. Using our approach, we first need to learn a neural network with ϵ excess risk, and it needs n = Θ ( ( κ ∗ 2 d/ϵ ) 1+ N/ 2 ) samples, which implies that we need to solve this many weighted ERM problems, resulting a complexity of Θ ( ( κ ∗ 2 d/ϵ ) 1+ N/ 2 · κ log(1 /ϵ ) ) . Once we learn a neural network, we just need to pay for the inference cost to predict w ∗ for each α i . Putting things together, the total time complexity is Θ ( ( κ ∗ 2 d/ϵ ) 1+ N/ 2 · κ log(1 /ϵ ) + M ) . We observe the following regimes:

When M ≥ Ω ( ( κ ∗ 2 d/ϵ ) 1+ N/ 2 · κ log(1 /ϵ ) κ log(1 /ϵ ) -1 ) , learning is more efficient than solving M ERMs.

Otherwise, directly solving M ERMs is more efficient than learning a model predictor.

Intuitively, when the number of target domains is much larger than number of source domains, our learning based approach is strictly more efficient. It is also interesting to note that our learning based approach can avoid computational overhead of M , but suffers exponential cost from the number of sources N . While solving ERMs avoids the price for N , the computational cost increases linearly in terms of M .

3.2 Online Setting: Label Efficient Nonparametric Online Regression

In previous section we discussed nonparametric offline learning of w ∗ with distributional assumption on α 1 , ..., α M . In this section we consider the following online learning protocol with oblivious adversary. Given a known and fixed parameter p ∈ [0 , 1] , and an unknown sequence ( α 1 , w ∗ ( α 1 )) , ( α 2 , w ∗ ( α 2 )) , · · · ∈ ∆ N × B d 2 ( D ) of inputs and labels, at every round t = 1 , 2 , . . .

the environment reveals mixture weights α t ∈ ∆ N ;

the learner selects a label ˆ w t ∈ B d 2 ( D ) and incurs loss ℓ t ( ˆ w t ) = ∥ ˆ w t -w ⋆ ( α t ) ∥ ∥ 2 ;

the learner samples Z t ∼ Bern( p ) and observes I { Z t = 1 } w t , when w t is a GD-optimized approximation of w ⋆ ( α t ) .

In particular, we introduce Algorithm 4, a modified version of the online nonparametric regression algorithm proposed by [13]. Algorithm 4 iteratively constructs a packing of ∆ N using ℓ 2 balls centered on a subset of previously observed inputs. At each step t , the label (parameters) associated with the current input α t is predicted by averaging the labels of past inputs within the ball whose center α s is closest to α t in the ℓ 2 metric (note that labels are vector-valued). If α t lies outside the nearest ball, a new ball with center α t is created. The radii ε t of all balls shrink at a rate t -1 / (1+ N ) .

Note that efficient (log in the number of centers) algorithms for approximate nearest-neighbor search are well-known [19], as well as highly-optimized open-source packages are readily available [16].

In contrast to the original algorithm by [13], where all sequentially observed labels are used to generate predictions (update local learners), our algorithm variant uses only a p -fraction of labels on average. This reduces label complexity at the cost of increased regret. This approach is referred to as label-efficient prediction in online learning [6, Sec. 6.2] and is beneficial when accessing labels is costly. The following theorem, shown in appendix C, establishes the regret bound of Algorithm 4.

Theorem 4 (Regret bound) . Let f : ∆ N → R d be arbitrary κ ∗ -Lipschitz function with respect to ∥ · ∥ metric and let C N > 0 be a metric-dependent constant. Then, Algorithm 4 with ε t = t -1 1+ N satisfies

Moreover, by definition ℓ t ( w ∗ ( α t )) = 0 , so choosing f ( · ) = w ∗ , the above bound implies that:

The core idea behind the proof (deferred to appendix C) involves maintaining a balance between the regret contribution of each ball and an added regret term arising from approximating the target function using Voronoi partitioning. The regret contribution of each ball is logarithmic in the number of predictions made due to regret of online quadratic optimization [6, p. 42]. Ignoring log factors, the overall regret contribution equals the number of balls, which is essentially governed by the packing number with respect to the ℓ 2 metric. The additional term in the regret comes from the algorithm's prediction being constant within the Voronoi cells of ∆ N induced by the current centers (considering that we predict using the nearest neighbor). Thus, an extra term equal to the product of the balls' radius and the Lipschitz constant is incurred. Finally, the label efficient algorithm we present here incurs yet another, p -dependent terms, which accounts for the missed labels.

Corollary 1. If our desired average regret is ϵ > 0 , then Algorithm 4 has label complexity:

Note that we recover the standard version of the algorithm (non-label efficient) by trivially setting p = 1 , which in contrast has label complexity of order ˜ O (max { ( Dκ ∗ /ϵ ) 1+ N , (1 /ϵ ) 1+ N } ) , which is strictly larger than the label efficient version as long as ϵ is not zero. When our desired regret goes to zero, the label complexity of two algorithms will tend to be the same asymptotically.

4 Experiments

To demonstrate the effectiveness of our proposed mixture weights estimation algorithm, we conducted an experiment using MNIST dataset [1] according to the following specifications. We consider a scenario with 15 source distributions, and dividing them into 3 groups. For group 1, it contains 5 source distributions and each distribution contains 100 data (80 for training and 20 for testing) samples which are drawn uniformly randomly from class '0', '1' and '2'. Group 2 and 3 share similar settings but their distributions' data are drawn from class '3', '4', '5', and class '6', '7', '9', '10' respectively. The data generation process is summarized in Table 1.

To demonstrate the effectiveness of our Algorithm 1, we implemented and run experiments with two-layer MLP neural network. We choose four different target setting: (1) target distribution

from Group 1, (2) target distribution from Group 2, (3) target distribution from Group 3, (4) target distribution from the mix of Group 1 and Group 2. We compare three algorithms: 1. Weighted ERM using our learnt weights, 2. ERM on averaged weights and 3. ERM solely on target domain, and presented our findings in Table 2. The results indicate that the accuracy achieved using the learnt alphas outperforms the other two approaches.

5 Discussion and Conclusions

In this paper we studied the multi-source multi-target domain adaptation problem. In the first part of the paper we gave an algorithm for adressing a minimax problem, that provably finds good mixture weights of source domains, given a single target domain. In the second part we studied the problem of domain adaptation with multiple target domains, and introduced the co-component ERM problem. We gave two concrete algorithms to solve co-component ERM problem, in offline and online settings. There are several potential future venues for future work, which we briefly discuss below.

Online mixture weight prediction Throughout Section 3, we assumed that the target domains' α s are given. However, it would be interesting if given a new target domain, one could predict good mixture weight in an online fashion, and our Algorithm 1 could serve as an oracle to give the inexact label. Meanwhile, since the Algorithm 1 takes considerable time to converge, a desired online algorithm should also be label-efficient.

The complexity of solving co-component ERM The co-component ERM problem we introduced in section 3 is interesting from pure optimization perspective. Even though we proposed the learningbased approach to avoid heavy computation, one alternative direction is to develop efficient algorithms to directly solve M co-component ERM problems, and give upper and lower complexity bounds.

More structure in w ⋆ and better phase transition laws In this work we considered a basic structure in co-component ERM problems (strong-convexity and smoothness), which gave rise to Lipschitzness of w ⋆ . Lipschitz class of functions is very large and in general can only be learned at a rate Θ( n -2 2+ N ) . As discussed in section 3.1 this allowed us to argue that the learning approach is more efficient than solving co-component ERM whenever M ≥ Ω((1 /ϵ ) N/ 2 ) . However, we could potentially obtain better rates (and better laws) of learning w ⋆ having more structure in w ⋆ . For example, assuming that w ⋆ is H -times differentiable, the excess risk would behave as Θ( n -2 H 2 H + N ) [11]. Thus, the learning approach would be more efficient when M ≥ Ω((1 /ϵ ) N/ (2 H ) ) , that is with potentially much fewer sources than targets. It is an intriguing question to figure our which co-component ERM problems would allows for a nicer structure in w ⋆ .

Alternative learning based approach for co-component ERM There may be several other learning based method to solve co-component ERM. One potential approach is Meta learning [10]. The idea is to train a meta model w meta by optimizing a pre-defined meta objective based on some sampled mixture weights, and the goal would be to find a model that can quickly adapt to tasks with different mixture weights α . We leave this as a promising open problem.

References

The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ .

F. Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research , 18(1):629-681, 2017.

P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

P. L. Bartlett, A. Montanari, and A. Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica , 2021.

S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning , 79(1-2):151-175, 2010.

N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge University Press, 2006.

T. Chen, Y. Sun, and W. Yin. Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization. IEEE Transactions on Signal Processing , 69:4937-4948, 2021.

F. Cucker and D. X. Zhou. Learning theory: an approximation theory viewpoint , volume 24. Cambridge University Press, 2007.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learing (ICML) , pages 1126-1135. PMLR, 2017.

L. Györfi, M. Köhler, A. Krzy˙ zak, and H. Walk. A distribution-free theory of nonparametric regression . Springer, 2006.

S. Hanneke and S. Kpotufe. A no-free-lunch theorem for multitask learning. The Annals of Statistics , 50(6):3119-3143, 2022.

E. Hazan and N. Megiddo. Online Learning with Prior Knowledge. In Learning Theory , pages 499-513. Springer, 2007.

T. Hu, W. Wang, C. Lin, and G. Cheng. Regularization matters: A nonparametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 829-837. PMLR, 2021.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Conference on Neural Information Processing Systems , 2018.

J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019.

N. Konstantinov and C. H. Lampert. Robust learning from untrusted sources. In International Conference on Machine Learing (ICML) , pages 3488-3498. PMLR, 2019.

S. Kpotufe and F. Orabona. Regression-tree tuning in a streaming setting. Conference on Neural Information Processing Systems , 26, 2013.

R. Krauthgamer and J. R. Lee. Navigating nets: simple algorithms for proximity search. In Proceedings of the 15th annual ACM-SIAM Symposium on Discrete algorithms , pages 798-807. Society for Industrial and Applied Mathematics, 2004.

I. Kuzborskij and N. Cesa-Bianchi. Nonparametric Online Regression while Learning the Metric. In Conference on Neural Information Processing Systems , 2017.

I. Kuzborskij and Cs. Szepesvári. Learning lipschitz functions by gd-trained shallow overparameterized relu neural networks. arXiv:2212.13848, 2022.

T. Lin, C. Jin, and M. Jordan. Near-optimal algorithms for minimax optimization. arXiv preprint arXiv:2002.02417 , 2020.

T. Lin, C. Jin, and M. I. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. arXiv preprint arXiv:1906.00331 , 2019.

Y. Mansour, M. Mohri, J. Ro, A. T. Suresh, and K. Wu. A theory of multiple-source adaptation with limited target labeled data. In Arindam Banerjee and Kenji Fukumizu, editors, International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 130 of Proceedings of Machine Learning Research , pages 2332-2340. PMLR, 13-15 Apr 2021.

Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Conference on Computational Learning Theory (COLT) , 2009.

H. B. McMahan, E. Moore, D. Ramage, and S. Hampson. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629 , 2016.

A. Mokhtari, A. Ozdaglar, and S. Pattathil. Convergence rate of o (1/k) for optimistic gradient and extra-gradient methods in smooth convex-concave saddle point problems. arXiv preprint arXiv:1906.01115 , 2019.

A. Mokhtari, A. Ozdaglar, and S. Pattathil. Proximal point approximations achieving a convergence rate of o(1/k) for smooth convex-concave saddle point problems: Optimistic gradient and extra-gradient methods. arXiv preprint arXiv:1906.01115 , 2019.

Q. Nguyen, M. Mondelli, and G. F. Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learing (ICML) , pages 8119-8129. PMLR, 2021.

M. Nouiehed, M. Sanjabi, J. D. Lee, and M. Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. arXiv preprint arXiv:1902.08297 , 2019.

S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge University Press, 2014.

N. Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Conference on Neural Information Processing Systems , pages 2199-2207, 2010.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Z. Xu, H. Zhang, Y. Xu, and G. Lan. A unified single-loop alternating gradient projection algorithm for nonconvex-concave and convex-nonconcave minimax problems. Mathematical Programming , pages 1-72, 2023.

C. Zhang, L. Zhang, and J. Ye. Generalization bounds for domain adaptation. Conference on Neural Information Processing Systems , 25, 2012.

A Proof of theorem 2: Stationarity of Algorithm 1

A.1 Proof Sketch

To prove the convergence of Algorithm 1 to a stationary point, we first need to relate our convergence measure to actual iterates during the algorithm dynamics. Defining

the following statements hold:

Bounds above show that the gradient norm can be bounded by the difference between two iterates generated by Algorithm 1, and the approximation error of our stochastic correction steps. Hence, we need to quantify how well the proposed stochastic correction step can approximate inner finite functions, i.e., the difference between z t +1 j and f ̂ T ( w t ) -f j ( w t ) . This gap can be bounded following a standard result from [7]:

Besides above tracking error, we will also need to control the iterates gap of auxiliary variable z t j :

It shows that as the primal iterates approach the stationary point (i.e., iterates will not change too much), our auxiliary variable z t j also will not change too much.

Next, we are going to characterize the upper bound of E ∥ ∥ α t -α t +1 ∥ ∥ 2 , E ∥ ∥ w t -w t +1 ∥ ∥ 2 . To this end, our proof relies on constructing two-level potential functions. Our first level potential function is defined as

Then we prove the following one-iteration relationship:

where C 1 and C 2 are some constant depending on η, γ, L, µ . To eliminate the approximation error E ∥ ∥ z t j -( f ̂ T ( w t -1 ) -f j ( w t -1 )) ∥ ∥ 2 and difference of auxiliary iterates E ∥ ∥ z t j -z t -1 j ∥ ∥ 2 , we need construct another potential function ˜ F t +1 :

Finally, we arrive at the following bounds.

Conducting telescoping summation will yield the desired result.

A.2 Technical Lemmas

Proposition 2. If g ( · ) is G g Lipschitz and L g smooth, and f ̂ T ( · ) , f 1 ( · ) , ..., f N ( · ) are all G f Lipschitz and L f smooth, then F ( α , w ) is L := max { 4 G 2 f L g +2 G g L f , 2 C m min } smooth, and µ = 2 C m max strongly convex in α .

Proof. We first examine the Lipschitzness of gradient of F w.r.t. α .

Hence

Similarly for gradient of F w.r.t. w :

For strong convexity, we compute Hessian w.r.t. α :

so F ( α , w ) is 2 C M strongly convex in α

The following lemma bounds the tracking error of the stochastic correction algorithm.

Lemma 2 (Tracking Error [7]) . For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

An immediate implication of Lemma 2 is the following corollary:

Corollary 2. For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

Proof. Due to Lemma 2 and updating rule for w , we have

Then we unroll the recursion in Lemma 2:

Besides the above lemma, we also need the following bound on tracking error between two consecutive iterates.

Lemma 3 (Second Order Tracking Error) . For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

Proof. For the ease of presentation, we define the following two auxiliary variables:

According to updating rule of z , we have:

Taking expectation w.r.t. ξ t j , ξ t i , ξ t -1 j and ξ t -1 i yields:

where in (1) and (3) we use Young's inequality that ∥ a + b ∥ 2 ≤ (1 + a ) ∥ a ∥ 2 +(1 + 1 a ) ∥ b ∥ 2 . Now we bound T 1 as follows:

and for T 2 :

Putting pieces together will conclude the proof.

The following lemma establishes the difference between the exact gradients computed on F and the gradients we actually used in Algorithm 1.

Lemma 4 (Gradient difference) . For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

Proof. By the definition of g t w , we have

where at the last step we use the bounded variance assumption. Similarly by the definition of g t α we have:

Lemma 5 (Connection between stationary measure and iterates) . For Algorithm 1, if we define

then the following statements hold:

Proof. We begin with proving the first statement. According to updating rule we have

where at last step we apply Lemma 4. Similarly, for the second statement we have:

The following lemma connects F ( α t +1 , w t +1 ) and F ( α t +1 , w t ) . Lemma 6 (Dual Variable One Iteration Analysis) . Let L := max { 4 G 2 f L g +2 G g L f , 2 C } . For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

Proof. By the optimality of projection and updating rule of w , we have:

Adding and subtracting ∇ w F ( α t , w t ) gives:

Applying Cauchy-Schwartz inequality yields:

Now, we take expectation over the randomness of ζ t T , ζ t j , and apply Lemma 4 to get:

Re-arranging terms yields:

Plugging in Lemma 2 yields:

On the other hand, from Proposition 2 on the smoothness of F with parameter L we have:

Putting pieces together will conclude the proof:

The following lemma connects F ( α t +1 , w t +1 ) and F ( α t , w t ) .

Lemma 7 (One Iteration Descent Lemma) . For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:

Proof. According to property of projection we have:

Since F ( · , w ) is strongly convex for any w ∈ W , we have:

where at second inequality we use the fact of (7) . For the first dot product term, we observe that:

For the second dot product, we have:

where we plug in Lemma 4 at last step.

For the third dot product, we use the following identity:

So we have:

which concludes the proof.

Lemma 8. For Algorithm 1, under the assumptions of Theorem 2, define the following auxiliary quantity ˆ F t +1 :

where

Then the following statement holds:

where

Proof. According to (6) and (7):

If we define g α ( z, α ) = [ g ( z 1 ) , ..., g ( z N )] + 2 C M α .

Since 1 η 〈 ∆ t +1 , α t -α t +1 〉 = 1 2 η ∥ ∥ α t -α t -1 ∥ ∥ 2 -1 2 η ∥ ∥ α t +1 -α t ∥ ∥ 2 -1 2 η ∥ ∥ ∆ t +1 ∥ ∥ 2 Hence we have:

Multiplying both sides with 4 µη yields:

and so, putting all together,

Now evoking Lemma 7 together with (14) yields:

Now we plug in Lemma 3 with the fact that 1 β ≥ 1 and get:

Recall our definition of potential function ˆ F t +1

We conclude that:

Lemma 9. For Algorithm 1, under the assumptions of Theorem 2, define the following auxiliary quantity ˜ F t +1 :

Let C1, C2 be defined in Lemma 8. If the following conditions hold

then the following statement hold:

Proof. According to Lemma 8:

where

Now we plug in Lemma 5:

Plugging in Lemma 2 yields:

Rearranging the terms yields:

Recall our definition of potential function ˜ F t +1 :

Hence we have:

Now we plug in Lemma 2 and 3:

By our assumption in (15)-(17):

then we can have the 'clean' bound:

A.3 Proof of theorem 2

Summing the inequality from t = 1 to T yields:

We compute the upper and lower bound of C 1 and C 2 . For C 1

The upper bound C 1 ≤ 1 4 γ holds trivially. For lower bound, since we choose

we know that C 1 ≥ 1 8 γ . For C 2 :

it holds that C 2 ≥ 1 2 η .

Since 1 8 γ ≤ C 1 ≤ 1 4 γ and 1 2 η ≤ C 2 ≤ 1 η + 3 µ 2 ≤ 2 η , we have:

We then need to verify the conditions (15), (16) and (17) in Lemma 9 can hold under our choice of η x , η y and β . To guarantee (15) holding, we need:

The upper bound C 2 ≤ 1 η + 3 µ 2 holds trivially. For lower bound, since we choose:

To guarantee (16) holding, we need:

To guarantee (17) holding, we need:

Next we examine how large the E [ ˜ F T -˜ F 0 ] is. By definition of potential function, we have:

By our choice z 0 j = z -1 j = f ̂ T ( w -1 ) -f j ( w -1 ) , w 0 = w -1 , we have E [ ˜ F T -˜ F 0 ] ≤ ˆ F T -ˆ F 0

Next we examine how large the E [ ˆ F T -ˆ F 0 ] is.

Notice that α -1 = α 0 , w 0 = w -1 = w -2 we have

.

According to updating rule,

Putting pieces together we can have the upper bound of E [ ˜ F T +1 -˜ F 1 ] :

Plugging above bound back to (19) yields:

Define κ = L/µ . Recall that we choose:

To guarantee RHS is less than ϵ 2 , we need:

which yields the total gradient complexity:

B Proofs for w ∗ Approximation Algorithm

B.1 Proof of Lemma 1

Proof. First, we define Φ( α , w ) = ∑ N j =1 α ( j ) · f j ( w ) . Indeed, Φ( · , w ) is √ NG f smooth for all w ∈ W . To see this, we consider

According to optimality conditions we have:

Substituting v with w ∗ ( α ′ ) and w ∗ ( α ) in the above first and second inequalities respectively yields:

Adding up the above two inequalities yields:

Since Φ( α , · ) is µ strongly convex, we have:

Adding up (20) and (21) yields:

Finally, using √ NG f smoothness of Φ will conclude the proof:

B.2 Preliminaries of the proof of theorem 3

In this section, we introduce some notational preliminaries for the proof for Theorem 3. Following the framework in [21], we introduce three models, neural network predictor, random feature predictor and kernel least square predictor. In the following sections we will use w ( j ) to denote the j th coordinate of vector w .

Two-layer ReLU neural network (NN) predictor Recall that in Section 3.1 we consider a two layer vector-valued neural network h θ : R N → R d :

where parameters of the hidden layer are matrices U i ∈ R m × N , collectively captured by the parameter vector θ = (vec( U 1 ) , . . . , vec( U d )) ∈ R dmN , and we also define θ j = vec( U j ) ∈ R Nm . Here a i ∈ {± 1 / √ m } m are parameters of the output layer .

Next, induce the Neural Tangent Feature (NTF) operator, defined as

where u j t ⊤ ( r ) is r th row of U j t . Note that we dropped dependence on α on the left hand side - it is always assumed that the operator is applied to variable α . Also, note that the operator comes by differentiation of the neural network with respect to a j -th component of the hidden layer, that is ϕ j t ( α ) = ∇ θ j h θ ( α ) | θ j = θ j t . For convenience, we also introduce the following NTF matrix notation, given multiple input vectors α 1 , . . . , α n :

Finally, it is not hard to see that the use of NTF operator recovers the neural network predictor:

Using the above definitions, we can write least squares objective as

and similarly for the pointwise loss function,

and for the GD update rule:

Neural Tangent Feature (NTF) predictor For parameter vector ¯ θ j t ∈ R Nd and ¯ θ t = ( ¯ θ 1 t , . . . , ¯ θ d t ) (the procedure for obtaining them is discussed later), we have the following NTF predictor defined as

We also use h rf ,j t ( α ) to indicate the j th coordinate of h rf t ( α ) . Then, the objective of the NTF predictor is

its pointwise counterpart is

and GD update rule is defined as

Kernel Least Squares (KLS) predictor In the following we will work with a kernel function: Proposition 3 (Kernel induced by Rectified Linear Unit (ReLU) activation) . The following kernel function is called the NTK function induced by the ReLU activation:

The following holds for κ :

It is a reproducing kernel and has analytic form κ ( α , α ′ ) = ( α ⊤ α ′ )( π -arccos( α ⊤ α ′ )) [15].

sup α , α ′ ∈ S d -1 κ ( α , α ′ ) ≤ 1 .

Eigenvalues of κ satisfy µ k ≤ C k -d 2 for k ∈ N [2].

The NTK matrix is a symmetric matrix K ∈ R n × n with entries ( K ) i,j = κ ( α i , α j ) .

Throughout, H is the RKHS induced by κ .

Having established the kernel function, we assume the following about matrix K :

Assumption 2 (Smallest eigenvalue of the NTK matrix) . Assume that there exists fixed λ 0 > 0 such that P ( λ min ( K ) ≥ λ 0 ) ≥ 1 -δ λ 0 and δ λ 0 ∈ [0 , 1] .

Assumption 2 is fairly standard and often satisfied with sample-dependent lower bounds:

In particular, [9] shows that λ 0 > 0 whenever no two distinct inputs are parallel.

In a random design setting, [4, Lemma 5.2] show that when inputs are sampled from isotropic Gaussian and n ≤ d C for some activation function-dependent C , λ 0 = Ω( d ) with probability at least 1 -e -O ( n ) .

In a random design setting, [29] show that (for a certain well-behaved family of input distributions), λ min ( K ) = Θ P ( d ) with high probability. More precisely, their Theorem 3.2 implies that we have λ min ( K ) ≥ C polylog( n, d ) d with probability at least 1 -n 2 e -O ( √ d ) .

In addition to the NTK matrix, we define its empirical counterpart for the j th component of the hidden layer, namely

Then, it is clear that K = E [ ˆ K j | data] .

Now, for some vector c ∈ R d , we define the KLS predictor as

We also use h kls ,j t ( α ) to indicate the j th coordinate of h kls t ( α ) . The objective for KLS predictor is defined as:

for the pointwise loss function as:

and finally GD update rule is defined as:

Proposition 4. Let w t i be iterates generated by Algorithm 2. Then the following statement holds true:

B.3 Properties of ReLU Networks

In this section we include standard lemmata for analysis of shallow neural networks, and where appropriate we account for the fact that we are working with vector-valued predictors.

Proposition 5 (Activation patterns [21]) . Assume that initial parameters of a neural network are chosen as in section 3.1. Consider the set of indices of neurons that changed their activation patterns on input α , when θ 0 is replaced by some parameters ˜ θ = (˜ u 1 , . . . , ˜ u m ) :

Then, for ˜ θ whose components satisfy max k ∥ ˜ u k -u 0 ,k ∥ ≤ ρ for some fixed ρ ≥ 0 , and any α ∈ ∆ N , the following facts hold:

Proposition 6 (Bounded gradient) . The gradients of neural network are bounded:

Proof. The proof simply follows by definition:

and

Lemma 10 (Spectral Property of Empirical Kernel Matrix) . Given a set of parameter θ t = { u t (1) , ..., u t ( m ) } , if we assume max r ∈ [ m ] ∥ u t ( r ) -u 0 ( r )) ∥ ≤ ρ , then the following statements about the spectral property of its random feature and empirical gram matrix hold true:

Also

Proof. We begin with proving ( a ) , the spectral norm ϕ t ( α i )

the spectral bound for Φ t .

so we know ∥ Φ t ∥ ≤ √ n

For ∥ Φ -Φ ∥

. t 0 , by definition we have:

Nowweshowthe bound on the norm of ˆ K t . Wefirst examine ∥ ∥ ϕ ⊤ t ( α i ) ϕ t ( α j ) -ϕ ⊤ 0 ( α i ) ϕ 0 ( α j ) ∥ ∥ F . Consider

Notice that due to triangle inequality we have ∥ ∥ ∥ ˆ K t ∥ ∥ ∥ ≤ ∥ ∥ ∥ ˆ K 0 ∥ ∥ ∥ + ∥ ∥ ∥ ˆ K t -ˆ K 0 ∥ ∥ ∥ . Now we examine ∥ ∥ ∥ ˆ K t -ˆ K 0 ∥ ∥ ∥ :

At last, we prove statement (d). According to Weyl's inequality:

According to basic concentration inequality we know that the following statement holds with probability at least 1 -ν :

Taking union bound over all n 2 entries and summing over all entries yields:

To ensure the RHS less than 1 2 λ 0 , we need

Last, since ∥ ∥ ∥ ˆ K 0 -K ∥ ∥ ∥ ≤ 1 2 λ 0 , we know that it is also true ∥ ∥ ∥ ˆ K 0 -K ∥ ∥ ∥ ≤ 1 2 σ max = 1 2 ∥ K ∥ .

Lemma 11 (Deviation of Gradient Computed on Inexact Labels) . For Algorithm 2, the following statements hold true:

Proof. Recall the definition of ∇ θ t ̂ R ( θ t ) and g t :

where w t ( j ) = [ w t 1 ( j ) , ..., w t n ( j )] ⊤ ∈ R n and w ∗ ( j ) = [ w ∗ ( α 1 )( j ) , ..., w ∗ ( α n )( j )] ⊤ ∈ R n . Hence

For the second result we have

The following theorem shows that Algorithm 2 can converge on empirical risk defined in (4). Theorem 5 (Convergence of Algorithm) . For Algorithm 2, if we choose

and assume the inner iteration number of Algorithm 1 satisfying

then the following statement holds for any t ∈ [ T ] and j ∈ [ d ] :

with

Proof. We prove by induction. We make the following inductive hypotheses:

where C is the constant to be determined later. First we are going to show hypothesis ( II ) holding for t +1 . First, by our choice of η , we know (1 -µγ ) ≤ ( 1 -ηλ 0 4 n ) . Hence

Now we are going to bound maximal neuron drifting from initialization: for any j ∈ [ d ]

where at last step we use the fact √ 1 -a ≤ 1 -a 2 for a ∈ (0 , 1) , and triangle inequality to split terms inside square root. Since we choose γ = 1 L , we have

According to our choice of K :

we have:

Hence we conclude that:

Now we switch to proving hypothesis ( I ). According to updating rule, we have

t t t

where w ∗ ( j ) = [ w ∗ ( α 1 )( j ) , ..., w ∗ ( α n )( j )] . For T 1 :

Notice that:

According to updating rule, we have:

Since we prove that max r ∈ [ m ] ∥ ∥ ∥ u j t +1 ( r ) -u j t ( r ) ∥ ∥ ∥ ≤ ρ , we have:

| P ( θ t +1 , α i ) | ≤ mρ + √ mν.

Putting pieces together yields:

For T 2 :

We choose q = n 2 ηλ min ( ˆ K t ) , and the fact that (1 + a )(1 -a ) 2 ≤ (1 -a ) :

Since we choose η ≤ n 2 λ min ( ˆ K t ) , we can re-write the above inequality as :

Plugging T 1 and T 2 back to (27), and using the eigenvalue lower bound from Lemma 10 (d) yields:

Now we plug in ρ = 48 nB y λ 0 √ m

According to our choice of m :

we know that

Plugging above inequality back to Eq.(29) yields:

According to the fact that:

we choose p = 8 n 3 ηλ 0 -1 , and it yields:

Using the fact η ≤ n λ 0 yields:

Since n 2 ≥ 4 3 , we know ηλ 0 6 n 3 ≤ ηλ 0 8 n . Hence we have:

̂ R ( θ t +1 )

Plugging lower bound of m ≥ ( 48 nB y λ 0 + √ ν ) 2 256 n 4 λ 2 0 yields:

Finally, according to inductive hypothesis ( I ), we plug in the convergence rate for ̂ R ( θ t ) and get

Now we just need to ensure that

We hence choose C as:

Hence we proved that:

with

B.4 Relations Between Different Predictors

In this section, we are going to relate the predictions of neural network, random feature predictor and kernel OLS predictor.

Lemma 12 (Neural Network Predictor and Random Feature Predictor Coupling) . Let ˜ n := ( 48 nB y λ 0 √ m + √ ν m ) . For Algorithm 2, the following statement holds for any j ∈ [ d ] and t ∈ [ T ] :

Proof. Notice the following canonical decomposition:

Define r t ( α ) = h j t ( α ) -w ∗ ( α )( j ) , and r rf t ( α ) = h rf ,j t ( α ) -w ∗ ( α )( j ) . Notice that

We define the following stacked vector representation :

Hence we can write the pointwise difference into the following compact form:

Unrolling the recursion to t = 0 yields:

Taking spectral norm on both sides yields:

We first bound T 1 as follows:

where we apply triangle inequality and Cauchy-Schwartz inequality. To bound ∥ ∥ ( Φ ⊤ s +1 -Φ ⊤ s ) θ t +1 ∥ ∥ , we evoke Eq.(28)):

Now we plug in the convergence rate from Theorem 5, and obtain

Putting pieces together yields:

Using the fact that 1 -λ 0 η n ≤ 1 -λ 0 η 4 n and our choice of learning rate such that 1 -µγ ≤ 1 -λ 0 η 4 n , we have:

For T 2 , similarly, we apply triangle and Cauchy-Schwartz inequality:

To bound ∥ ∥ ∥ Φ j s ⊤ ( ∇ θ j t ̂ R j ( θ j s ) -g s ) ∥ ∥ ∥ , we evoke Lemma 11:

Putting the above inequality back to T 2 yields:

For T 3 , we also apply Cauchy-Schwartz and get:

where at last step we plug in empirical gram matrix bound from Lemma 10. Now it suffices to plug in bound for ̂ R ( θ t )

Putting pieces together yields:

Lemma 13 (Neural network parameter and RF parameter coupling) . For Algorithm 2, the following statement holds for any j ∈ [ d ] and t ∈ [ T ] :

Proof. According to updating rule of neural network and random feature predictor, we have

Now taking norm on both side yields:

For T 1 :

where we plug in results from Lemma 10 and Theorem 5. For T 2 :

For T 3 , we evoke Lemma 11:

Putting pieces together we have:

Conducting telescoping sum yields:

Theorem 6. For Algorithm 2, the following statement holds true for any j ∈ [ d ] , t ∈ [ T ] :

If we choose K ≥ κ log ( η 3 T 2 n 3 D λ 3 0 ) , then we have:

Proof. We consider the following canonical decomposition:

For the first term, we

where we plug in Proposition 5 (e) to bound ∥ ∥ ∥ ϕ j t ( α ) ⊤ θ j t -ϕ j 0 ( α ) ⊤ θ j t ∥ ∥ ∥ , and use Lemma 13 to bound ∥ ∥ ∥ ϕ j 0 ( α ) ⊤ ( θ j t -¯ θ j t ) ∥ ∥ ∥ . Plugging in ρ = 48 nB y λ 0 √ m will complete bounding first term:

For the second term, we use the intermediate result from [21]

Lemma 14 (Approximation of Lipschitz functions on the ball [2, Proposition 6]) . For R larger than a constant c that depends only on N , for any function f ⋆ : R N → R such that for all x , ˜ x ∈ B N q , sup x ∈ B q N | f ⋆ ( x ) | ≤ Λ and | f ⋆ ( x ) -f ⋆ ( ˜ x ) | ≤ Λ ∥ x -˜ x ∥ q , there exists h ∈ H , such that ∥ h ∥ 2 H ≤ R and

Theorem 7 ([32, Theorem 1]) . Let F = { f : S d -1 → [0 , b ] } for some b < ∞ . Then with probability at least 1 -e -ν , ν > 0 , for all f ∈ F simultaneously

where the worst-case empirical Rademacher complexity is defined as

Lemma 15. [Rademacher complexity of kernel predictor class [3] Lemma 22] Consider the following function class with bounded RKHS norm:

Then its empirical Rademacher complexity is bounded by:

Lemma 16 (Excess risk of virtual KLS predictor learnt on optimal KLS approximator) . Let h kls ,j ∗ be the optimal approximator of f ∗ ,j in RKHS, and ˜ h kls ,j t ( α ) be KLS predictor trained by GD on sample { α i , h kls ,j ∗ ( α i ) } n i =1 . Then the following excess risk bound holds with probability at least 1 -e -ν :

Proof. Given a fixed function h kls ,j ∗ with ∥ ∥ ∥ h kls ,j ∗ ∥ ∥ ∥ 2 H ≤ R , we consider the following class:

According to constant shift property of Rademacher complexity [33] and Lemma 15, we know

According to Theorem 7, we have

Now, it suffices to bound (i) empirical risk term 1 n ∑ n i =1 ∥ ∥ ∥ ˜ h kls ,j t ( α i ) -h kls ,j ∗ ( α i ) ∥ ∥ ∥ 2 and (ii) RKHS norm of ˜ h kls ,j t .

For (i), we define ̂ R t = 1 n ∑ n i =1 ∥ ∥ ∥ ˜ h kls ,j t ( α i ) -h kls ,j ∗ ( α i ) ∥ ∥ ∥ 2 and recall the updating rule of kernel OLS predictor:

Hence we know that:

(ii) RKHS norm of ˜ h kls ,j t

Lemma 17. [21, Lemma 8] Let f κ t , ˜ f κ t be GD-trained KLS predictors (as discussed in appendix B.2) given training samples ( x i , y i ) n i =1 and ( x i , ˜ y i ) n i =1 respectively, where y i = f ⋆ ( x i ) + ε i and ˜ y i = h ( x i ) + ε i with ∥ h ∥ 2 H ≤ R characterized by lemma 14. Then, with A ( R ) defined in lemma 14, for any t ∈ N ,

B.5 Proof of Theorem 3

Proof. We notice the following risk decomposition:

where respectively we plug in the NN-KLS coupling result (Theorem 6), Lemma 17, excess risk of KLS (Lemma 16) and approximation error of RKHS to Lipschitz function (Lemma 14). By choosing m ≥ Ω( n 8+ 2 2+ N ) , T ≥ Ω ( n Nλ 2 0 η log( n ) ) and optimizing over R , we recover the rate:

Hence by summing over all coordinates j ∈ [ d ] we conclude the result:

where in our case lemma 1 implies Λ = κ ∗ . Finally we lower bound λ 0 using [29, Theorem 3.2] which implies that λ 0 = Ω(polylog( n, N ) N ) with probability at least 1 -n 2 e -O ( √ N ) for a distribution on a simplex.

C Regret Analysis of Algorithm 4

In this section, we are going to provide the proof for Theorem 4. Let us first introduce the following definitions that will be used in our proof.

Definition 2 (Covering and Packing Numbers) . An ε -cover of a set S w.r.t. some metric ∥ · ∥ is a set { x ′ 1 , . . . , x ′ n } ⊆ S such that for each x ∈ S there exists i ∈ { 1 , . . . , n } such that ∥ x -x ′ i ∥ ≤ ε . The covering number N ( S, ε, ∥ · ∥ ) is the smallest cardinality of a ε -cover. An ε -packing of a set S w.r.t. some metric ∥ · ∥ is a set { x ′ 1 , . . . , x ′ m } ⊆ S such that for any distinct i, j ∈ { 1 , . . . , m } , we have ∥ x ′ i -x ′ j ∥ > ε . The packing number M ( S, ε, ∥ · ∥ ) is the largest cardinality of a ε -packing.

It is well known that M ( S, 2 ε, ∥ · ∥ ) ≤ N ( S, ε, ∥ · ∥ ) ≤ M ( S, ε, ∥ · ∥ ) .

Definition 3 (Metric dimension) . The metric space ( X , ∥ · ∥ ) had metric dimension d , if there exists a constant C d such that for all ε > 0 , X has an ε -cover at most C d ε -d .

Proof. We start from the standard analysis framework for online non-parametric regression [13, 18, 20]. Let S t be the value of the variable S at the end of time t . Hence S 0 = ∅ . The functions π t : X → { 1 , . . . , t } for t = 1 , 2 , . . . map each data point x to its closest center (in norm ∥ · ∥ ) in S t -1 ,

̸

The set T s contain all data points α t that at time t belonged to the ball with center α s and radius ε t ,

Finally, w ⋆ s is the best fixed prediction for all examples ( α t , w t ) such that t ∈ T s ,

We proceed by decomposing the regret into estimation and approximation cumulative errors,

We study the cumulative regret within a fixed ball T s . We use T s ( t ) to denote the set of all points belong to ball with center w s , at time t . That is,

One property of T s ( t ) is that, if x t ∈ T s , then | T s ( t ) | = | T s ( t -1) | + 1 . We further define ˜ w t = 1 | T s ( t ) | ∑ j ∈ T s ( t ) w ∗ ( α j ) . According to [6, Lemma 3.1], we have:

According to the convergence rate of GD on strongly-convex and smooth function, the first term is bounded by p (1 -1 κ ) K D . For second term:

where at the second last step we use the fact that | T s ( t ) | = | T s ( t -1) | +1 .

We further sum above statement over all s ∈ S T :

The first inequality is a known bound on the regret under square loss [6, page 43]. We upper bound the size of the final packing S T as

Thus,

Next, we bound the approximation term. Using (30) we have

Note that ℓ t is 4 D -Lipschitz because y t , ˆ y t ∈ B 2 ( D ) . Hence,

ℓ t ( f ( α π t ( α t ) ) ) -ℓ t ( f ( α t ) ) ≤ 4 D ∣ ∣ f ( α π t ( α t ) ) -f ( α t ) ∣ ∣ ≤ 4 Dκ ∗ ∥ α π t ( α t ) -α t ∥ ≤ 4 Dε t by κ ∗ -Lipschitzness of f . Thus, putting all together,

and recalling that ε t = t -1 1+ N we have

which completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper motivates the mixture weight estimation problem using Multi-source Multi-target Domain Adaptation problem. More specifically, this paper considers how to estimate the optimal mixture of sources, given a target domain; also, when there are multiple target domains, how to solve empirical risk minimization (ERM) for each target using a possibly unique mixture of data sources in a computationally efficient manner. This paper tackles both problems by constructing new efficient algorithms with a convergence guarantee.\n\nSTRENGTHS:\nThis paper provides a rigorous theoretical analysis of the optimization algorithm proposed in the paper and considers both offline and online settings for the problem.\n\nWEAKNESSES:\nThis paper focuses more on the theoretical analysis of a specific optimization problem instead of addressing the Multi-source Multi-target Domain Adaptation (M2DA) problem.\n\n1. The bound in Theorem 1 can be quite loose; therefore, minimizing the right-hand side does not necessarily result in good weights in the target domain. As the right-hand side of this bound is minimized using empirical data, will there be an extra overfitting issue? Is there a way to bound this gap?\n\n2. If some training data from the target domain is available, i.e., $\\hat{\\mathcal{T}}$, why not use these target data in the training of $h$ so that you have N+1 dimensional domain weights?\n\n3. The relaxation from (1) to (2) seems arbitrary, especially the removal of the concave function of the square root.\n\n4. There are no experimental results to verify the effectiveness of the proposed algorithm, not even on synthetic data.\n\nQUESTIONS:\nI appreciate the theoretical contribution made in the paper by analyzing the convergence of the proposed stochastic corrected gradient descent ascent algorithm. However, the current way of presenting these results is nothing but motivating the specific convex-nonconcave minimax problem using a contrived application of M2DA, which seems to be quite off. I would suggest the authors revise the paper by focusing on the theoretical contribution made to the convex-nonconcave minimax problem considered in eq (2) and the co-component ERM problems in eq. (3), which are interesting in their own right. However, this may require a significant rewrite of the paper and even a change in the title.\n\nLIMITATIONS:\nThere are no experimental results to verify the effectiveness of the proposed algorithm, not even on synthetic data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors formulate the problem of optimizing mixture weights given the target domain as a compositional convex-concave minimax optimization problem. Then, they propose a stochastic descent ascent algorithm for solving the problem, which improves upon previous method of [31] by allowing stochastic updates. Then, authors address the second problem of: given a large number of target domain distributions, how to efficiently find model weights for all of target distributions? This is discussed in both offline and online setting. In the offline setting, a two-layer ReLU neural network is trained to output model weights; the model is essentially a \"hypernetwork\" that outputs model weights. In the online setting, nonparametric online regression method is adapted for the problem.\n\nSTRENGTHS:\nAuthors identify an important instance of a convex-nonconcave minimax problem. The learning of mixture weights has become both theoretically and empirically important topic of research. For example, methods like Group DRO (Sagawa et al https://arxiv.org/abs/1911.08731 ) and DoReMi (Xie et al, https://arxiv.org/abs/2305.10429 ) have been drawing attention. The reduction of this learning problem to an abstract convex-nonconcave optimization problem will facilitate the adoption of techniques from a broader optimization literature.\n\nAuthors also push the state of the art on convex-nonconcave minimax problem by developing a stochastic version of the algorithm, adopting recently developed techniques such as stochastic corrected gradients.\n\nWEAKNESSES:\nThe first half of the paper (mixture weight optimization) and the second half (model weight prediction) are related, but the connection is not very strong. These two ideas could've made good two separate papers. While making these results a single paper made this paper very rich with technical content, but on the other hand, readers of the main body of the paper shall learn much less than what they would with two separate papers. The rationale of authors seems to be that mixture weights for the second problem could be found from the algorithm from first half, but in this case, we already know model weights; thus, the second problem is not very well motivated from the first problem.\n\nThere is no numerical experiments in this paper, and hence results in the paper are not numerically validated. Also, this makes the practical utility of proposed algorithms less clear.\n\nQUESTIONS:\nThe offline version of the algorithm (Section 3.1) is quite different from the online version of the algorithm (Section 3.2). Would it be meaningful to compare them against each other? For ex, the offline version of the algorithm could be applied to online setting by occasionally computing the label, and the online version of the algorithm could be applied to offline setting as a baseline?\n\nIn line 139, two relaxations in 139-144 are explained as standard. But within which literature are these standard? Can authors provide references for such relaxations? \n\nCan stochastic versions of the Algorithm 2 exist (in terms of both $i$ and data points), which can alleviate the dependency on $M$?\n\nLIMITATIONS:\nWhat are alternative formulations for mixture weight estimation (1), and how would convex-nonconcave formulation compare against them? Practically, wouldn't it be too pessimistic to consider the supremum over $\\mathcal{H}$?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper is about the multi-source multi-target domain adaptation problem. The authors formulate a minimax algorithm to find the mixture weights of source domains. Furthermore, the authors extend it to the scenario of multi-target domains and introduce the co-component ERM problem. For this problem, this paper proposes algorithms to efficiently solve co-component ERM problems, in offline and online fashions.

STRENGTHS:
This paper extensively and theoretically studied the domain adaptation problem from the minimax optimization perspective. 

The paper gives the convergence analysis of the proposed algorithms.

For the multi-target domain scenario, the authors provide solutions to both the offline and online settings. The solution is more efficient than training each target domain adaptation independently.

WEAKNESSES:
This paper is fully theoretical. The research focus of this paper, i.e., domain adaptation, has many open benchmark datasets and thus it would be better to experimentally evaluate the proposed methods. 

In line 148, it is mentioned that a deterministic algorithm exists in the literature, while this paper is focused on the stochastic one. It would be good to elaborate more on the benefits of the stochastic one and meanwhile experimental compare them.

QUESTIONS:
As for the minimax optimization, in Eq.(1) and Eq.(2), why are the model/hypothesis parameters $\omega$ to maximize the difference between the target and source domain? Should it be the minimization?

LIMITATIONS:
No potential negative societal impact is found in this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a new way to compute mixture coefficients for combining multiple empirical risk minimization objectives (w.r.t. different sources in domain adaptation) in a way that takes into account the relation to a new target domain. As an application, the authors consider the multi-source multi-target domain adaptation scenario and solve it by predicting (the weights of) new target classifiers from the mixture weights.\n\nSTRENGTHS:\n- The considered phase transition is interesting.\n- The weights could be provided by a domain expert who is, e.g., certain about a physical relation between the domains.\n- Convergence of the algorithm as extension of [31] is interesting.\n\nWEAKNESSES:\n- No empirical intuition if the algorithm can be implemented with reasonable effort. I consider the result as of purely theoretical interest.\n- The error in the computation of the weights $\\alpha_i$ is not taken into account in the error rates results. This could dominate the convergence rates results.\n- My impression is that, if we are able to solve Eq. (1) efficiently, then also a separate training for each target domain should give us a comperable accuracy. I don't see any argument, neither theoretical or practical, which guarantees that the proposed lagorithm improves the separate learning (which is possible with labels in the target domain).\n\nQUESTIONS:\n- For computing $\\alpha$ in Eq. (1), one needs to compute differences in empirical risks (on target vs. source datasets). Errors in this difference seem to aggregate to errors in the solution for $\\alpha$. How does this effect the final prediction of the target model from mixture weights?\n- In meta-learning there is the approach of representing novel domains by \"meta-features\" and then map these meta-features to the hyper-parameters of novel target domains. The hyper-parameters can also be model weights, as in your case given by $w^\\ast(\\alpha)$ cited in the introducion. Consequently, the theory of meta-learning should apply also to your setting. How does this theory, e.g. [1], compare to your convergence results? Can the same phase transition be observed assuming the mixture weights are exact?\n\n[1] https://www.jmlr.org/papers/volume6/maurer05a/maurer05a.pdf\n\n------------------------------\nAfter rebuttal: My questions are addressed. I increase my score by 2.\n\nLIMITATIONS:\n- Influence of error of weight estimation should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\nThe paper addresses the problem of multi-source multi-target domain adaptation, where the goal is to learn a model from multiple sources in such a way that it performs well on a new target distribution. The context for this problem includes scenarios like learning from data collected from various sources (e.g., crowdsourcing) or in distributed systems with highly heterogeneous data. The two main unsolved problems in this context are: 1) how to estimate the optimal mixture of sources for a given target domain, and 2) how to efficiently solve empirical risk minimization for each target domain when there are numerous target domains, which can be computationally expensive. The paper proposes solutions to both of these problems using convex-nonconcave compositional minimax and overparameterized neural networks with provable guarantees. Additionally, an online algorithm for predicting parameters for new models given mixing coefficients is proposed.\n\nSTRENGTHS:\nTheoretical Contributions: The paper proposes novel approaches to tackle the problems of mixture weight estimation and empirical risk minimization, utilizing convex-nonconcave compositional minimax and overparameterized neural networks, respectively. These contributions are supported with provable guarantees, which add rigor to the proposed methods.\n\nEfficiency and Scalability: The paper emphasizes the efficiency of the proposed algorithms, particularly for mixture weight estimation and empirical risk minimization for multiple target domains. The avoidance of individual ERM for each target domain in certain cases helps reduce computational overhead.\n\nWEAKNESSES:\nComplexity: The proposed methods, such as convex-nonconcave compositional minimax and overparameterized neural networks, might be complex and difficult to implement for practitioners who are not familiar with these advanced techniques.\n\nApplicability to All Domains: The paper may not clearly address the limitations or specific domains where the proposed techniques might not be directly applicable or might require additional adjustments.\n\nEmpirical Evaluation: The paper lacks details about empirical evaluations, such as experiments on real datasets or comparisons with other state-of-the-art methods. This could raise concerns about the practical effectiveness of the proposed algorithms.\n\nQUESTIONS:\nHow does the proposed convex-nonconcave compositional minimax approach differ from existing methods used for mixture weight estimation?\n\nCan you provide more details on the theoretical guarantees of the proposed overparameterized neural network approach for empirical risk minimization and its relationship to the specific problem context?\n\nAre there any assumptions made about the data distribution or source/target domains that could limit the generalizability of the proposed methods?\n\nHow does the proposed method perform in practice?\n\nLIMITATIONS:\nLimited Empirical Validation: The lack of empirical evaluation or real-world case studies might raise questions about the practical effectiveness and applicability of the proposed methods to real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}