SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
O \left(\frac {\kappa L F _ {\mathrm {m a x}}}{\epsilon^ {2}} \cdot \max \left\{\frac {\kappa L \sigma^ {2}}{\epsilon^ {2}}, \frac {G _ {g} ^ {2} N \sigma^ {2}}{\epsilon^ {2}}, 1 \right\}\right).
$$
To the best of our knowledge, this is the first convergence proof for stochastic algorithm on solving strongly-convex-nonconcave problem. We achieve  $O(\epsilon^{-4})$  gradient complexity required to reach an  $\epsilon$  stationary point. In contrast to the most relevant result of [34], they show the rate  $O(\epsilon^{-2})$  for a deterministic Alternating Gradient Projection (AGP) in a strongly-convex-nonconcave setting. Note that our result also positively answers the question posed by [34], on whether some algorithm performing simultaneous instead of alternative updates can optimize strongly-convex-nonconcave minimax problem. Finally, compared to  $O(\epsilon^{-4})$  rate of SGDA given a nonconvex-strongly-concave problem [23], we need roughly same stochastic gradient evaluations.
3 Multiple Target Domains: Learning to Solve Co-component ERM
Up till now, the main focus was on the problem of learning good mixture parameters given a single target domain. Now we turn to a more general setting where we have  $M$  target domains, each associated with a different data distribution which necessitates per target mixture weights  $\alpha_{1},\ldots ,\alpha_{M}$ , either obtained by our algorithm, or provided by the user to guarantee good generalization on individual domains. Next, to get personalized models for these  $M$  domains, we have to solve  $M$  different ERM problems based on these mixture weights:
$$
\min  _ {\mathbf {w} \in \mathcal {W}} f _ {\boldsymbol {\alpha} _ {1}} (\mathbf {w}) := \sum_ {j = 1} ^ {N} \alpha_ {1} (j) f _ {j} (\mathbf {w}), \quad \dots , \quad \min  _ {\mathbf {w} \in \mathcal {W}} f _ {\boldsymbol {\alpha} _ {M}} (\mathbf {w}) := \sum_ {j = 1} ^ {N} \alpha_ {n} (j) f _ {j} (\mathbf {w});
$$
A naive way is to solve each of them, which will result in a computational complexity of  $M$  multiplied by the cost required to minimize each individual  $f_{\alpha_i}$  to a desired precision. We note that such a solution does not exploit the benign structure of these ERM problems: they share the same component functions  $(f_j)_j$ , and the only difference is in the mixture weights. It naturally motivates us to ask, can we propose an efficient algorithm which avoids solving all these  $M$  co-component ERM problems from scratch? Consider the solution of  $\min_{\mathbf{w} \in \mathcal{W}} f_{\alpha}(\mathbf{w}) := \sum_{j=1}^{N} \alpha(j) f_j(\mathbf{w})$  as a function of  $\alpha$ ,
$$
\mathbf {w} ^ {*} (\boldsymbol {\alpha}) := \arg \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {j = 1} ^ {N} \alpha (j) f _ {j} (\mathbf {w}). \tag {3}
$$
Fortunately, if we assume that each source empirical risk  $f_{j}$  is strongly convex and  $L_{f}$  smooth in model parameters, we have the following Lipschitz property (shown in appendix B.1):
Lemma 1. If each  $f_{j}$  is  $\mu_{f}$  strongly convex and  $L_{f}$  smooth, then  $\mathbf{w}^{*}(\cdot)$  is  $\kappa^{*} = \sqrt{N} G_{f} / \mu_{f}$  Lipschitz.
Some basic algebra shows that in the above example  $\mathbf{w}^{*}(\alpha)$  is indeed Lipschitz in  $\alpha$  with respect to  $\ell^2$  metric. The Lipschitz property allows us to learn  $\mathbf{w}^{*}$  efficiently. In particular, learning arbitrary Lipschitz (and bounded) vector-valued function  $\mathbf{w}^{*}: \mathbb{R}^{N} \to \mathbb{R}^{d}$  is an instance of a well-studied nonparametric regression problem [11]. In the following we will consider algorithms for learning  $\mathbf{w}^{*}$  in both offline and online setting and which are provably capable of estimating  $\mathbf{w}^{*}$  at an almost optimal rate. In offline setting, we assume that we have access to a subset of  $M$  mixture weights, say  $\alpha_{1}, \ldots, \alpha_{n}$ , and we shall use a two layer neural network  $\mathbf{h}_{\theta}(\cdot)$  to learn  $\mathbf{w}^{*}(\cdot)$ . Our algorithm is GD based empirical risk minimization with adaptive label refining. In a nutshell, given an  $\alpha$ , since we do not have access to  $\mathbf{w}^{*}(\alpha)$ , we will use gradient descent to jointly solve weighted ERM with  $\alpha$  to get an approximation of  $\mathbf{w}^{*}(\alpha)$  as well as optimizing neural network parameters. With a mild distributional assumption on  $\alpha$ , we show that our algorithm guarantees that the two layer network learns  $\mathbf{w}^{*}(\alpha)$ , that is, it achieves a small excess risk  $\mathbb{E}_{\alpha} \| \mathbf{h}_{\theta}(\alpha) - \mathbf{w}^{*}(\alpha) \|^2$ .
In online setting, we assume that we observe an arbitrary sequence  $\alpha_{1},\alpha_{2},\ldots$  on a simplex, and we wish to predict parameters close to  $\mathbf{w}^{*}(\alpha_{1}),\mathbf{w}^{*}(\alpha_{2}),\ldots$ . As baseline algorithm we will consider a well-known online nonparametric regression that greedily covers the simplex with local online learners and which enjoys almost-optimal regret [13]. However, in the considered online protocol, the algorithm will need access to labels, and revealing each label requires to solve (3) to some desired accuracy. Here we explore a possibility that in practice we might be satisfied with  $\epsilon$ -average regret, while saving the labelling cost. To this end we propose a modification of the algorithm that randomly skips some labels, while incurring a slightly larger regret.
3.1 Offline Setting: Learning Lipschitz function with ReLU Neural Network
In this section we consider offline learning of  $\mathbf{w}^{\star}$ . The Lipschitzness guarantees that the  $\mathbf{w}^{*}(\alpha)$  function can be efficiently learnt on finite  $\alpha$ s, and generalizable to unseen  $\alpha$ . Hence, we propose to use a vector-valued two layer ReLU neural network  $\mathbf{h}_{\theta}$  to learn  $\mathbf{w}^{*}(\alpha)$ .
We consider a two layer vector-valued neural network  $\mathbf{h}_{\pmb{\theta}}: \mathbb{R}^N \to \mathbb{R}^d$ ,  $\mathbf{h}_{\pmb{\theta}}(\mathbf{x}) = [\mathbf{a}_1^\top (\mathbf{U}^1\mathbf{x})_+,\dots,\mathbf{a}_d^\top (\mathbf{U}^d\mathbf{x})_+]$ , where parameters of the hidden layer are matrices  $\mathbf{U}^i \in \mathbb{R}^{m \times N}$ , collectively captured by the parameter vector  $\pmb{\theta} = (\mathrm{vec}(\mathbf{U}^1),\dots,\mathrm{vec}(\mathbf{U}^d)) \in \mathbb{R}^{dmN}$ . Here  $\mathbf{a}_i \in \{\pm 1 / \sqrt{m}\}^m$  are parameters of the output layer. In the following the hidden layer is tuned by Algorithm 2, while parameters of the output layer are fixed throughout training. We assume that at initialization, for each  $\mathbf{U}^i$ , the first half of its rows are drawn i.i.d. from isotropic standard Gaussian and the remaining half is identical to the first half. Similarly, for each  $\mathbf{a}_i$ , half of the entries are set
Algorithm 2: Learning  $\mathbf{w}^*$  function by a neural network
Input: Number of global iteration  $T$ , Number of Iteration for inner problem  $R$ .  
for  $t = 1, \dots, T$  do  
 $\boldsymbol{\theta}^{t + 1} = \boldsymbol{\theta}^t - \eta \nabla_{\boldsymbol{\theta}^t} \mathbf{h}_{\boldsymbol{\theta}^t}(\boldsymbol{\alpha}_i) (\mathbf{h}_{\boldsymbol{\theta}^t}(\boldsymbol{\alpha}_i) - \mathbf{w}_i^t)$   
for  $i = 1, \dots, n$  do  
 $\mathbf{w}_i^{t + 1} = \mathsf{GD}(\mathbf{w}_i^t, \boldsymbol{\alpha}_i, K)$ $\triangleright$  Label refining by  $K$ -step gradient descent
Algorithm 3: GD(v,  $\alpha ,K)$
Initialize  $\mathbf{v}^0 = \mathbf{v}$    
for  $t = 0,\dots,K - 1$  do  $\mathbf{v}^{t + 1} = \mathcal{P}_{\mathcal{W}}\left(\mathbf{v}^t -\gamma \sum_{j = 1}^N\alpha_i(j)\nabla f_j(\mathbf{v}^t)\right),$
Output:  $\mathbf{v}^K$
to  $-1 / \sqrt{m}$  and the rest to  $1 / \sqrt{m}$  (we assume that  $m$  is even). This initialization ensures that each output coordinate is 0 and so the empirical risk is bounded by a constant at initialization.
We assume that we observe mixture weights  $\alpha_{1},\ldots ,\alpha_{n}\in \Delta^{N}$  i.i.d. according to some underlying distribution  $\mathcal{U}$ . Such mixture weights can be obtained by Algorithm 1 and their independence means that samples originating from target domains are independent from each other.
We learn the neural network by solving the following Bi-level ERM:
$$
\min  _ {\boldsymbol {\theta}} \widehat {\mathcal {R}} (\boldsymbol {\theta}) := \frac {1}{n} \sum_ {i = 1} ^ {n} \| \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) \| ^ {2}, \quad \text {s . t .} \quad \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) = \arg \min  _ {\mathbf {w} \in \mathcal {W}} \sum_ {j = 1} ^ {N} \alpha_ {i} (j) f _ {j} (\mathbf {w}). \tag {4}
$$
The parameters of a neural network  $\theta$  should have the well-controlled excess risk on unseen  $\alpha$ :
$$
\mathcal {R} (\boldsymbol {\theta}) = \int_ {\Delta^ {N}} \| \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha}) - \mathbf {w} ^ {\star} (\boldsymbol {\alpha}) \| ^ {2} \mathrm {d} \mathcal {U} (\boldsymbol {\alpha}).
$$
To solve (4), we use a nested loop procedure which performs a GD step on a neural network objective  $\widehat{\mathcal{R}}$ , while in the inner loop we approximately find 'labels'  $\mathbf{w}^{*}(\alpha_{1}),\dots,\mathbf{w}^{*}(\alpha_{n})$  using a  $K$ -step GD. The entire procedure for solving (4) is described in Algorithm 2. Then, the following theorem shows that the two layer neural net optimized by Algorithm 2 learns  $\mathbf{w}^{*}(\alpha)$ :
Theorem 3. Let  $\lambda_0 = N\cdot \mathrm{polylog}(N,n)$ . Consider a neural network of with  $m\geq \Omega \left(n^{8 + \frac{2}{2 + N}}\right)$ . Then, for the Algorithm 2 with  $\eta \leq \frac{1}{2}$ ,  $\gamma = \frac{1}{L_f}$ ,  $T\geq \Omega \left(\frac{n}{N\lambda_0^2\eta}\log (n)\right)$ , and  $K\geq \Omega \left(\kappa \log \left(\frac{\eta T n D}{\lambda_0}\right)\right)$ , the following excess risk bound holds with probability at least 0.99:
$$
\mathcal {R} \left(\boldsymbol {\theta} _ {T + 1}\right) \leq O \left(\left(\kappa^ {*}\right) ^ {2} d n ^ {- \frac {2}{2 + N}}\right).
$$
The proof given in appendix B.5 is based on a more-or-less standard Neural Tangent Kernel (NTK) approximation argument [15, 4], namely we use the key fact that predictions made by a GD-trained overparameterized neural network are close to those made by a Kernelized Least-Squares (KLS) predictor (given that the width of the network is sufficiently large). Now, such a KLS GD-trained predictor can learn Lipschitz target functions: It is well known that by learning on a sufficiently large Reproducing kernel Hilbert space (RKHS) (with polynomial spectral decay), one can approximate Lipschitz functions well [8, 2]. Here our goal is to approximate a vector-valued function, however, by treating each output independently we follow existing proofs [14, 21] for scalar-valued Lipschitz regression by GD-trained neural networks and arrive at the same excess risk times  $d$ .
Optimality of our rate. Here we show that a two-layer neural network trained by a bi-level Gradient Descent (GD) can learn a vector-valued function with  $O\left( {{dn}^{-\frac{2}{2 + N}}}\right)$  excess risk. If we ignore the dependency on  $d$  ,our result matches the minimax rate of learning a scalar valued Lipschitz function [11].
Algorithm 4: Label efficient nonparametric online regression
Data: Radii schedule  $\varepsilon_1,\varepsilon_2\dots$  with  $\varepsilon_t\in \mathbb{R}_+$  , Label efficiency parameter  $p\in [0,1]$ $S\gets \emptyset$  . Set of centers   
for  $t = 1,2,\ldots$  do   
Observe  $\alpha_{t}$    
if  $S = \varnothing$  then   
 $|S\gets \{t\} ,T_t\gets \emptyset ;$  Create initial ball   
 $s\gets \arg \min_{s\in S}\| \pmb {\alpha}_t - \pmb {\alpha}_s\|$  Find active center if  $T_{s} = \varnothing$  then   
 $|\hat{\mathbf{w}}_t = \frac{1}{d'}\mathbf{1}$    
else   
 $|\hat{\mathbf{w}}_t\gets \frac{1}{|T_s|}\sum_{t'\in T_s}\mathbf{w}_{t'};$  Predict using active center   
if  $\| \alpha_{t} - \alpha_{s}\| \leq \varepsilon_{t}$  then   
 $|T_{s}\gets T_{s}\cup \{t\} ;$  Update list for active center   
else   
 $|S\gets S\cup \{s\} ,T_s\gets \emptyset ;$  Create new center   
Draw a Bernoulli random variable  $Z_{t}$  such that  $\mathbb{P}(Z_t = 1) = p$    
Observe  $\mathbf{w}_t = \mathbb{I}\left\{Z_t = 1\right\} \mathbf{GD}(\mathbf{0},\pmb {\alpha}_t,K)$  Update pseudo label for  $\alpha_{t}$
Efficiency of our learning-based approach. Given  $M$  mixture weights  $(\alpha_{i})_{i = 1}^{M}$ , the baseline naive approach is to solve all  $M$  weighted ERM with gradient descent, to accuracy level  $\epsilon$ , which requires  $\Theta (M\kappa \log (1 / \epsilon))$  time complexity. Using our approach, we first need to learn a neural network with  $\epsilon$  excess risk, and it needs  $n = \Theta\left((\kappa^{*2}d / \epsilon)^{1 + N / 2}\right)$  samples, which implies that we need to solve this many weighted ERM problems, resulting in a complexity of  $\Theta\left((\kappa^{*2}d / \epsilon)^{1 + N / 2}\cdot \kappa \log (1 / \epsilon)\right)$ . Once we learn a neural network, we just need to pay for the inference cost to predict  $\mathbf{w}^*$  for each  $\alpha_{i}$ . Putting things together, the total time complexity is  $\Theta\left((\kappa^{*2}d / \epsilon)^{1 + N / 2}\cdot \kappa \log (1 / \epsilon) + M\right)$ . We observe the following regimes:
- When  $M \geq \Omega \left( \frac{(\kappa^{*2} d / \epsilon)^{1 + N / 2} \cdot \kappa \log (1 / \epsilon)}{\kappa \log (1 / \epsilon) - 1} \right)$ , learning is more efficient than solving  $M$  ERMs.
- Otherwise, directly solving  $M$  ERMs is more efficient than learning a model predictor.
Intuitively, when the number of target domains is much larger than number of source domains, our learning based approach is strictly more efficient. It is also interesting to note that our learning based approach can avoid computational overhead of  $M$ , but suffers exponential cost from the number of sources  $N$ . While solving ERMs avoids the price for  $N$ , the computational cost increases linearly in terms of  $M$ .
3.2 Online Setting: Label Efficient Nonparametric Online Regression
In previous sections we discussed nonparametric offline learning of  $\mathbf{w}^*$  with distributional assumption on  $\alpha_{1},\ldots ,\alpha_{M}$ . In this section we consider the following online learning protocol with oblivious adversary. Given a known and fixed parameter  $p\in [0,1]$ , and an unknown sequence  $(\pmb {\alpha}_1,\mathbf{w}^* (\pmb {\alpha}_1)),(\pmb {\alpha}_2,\mathbf{w}^* (\pmb {\alpha}_2)),\dots \in \Delta^N\times \mathbb{B}_2^d (D)$  of inputs and labels, at every round  $t = 1,2,\ldots$
1. the environment reveals mixture weights  $\alpha_{t}\in \Delta^{N}$
2. the learner selects a label  $\hat{\mathbf{w}}_t\in \mathbb{B}_2^d (D)$  and incurs loss  $\ell_t\bigl (\hat{\mathbf{w}}_t\bigr) = \big\| \hat{\mathbf{w}}_t - \mathbf{w}^\star (\pmb {\alpha}_t)\big\| ^2$
3. the learner samples  $Z_{t} \sim \mathrm{Bern}(p)$  and observes  $\mathbb{I}\{Z_t = 1\} \mathbf{w}_t$ , when  $\mathbf{w}_t$  is a GD-optimized approximation of  $\mathbf{w}^{\star}(\alpha_{t})$ .
In particular, we introduce Algorithm 4, a modified version of the online nonparametric regression algorithm proposed by [13]. Algorithm 4 iteratively constructs a packing of  $\Delta^{N}$  using  $\ell^2$  balls centered on a subset of previously observed inputs. At each step  $t$ , the label (parameters) associated with the current input  $\alpha_{t}$  is predicted by averaging the labels of past inputs within the ball whose center  $\alpha_{s}$  is closest to  $\alpha_{t}$  in the  $\ell^2$  metric (note that labels are vector-valued). If  $\alpha_{t}$  lies outside the nearest ball, a new ball with center  $\alpha_{t}$  is created. The radii  $\varepsilon_{t}$  of all balls shrink at a rate  $t^{-1/(1+N)}$ .
Note that efficient (log in the number of centers) algorithms for approximate nearest-neighbor search are well-known [19], as well as highly-optimized open-source packages are readily available [16].
In contrast to the original algorithm by [13], where all sequentially observed labels are used to generate predictions (update local learners), our algorithm variant uses only a  $p$ -fraction of labels on average. This reduces label complexity at the cost of increased regret. This approach is referred to as label-efficient prediction in online learning [6, Sec. 6.2] and is beneficial when accessing labels is costly. The following theorem, shown in appendix C, establishes the regret bound of Algorithm 4.
Theorem 4 (Regret bound). Let  $f: \Delta^{N} \to \mathbb{R}^{d}$  be arbitrary  $\kappa^{*}$ -Lipschitz function with respect to  $\| \cdot \|$  metric and let  $C_N > 0$  be a metric-dependent constant. Then, Algorithm 4 with  $\varepsilon_{t} = t^{-\frac{1}{1 + N}}$  satisfies
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} (f (\boldsymbol {\alpha} _ {t}))\right) \right] \leq \left(p C _ {N} \ln (e T) + 4 D \kappa^ {*}\right) T ^ {\frac {N}{1 + N}} + (1 - p) T D + 4 p D T (1 - \kappa^ {- 1}) ^ {K} D
$$
Moreover, by definition  $\ell_t(\mathbf{w}^*(\boldsymbol{\alpha}_t)) = 0$ , so choosing  $f(\cdot) = \mathbf{w}^*$ , the above bound implies that:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \ell_ {t} (\hat {\mathbf {w}} _ {t}) \right] \leq 8 \left(p C _ {N} \ln (e T) + 4 D \kappa^ {*}\right) T ^ {- \frac {1}{1 + N}} + (1 - p) D + 4 p D \left(1 - \kappa^ {- 1}\right) ^ {K} D.
$$
The core idea behind the proof (deferred to appendix C) involves maintaining a balance between the regret contribution of each ball and an added regret term arising from approximating the target function using Voronoi partitioning. The regret contribution of each ball is logarithmic in the number of predictions made due to regret of online quadratic optimization [6, p. 42]. Ignoring log factors, the overall regret contribution equals the number of balls, which is essentially governed by the packing number with respect to the  $\ell^2$  metric. The additional term in the regret comes from the algorithm's prediction being constant within the Voronoi cells of  $\Delta^{N}$  induced by the current centers (considering that we predict using the nearest neighbor). Thus, an extra term equal to the product of the balls' radius and the Lipschitz constant is incurred. Finally, the label efficient algorithm we present here incurs yet another,  $p$ -dependent terms, which accounts for the missed labels.
Corollary 1. If our desired average regret is  $\epsilon >0$ , then Algorithm 4 has label complexity:
$$
\tilde {O} \left(\max  \left\{\left(\frac {D \kappa^ {*}}{\epsilon}\right) ^ {1 + N}, \left(\frac {1 - \epsilon / D}{\epsilon}\right) ^ {1 + N} \right\}\right) \left(1 - \frac {\epsilon}{D}\right).
$$
Note that we recover the standard version of the algorithm (non-label efficient) by trivially setting  $p = 1$ , which in contrast has label complexity of order  $\tilde{O}\big(\max \{(D\kappa^{*} / \epsilon)^{1 + N},(1 / \epsilon)^{1 + N}\} \big)$ , which is strictly larger than the label efficient version as long as  $\epsilon$  is not zero. When our desired regret goes to zero, the label complexity of two algorithms will tend to be the same asymptotically.
To demonstrate the effectiveness of our proposed mixture weights estimation algorithm, we conducted an experiment using MNIST dataset [1] according to the following specifications. We consider a scenario with 15 source distributions, and dividing them into 3 groups. For group 1, it contains 5 source distributions and each distribution contains 100 data (80 for training and 20 for testing) samples which are drawn uniformly randomly from class '0', '1' and '2'. Group 2 and 3 share similar settings but their distributions' data are drawn from class '3', '4', '5', and class '6', '7', '9', '10' respectively. The data generation process is summarized in Table 1.
Group	Classes	Domains per Group	Samples per Domain
1	0, 1, 2	5	100
2	3, 4, 5	5	100
3	6, 7, 8, 9	5	100
Table 1: Classes and samples per domain for each group
To demonstrate the effectiveness of our Algorithm 1, we implemented and run experiments with two-layer MLP neural network. We choose four different target setting: (1) target distribution
Target (Group 1)	Target (Group 2)	Target (Group 3)	Target (mix of Group 1 and 2)
Average ERM	69.9 %	40.0 %	34.9 %	59.9 %
Pure target training	69.9 %	55.0 %	40.0 %	55.0 %
Our method	80.0 %	69.9 %	55.0 %	65.0 %
Table 2: Accuracy comparison with two baseline algorithms. Each row represents the accuracy of model learnt by Average ERM, Pure target training or Our method, on different target domain. We can see that the models learnt using mixture weights from our algorithm (Algorithm 1) always yield the best accuracy.
from Group 1, (2) target distribution from Group 2, (3) target distribution from Group 3, (4) target distribution from the mix of Group 1 and Group 2. We compare three algorithms: 1. Weighted ERM using our learnt weights, 2. ERM on averaged weights and 3. ERM solely on target domain, and presented our findings in Table 2. The results indicate that the accuracy achieved using the learnt alphas outperforms the other two approaches.
In this paper we studied the multi-source multi-target domain adaptation problem. In the first part of the paper we gave an algorithm for addressing a minimax problem, that provably finds good mixture weights of source domains, given a single target domain. In the second part we studied the problem of domain adaptation with multiple target domains, and introduced the co-component ERM problem. We gave two concrete algorithms to solve co-component ERM problem, in offline and online settings. There are several potential future venues for future work, which we briefly discuss below.
Online mixture weight prediction Throughout Section 3, we assumed that the target domains'  $\alpha$ s are given. However, it would be interesting if given a new target domain, one could predict good mixture weight in an online fashion, and our Algorithm 1 could serve as an oracle to give the inexact label. Meanwhile, since the Algorithm 1 takes considerable time to converge, a desired online algorithm should also be label-efficient.
The complexity of solving co-component ERM The co-component ERM problem we introduced in section 3 is interesting from pure optimization perspective. Even though we proposed the learning-based approach to avoid heavy computation, one alternative direction is to develop efficient algorithms to directly solve  $M$  co-component ERM problems, and give upper and lower complexity bounds.
More structure in  $\mathbf{w}^{\star}$  and better phase transition laws In this work we considered a basic structure in co-component ERM problems (strong-convexity and smoothness), which gave rise to Lipschitzness of  $\mathbf{w}^{\star}$ . Lipschitz class of functions is very large and in general can only be learned at a rate  $\Theta(n^{-\frac{2}{2+N}})$ . As discussed in section 3.1 this allowed us to argue that the learning approach is more efficient than solving co-component ERM whenever  $M \geq \Omega((1/\epsilon)^{N/2})$ . However, we could potentially obtain better rates (and better laws) of learning  $\mathbf{w}^{\star}$  having more structure in  $\mathbf{w}^{\star}$ . For example, assuming that  $\mathbf{w}^{\star}$  is  $H$ -times differentiable, the excess risk would behave as  $\Theta(n^{-\frac{2H}{2H+N}})$  [11]. Thus, the learning approach would be more efficient when  $M \geq \Omega((1/\epsilon)^{N/(2H)})$ , that is with potentially much fewer sources than targets. It is an intriguing question to figure our which co-component ERM problems would allow for a nicer structure in  $\mathbf{w}^{\star}$ .
Alternative learning based approach for co-component ERM There may be several other learning based method to solve co-component ERM. One potential approach is Meta learning [10]. The idea is to train a meta model  $\mathbf{w}_{\mathrm{meta}}$  by optimizing a pre-defined meta objective based on some sampled mixture weights, and the goal would be to find a model that can quickly adapt to tasks with different mixture weights  $\alpha$ . We leave this as a promising open problem.