SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyperparameters to be optimized are the three regularization parameters  $\lambda_1, \lambda_2$  and  $\lambda_3$ . We performed 5-fold cross-validation to identify the best hyperparameter set from the candidate points;  $\{10^{-3}, 10^{-2}, 10^{-1}, 1\}$  for  $\lambda_1$  and  $\{10^{-2}, 10^{-1}, 1, 10\}$  for each of  $\lambda_2$  and  $\lambda_3$ .
To learn the AffineTL-full and AffineTL-const, we used the following objective functions:
$$
\text {A f f i n e T L - f u l l} \| \boldsymbol {y} - \left(K _ {1} \boldsymbol {a} + \left(K _ {2} \boldsymbol {b} + \boldsymbol {1}\right) \circ \left(K _ {3} \boldsymbol {c}\right) + d \boldsymbol {1}\right) \| _ {2} ^ {2} + \lambda_ {1} \boldsymbol {a} ^ {\top} K _ {1} \boldsymbol {a} + \lambda_ {2} \boldsymbol {b} ^ {\top} K _ {2} \boldsymbol {b} + \lambda_ {3} \boldsymbol {c} ^ {\top} K _ {3} \boldsymbol {c},
$$
$$
\text {A f f i n e T L - c o n s t} \frac {1}{n} \| \boldsymbol {y} - \left(K _ {1} \boldsymbol {a} + K _ {3} \boldsymbol {c} + d \mathbf {1}\right) \| _ {2} ^ {2} + \lambda_ {1} \boldsymbol {a} ^ {\top} K _ {1} \boldsymbol {a} + \lambda_ {3} \boldsymbol {c} ^ {\top} K _ {3} \boldsymbol {c}.
$$
Algorithm S.2 summarizes the block relaxation algorithm for AffineTL-full. For AffineTL-const, we found the optimal parameters as follows:
$$
\left[ \begin{array}{c} \hat {\boldsymbol {a}} \\ \hat {\boldsymbol {c}} \\ \hat {\boldsymbol {d}} \end{array} \right] = \left(\left[ \begin{array}{c} K _ {1} \\ K _ {3} \\ \mathbf {1} ^ {\top} \end{array} \right] \left[ \begin{array}{c c c} K _ {1} & K _ {3} & \mathbf {1} \end{array} \right] + \left[ \begin{array}{c c c} \lambda_ {1} K _ {1} & & \\ & \lambda_ {3} K _ {3} & \\ & & 0 \end{array} \right]\right) ^ {- 1} \left[ \begin{array}{c} K _ {1} \\ K _ {3} \\ \mathbf {1} ^ {\top} \end{array} \right] \boldsymbol {y}
$$
The stopping criterion for the algorithm was the same as Eq. (S.5).
Fine-tuning The target network was constructed by adding a one-dimensional output layer to the shared layers of the source network. As initial values for the training, we used the weights of the source neural network for the shared layer and the average of the multidimensional output layer of the source network for the output layer. Adagrad [41] was used for the optimization. The learning rate was fixed at 0.01 and the number of training epochs was selected from  $\{1,2,3,4,5,6,7,8,9,10,20,50,100\}$  through 5-fold cross-validation.
MAML A fully connected neural network with 256-64-32-16-1 layer width was used, and the initial values were searched through MAML [25] using the six source tasks. The obtained base model was fine-tuned with the target samples. Adam [42] with a fixed learning rate of 0.01 was used for the optimization. The number of training epochs was selected from  $\{1,2,3,4,5,6,7,8,9,10,20,50,100\}$  through 5-fold cross-validation.
$L^2$ -SP  $L^2$ -SP is a regularization method proposed by [26] in which the following regularization term is added so that the weights of the target network are estimated in the neighborhood of the weights of the source network:
$$
\Omega (\boldsymbol {w}) = \frac {\lambda}{2} \| \boldsymbol {w} - \boldsymbol {w} _ {s} \| _ {2} ^ {2}, \tag {S.6}
$$
where  $w$  and  $w_{s}$  are the weights of the target and source model, respectively, and  $\lambda > 0$  is a hyperparameter. We used the weights of the source network as the initial point for the training of the target model, and added a regularization parameter as in Eq. (S.6). Adagrad [41] were used for the optimizer, and the regularization parameters and learning rate were fixed at 0.01 and 0.001, respectively. The number of training epochs was selected from  $\{1,2,3,4,5,6,7,8,9,10,20,50,100\}$  through 5-fold cross-validation.
PAC-Net PAC-Net, proposed in [27], is a TL method that leverages pruning of the weights of the source network. Its training strategy consists of three steps: identifying the important weights in the source model, fine-tuning them using the source samples, and updating the remaining weights using the target samples.
Firstly, we pruned the bottom  $10\%$  of weights, based on absolute value, from the pre-trained source network. Following this, the remaining weights were retrained using the stochastic gradient descent (SGD). Finally, the pruned weights were retrained using target samples. For the final training phase, SGD with learning rate 0.01, was employed, and the number of training epochs was selected from  $\{1,2,3,4,5,6,7,8,9,10,20,50,100\}$  through 5-fold cross-validation.
C.2 Evaluation of Scientific Papers
C.2.1 Data
We used SciRepEval benchmark dataset for scientific documents, proposed in [23]. This dataset comprises abstracts, review scores, and decision statuses of papers submitted to various machine learning conferences. We conducted two primary tasks: predicting the average review score (a regression task) and determining the acceptance or rejection status of each paper (a binary classification task). The original input  $x$ , was represented as a two-gram bag-of-words vector derived from the abstract. As for source features  $f_{s}$ , we employed text embeddings from the abstracts, which were generated by various pre-trained language models, including BERT [28], SciBERT [29], T5 [30], and GPT-3 [31]. When building the vocabulary for the bag-of-words, we ignore phrases with document frequencies strictly higher than 0.9 or strictly lower than 0.01. Additionally, we eliminated certain stop-words using the default settings in scikit-learn [43]. The sentences containing URLs were removed from the abstracts because accepted papers tend to include GitHub links in their abstracts after acceptance, which may cause leakage of information to the prediction. The models were trained on a dataset comprising 8,166 instances, and their performance were subsequently evaluated on a test dataset of 2,043 instances.
C.2.2 Model Definition and Hyperparameter Search
For both the affine model transfer and feature extraction, we employed neural networks with ReLU activation and dropout layer with 0.1 dropout rate. The parameters were optimized using Adagrad [41] with 0.01 learning rate.
Affine Model Transfer For functions  $g_{1}$  and  $g_{2}$  in the affine model transfer, we used a neural network composed of layers with widths 128, 64, 32, and 16, wherein the source features  $f_{s}$  were used as inputs. The number of layers of each width was determined based on Bayesian optimization. Sigmoid activation was employed to the output of  $g_{2}$  in order to facilitate the interpretation of  $g_{3}$ . For  $g_{3}$ , we employed a linear model with the input  $x$ . To prevent overfitting and promote model simplicity, we applied  $\ell_{1}$  regularization to the parameters of  $g_{3}$  with a regularization parameter of 0.01. The final output of the model was computed as  $g_{1} + g_{2} \cdot g_{3}$ . In the case of the binary classification task, we applied sigmoid activation function to this final output.
Feature Extraction As in the affine model transfer, we used a neural network composed of layers with widths 128, 64, 32, and 16, wherein the source features  $f_{s}$  were used as inputs. The number of layers of each width was determined based on Bayesian optimization. In the case of the binary classification task, we applied sigmoid activation function to the final output.
D.1 Proof of Theorem 2.4
Proof. According to Assumption 2.3, it holds that for any  $p_t(y|x)$ ,
$$
\psi_ {f _ {s}} \left(\int \phi_ {f _ {s}} (y) p _ {t} (y | \boldsymbol {x}) \mathrm {d} y\right) = \int y p _ {t} (y | \boldsymbol {x}) \mathrm {d} y. \tag {S.7}
$$
(i) Let  $\delta_{y_0}$  be the Dirac delta function supported on  $y_0$ . Substituting  $p_t(y|\boldsymbol{x}) = \delta_{y_0}$  into Eq. (S.7), we have
$$
\psi_ {f _ {s}} \left(\phi_ {f _ {s}} \left(y _ {0}\right)\right) = y _ {0} (\forall y _ {0} \in \mathcal {Y}).
$$
Under Assumption 2.3, this implies the property (i).
(ii) For simplicity, we assume the inputs  $\pmb{x}$  are fixed and  $p_t(y|\pmb{x}) > 0$ . Applying the property (i) to Eq. (S.7) yields
$$
\int \phi_ {f _ {s}} (y) p _ {t} (y | \boldsymbol {x}) \mathrm {d} y = \phi_ {f _ {s}} \left(\int y p _ {t} (y | \boldsymbol {x}) \mathrm {d} y\right).
$$
We consider a two-component mixture  $p_t(y|x) = (1 - \epsilon)q(y|x) + \epsilon h(y|x)$  with a mixing rate  $\epsilon \in [0,1]$ , where  $q$  and  $h$  denote arbitrary probability density functions. Then, we have
$$
\int \phi_ {f _ {s}} (y) \left\{(1 - \epsilon) q (y | \boldsymbol {x}) + \epsilon h (y | \boldsymbol {x}) \right\} \mathrm {d} y = \phi_ {f _ {s}} \left(\int y \left\{(1 - \epsilon) q (y | \boldsymbol {x}) + \epsilon h (y | \boldsymbol {x}) \right\} \mathrm {d} y\right).
$$
Taking the derivative at  $\epsilon = 0$ , we have
$$
\int \phi_ {f _ {s}} (y) \left\{h (y | \boldsymbol {x}) - q (y | \boldsymbol {x}) \right\} \mathrm {d} y = \phi_ {f _ {s}} ^ {\prime} \left(\int y q (y | \boldsymbol {x}) \mathrm {d} y\right) \left(\int y \left\{h (y | \boldsymbol {x}) - q (y | \boldsymbol {x}) \right\} \mathrm {d} y\right),
$$
$$
\int \left\{h (y | \boldsymbol {x}) - q (y | \boldsymbol {x}) \right\} \left\{\phi_ {f _ {s}} (y) - \phi_ {f _ {s}} ^ {\prime} \left(\mathbb {E} _ {q} [ Y | \boldsymbol {X} = \boldsymbol {x} ]\right) y \right\} \mathrm {d} y = 0. \tag {S.8}
$$
For Eq. (S.8) to hold for any  $q$  and  $h$ ,  $\phi_{f_s}(y) - \phi_{f_s}'(\mathbb{E}_q[Y|\mathbf{X} = \mathbf{x}])y$  must be independent of  $y$ . Thus, the function  $\phi_{f_s}$  and its inverse  $\psi_{f_s} = \phi_{f_s}^{-1}$  are limited to affine transformations with respect to  $y$ . Since  $\phi$  depends on  $y$  and  $f_s(x)$ , it takes the form  $\phi(y, f_s(x)) = g_1(f_s(x)) + g_2(f_s(x))y$ .
D.2 Proof of Theorem 4.1
To bound the generalization error, we use the empirical and population Rademacher complexity  $\mathfrak{R}_S(\mathcal{F})$  and  $\Re (\mathcal{F})$  of hypothesis class  $\mathcal{F}$ , defined as:
$$
\hat {\mathfrak {R}} _ {S} (\mathcal {F}) = \mathbb {E} _ {\sigma} \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} f (\boldsymbol {x} _ {i}), \quad \mathfrak {R} (\mathcal {F}) = \mathbb {E} _ {S} \hat {\mathfrak {R}} _ {S} (\mathcal {F}),
$$
where  $\{\sigma_i\}_{i = 1}^n$  is a set of Rademacher variables that are independently distributed and each take one of the values in  $\{-1, + 1\}$  with equal probability, and  $S$  denotes a set of samples. The following proof is based on the one of Theorem 11 shown in [11].
Proof of Theorem 4.1. For any hypothesis class  $\mathcal{F}$  with feature map  $\Phi$  where  $\| \Phi \| ^2\leq 1$ , the following inequality holds:
$$
\mathbb{E}_{\sigma}\sup_{\| \boldsymbol {\theta}\|^{2}\leq \Lambda}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\langle \boldsymbol {\theta},\Phi (\boldsymbol{x}_{i})\rangle \leq \sqrt{\frac{\Lambda}{n}}.
$$
The proof is given, for example, in Theorem 6.12 of [44]. Thus, the empirical Rademacher complexity of  $\tilde{\mathcal{H}}$  is bounded as
$$
\begin{array}{l} \hat{\mathfrak{R}}_{S}(\tilde{\mathcal{H}}) = \mathbb{E}_{\sigma}\sup_{\substack{\| \boldsymbol {\alpha}\|_{\mathcal{H}_{1}}^{2}\leq \lambda_{\boldsymbol{\alpha}}^{-1}\hat{R}_{s},\\ \| \boldsymbol {\beta}\|_{\mathcal{H}_{2}}^{2}\leq \lambda_{\boldsymbol{\beta}}^{-1}\hat{R}_{s},\\ \| \boldsymbol {\gamma}\|_{\mathcal{H}_{3}}^{2}\leq \lambda_{\boldsymbol{\gamma}}^{-1}\hat{R}_{s}}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\Bigg\{\langle \boldsymbol {\alpha},\Phi_{1}(\boldsymbol {f}_{\boldsymbol{s}}(\boldsymbol{x}_{i}))\rangle_{\mathcal{H}_{1}} + \langle \boldsymbol {\beta},\Phi_{2}(\boldsymbol {f}_{\boldsymbol{s}}(\boldsymbol{x}_{i}))\rangle_{\mathcal{H}_{2}}\langle \boldsymbol {\gamma},\Phi (\boldsymbol{x}_{i})\rangle_{\mathcal{H}_{3}}\Bigg\} \\ \leq \mathbb {E} _ {\sigma} \sup  _ {\| \boldsymbol {\alpha} \| _ {\mathcal {H} _ {1}} ^ {2} \leq \lambda_ {\boldsymbol {\alpha}} ^ {- 1} \hat {R} _ {s}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \langle \boldsymbol {\alpha}, \Phi_ {1} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x} _ {i})) \rangle_ {\mathcal {H} _ {1}} \\ + \sup  _ { \begin{array}{c} \| \boldsymbol {\beta} \| _ {\mathcal {H} _ {2}} ^ {2} \leq \lambda_ {\boldsymbol {\beta}} ^ {- 1} \hat {R} _ {s}, \\ \| \boldsymbol {\gamma} \| _ {\mathcal {H} _ {3}} ^ {2} \leq \lambda_ {\boldsymbol {\gamma}} ^ {- 1} \hat {R} _ {s} \end{array} } \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \langle \boldsymbol {\beta} \otimes \boldsymbol {\gamma}, \Phi_ {2} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x} _ {i})) \otimes \Phi (\boldsymbol {x} _ {i}) \rangle_ {\mathcal {H} _ {2} \otimes \mathcal {H} _ {3}} \\ \leq \mathbb{E}_{\sigma}\sup_{\| \boldsymbol {\alpha}\|_{\mathcal{H}_{1}}^{2}\leq \lambda_{\boldsymbol{\alpha}}^{-1}\hat{R}_{s}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\langle \boldsymbol {\alpha},\Phi_{1}(\boldsymbol{f}_{\boldsymbol{s}}(\boldsymbol{x}_{i}))\rangle_{\mathcal{H}_{1}} \\ + \sup  _ {\| \boldsymbol {\beta} \otimes \boldsymbol {\gamma} \| _ {\mathcal {H} _ {2} \otimes \mathcal {H} _ {3}} ^ {2} \leq \lambda_ {\boldsymbol {\beta}} ^ {- 1} \lambda_ {\boldsymbol {\gamma}} ^ {- 1} \hat {R} _ {s} ^ {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \langle \boldsymbol {\beta} \otimes \boldsymbol {\gamma}, \Phi_ {2} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x} _ {i})) \otimes \Phi (\boldsymbol {x} _ {i}) \rangle_ {\mathcal {H} _ {2} \otimes \mathcal {H} _ {3}} \\ \leq \sqrt {\frac {\hat {R} _ {s}}{\lambda_ {\boldsymbol {\alpha}} n}} + \sqrt {\frac {\hat {R} _ {s} ^ {2}}{\lambda_ {\beta} \lambda_ {\gamma} n}} \tag {S.9} \\ \leq \sqrt {\frac {\hat {R} _ {s}}{n}} \left\{\sqrt {\frac {1}{\lambda_ {\alpha}}} + \sqrt {\frac {L}{\lambda_ {\beta} \lambda_ {\gamma}}} \right\}. \\ \end{array}
$$
The first inequality follows from the subadditivity of supremum. The last inequality follows from the fact that  $\hat{R}_s \leq P_n \ell(y, \langle 0, \Phi_1 \rangle) + \lambda_\alpha \|0\|^2 \leq L$ .
Let  $C = \sqrt{\frac{1}{\lambda_{\alpha}}} + \sqrt{\frac{L}{\lambda_{\beta} \lambda_{\gamma}}}$ , and applying Talagrand's lemma [44] and Jensen's inequality, we obtain
$$
\mathfrak {R} (\mathcal {L}) = \mathbb {E} \hat {\mathfrak {R}} _ {S} (\mathcal {L}) \leq \mu_ {\ell} \mathbb {E} \hat {\mathfrak {R}} _ {S} (\tilde {\mathcal {H}}) \leq C \mu_ {\ell} \mathbb {E} \sqrt {\frac {\hat {R} _ {s}}{n}} \leq C \mu_ {\ell} \sqrt {\frac {\mathbb {E} \hat {R} _ {s}}{n}}.
$$
To apply Corollary 3.5 of [20], we should solve the equation
$$
r = C \mu_ {\ell} \sqrt {\frac {r}{n}}, \tag {S.10}
$$
and obtain  $r^* = \frac{\mu_\ell^2 C^2}{n}$ . Thus, for any  $\eta > 0$ , with probability at least  $1 - e^{-\eta}$ , there exists a constant  $C' > 0$  that satisfies
$$
P _ {n} \ell (y, h) \leq C ^ {\prime} \left(\mathbb {E} \hat {R} _ {s} + \frac {\mu_ {\ell} ^ {2} C ^ {2}}{n} + \frac {\eta}{n}\right) \leq C ^ {\prime} \left(R _ {s} + \frac {\mu_ {\ell} ^ {2} C ^ {2}}{n} + \frac {\eta}{n}\right). \tag {S.11}
$$
Note that, for the last inequality, because  $\hat{R}_s \leq P_n\ell(y,\langle\alpha,\Phi_1\rangle) + \lambda_\alpha\|\alpha\|^2$  for any  $\alpha$ , taking the expectation of both sides yields  $\mathbb{E}\hat{R}_s \leq P\ell(y,\langle\alpha,\Phi_1\rangle) + \lambda_\alpha\|\alpha\|^2$ , and this gives  $\mathbb{E}\hat{R}_s \leq \inf_\alpha\{P\ell(y,\langle\alpha,\Phi_1\rangle) + \lambda_\alpha\|\alpha\|^2\} = R_s$ . Consequently, applying Theorem 1 of [45], we have
$$
P \ell (y, h (\boldsymbol {x})) \leq P _ {n} \ell (y, h (\boldsymbol {x})) + \tilde {O} \left(\left(\sqrt {\frac {R _ {s}}{n}} + \frac {\mu_ {\ell} C + \sqrt {\eta}}{n}\right) \left(\sqrt {L} C + \sqrt {L \eta}\right) + \frac {C ^ {2} L + L \eta}{n}\right). \tag {S.12}
$$
Here, we use  $\hat{\Re}_S(\tilde{\mathcal{H}})\leq C\sqrt{\frac{\hat{R}_s}{n}}\leq C\sqrt{\frac{L}{n}}$
Remark D.1. As in [11], without the estimation of the parameters  $\alpha$  and  $\beta$ , the right-hand side of Eq. (S.9) becomes  $\frac{1}{\sqrt{n}}\left(c_1 + c_2\sqrt{\hat{R}_s}\right)$  with some constant  $c_{1} > 0$  and  $c_{2} > 0$ , and Eq. (S.10) becomes
$$
r = \frac {1}{\sqrt {n}} (c _ {1} + c _ {2} \sqrt {r}).
$$
This yields the solution
$$
r ^ {*} = \left(\frac {c _ {2}}{2 \sqrt {n}} + \sqrt {\left(\frac {c _ {2}}{2 \sqrt {n}}\right) ^ {2} + \frac {c _ {1}}{\sqrt {n}}}\right) ^ {2} \leq \frac {c _ {2} ^ {2}}{n} + \frac {c _ {1}}{\sqrt {n}},
$$
where we use the inequality  $\sqrt{x} + \sqrt{x + y} \leq \sqrt{4x + 2y}$ . Thus, Eq. (S.11) becomes
$$
P _ {n} \ell (y, h) \leq C ^ {\prime} \left(R _ {s} + \frac {c _ {2} ^ {2}}{n} + \frac {c _ {1}}{\sqrt {n}} + \frac {\eta}{n}\right).
$$
Consequently, we have the following result:
$$
\begin{array}{l} P \ell (y, h (\boldsymbol {x})) \leq P _ {n} \ell (y, h (\boldsymbol {x})) \\ + \tilde {O} \bigg (\bigg (\sqrt {\frac {R _ {s}}{n}} + \frac {\sqrt {c _ {1}}}{n ^ {3 / 4}} + \frac {c _ {2} + \sqrt {\eta}}{n} \bigg) \bigg (c _ {1} + c _ {2} \sqrt {L} + \sqrt {L \eta} \bigg) + \frac {(c _ {1} + c _ {2} \sqrt {L}) ^ {2} + L \eta}{n} \bigg). \\ \end{array}
$$
This means that even if  $R_{s} = \tilde{O}(n^{-1})$ , the resulting rate only improves to  $\tilde{O}(n^{-3/4})$ .
D.3 Proof of Theorem 4.4
Recall that loss function  $\ell(\cdot, \cdot)$  is assumed to be  $\mu_{\ell}$ -Lipschitz for the first argument. In addition, we impose the following assumption.
Assumption D.2. There exists a constant  $B \geq 1$  such that for every  $h \in \mathcal{H}$ ,  $P(h - h^{*}) \leq BP(\ell(y, h) - \ell(y, h^{*}))$ .
Because we consider  $\ell(y, y') = (y - y')^2$  in Theorem 4.4, Assumption D.2 holds as stated in [20].
First, we show the following corollary, which is a slight modification of Theorem 5.4 of [20].
Corollary D.3. Let  $\hat{h}$  be any element of  $\mathcal{H}$  satisfying  $P_{n}\ell(y,\hat{h}) = \inf_{h\in \mathcal{H}}P_{n}\ell(y,h)$ , and let  $\hat{h}^{(m)}$  be any element of  $\mathcal{H}^{(m)}$  satisfying  $P_{n}\ell(y,\hat{h}^{(m)}) = \inf_{h\in \mathcal{H}^{(m)}}P_{n}\ell(y,h)$ . Define
$$
\hat {\psi} (r) = c _ {1} \hat {\Re} _ {S} \{h \in \mathcal {H}: \max  _ {m \in \{1, 2 \}} P _ {n} (h ^ {(m)} - \hat {h} ^ {(m)}) ^ {2} \leq c _ {3} r \} + \frac {c _ {2} \eta}{n},
$$
where  $c_{1}, c_{2}$  and  $c_{3}$  are constants depending only on  $B$  and  $\mu_{\ell}$ . Then, for any  $\eta > 0$ , with probability at least  $1 - 5e^{-\eta}$ ,
$$
P \ell (y, \hat {h}) - P \ell (y, h ^ {*}) \leq \frac {7 0 5}{B} \hat {r} ^ {*} + \frac {(1 1 \mu_ {\ell} + 2 7 B) \eta}{n},
$$
where  $\hat{r}^*$  is the fixed point of  $\hat{\psi}$ .
Proof. Define the function  $\psi$  as
$$
\psi (r) = \frac {c _ {1}}{2} \Re \left\{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max  P \left(h ^ {(m)} - h ^ {(m) *}\right) ^ {2} \leq r \right\} + \frac {\left(c _ {2} - c _ {1}\right) \eta}{n}.
$$
Because  $\mathcal{H}$ ,  $\mathcal{H}^{(1)}$  and  $\mathcal{H}^{(2)}$  are all convex and thus star-shaped around each of its points, Lemma 3.4 of [20] implies that  $\psi$  is a sub-root. Also, define the sub-root function  $\psi_{m}$  as
$$
\psi_ {m} (r) = \frac {c _ {1} ^ {(m)}}{2} \Re \{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq r \} + \frac {(c _ {2} - c _ {1}) \eta}{n}.
$$
Let  $r_m^*$  be the fixed point of  $\psi_{m}(r_{m})$ . Now, for  $r_m \geq \psi_m(r_m)$ , Corollary 5.3 of [20] and the condition on the loss function imply that, with probability at least  $1 - e^{-\eta}$ ,
$$
\mu_ {\ell} ^ {2} P (\hat {h} ^ {(m)} - h ^ {(m) *}) ^ {2} \leq B \mu_ {\ell} ^ {2} P (\ell (y, \hat {h} ^ {(m)}) - \ell (y, \hat {h} ^ {(m) *})) \leq 7 0 5 \mu_ {\ell} ^ {2} r _ {m} + \frac {(1 1 \mu_ {\ell} + 2 7 B) B \mu_ {\ell} ^ {2} \eta}{n}.
$$
Denote the right-hand side by  $s_m$ , and define  $r = \max r_m$  and  $s = \max s_m$ . Because  $s \geq s_m \geq r_m \geq r_m^*$ , we obtain  $s \geq \psi_m(s)$  according to Lemma 3.2 of [20], and thus,
$$
s \geq 1 0 \mu_ {\ell} ^ {2} \Re \{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq s \} + \frac {1 1 \mu_ {\ell} ^ {2} \eta}{n}.
$$
Therefore, applying Corollary 2.2 of [20] to the class  $\mu_{\ell}\mathcal{H}^{(m)}$ , it follows that with probability at least  $1 - e^{-\eta}$
$$
\{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq s \} \subseteq \{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P _ {n} (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq 2 s \}.
$$
This implies that with probability at least  $1 - 2e^{-\eta}$
$$
\begin{array}{l} P _ {n} \left(\hat {h} ^ {(m)} - h ^ {(m) *}\right) ^ {2} \leq 2 \left(7 0 5 r + \frac {\left(1 1 \mu_ {\ell} + 2 7 B\right) B \eta}{n}\right) \\ \leq 2 \left(7 0 5 + \frac {(1 1 \mu_ {\ell} + 2 7 B) B}{n}\right) r, \\ \end{array}
$$
where the second inequality follows from  $r \geq \psi(r) \geq \frac{c_2\eta}{n}$ . Define  $2\left(705 + \frac{(11\mu_{\ell} + 27B)B}{n}\right) = c'$ . According to the triangle inequality in  $L_2(P_n)$ , it holds that
$$
\begin{array}{l} P _ {n} (h ^ {(m)} - \hat {h} ^ {(m)}) ^ {2} \leq \left(\sqrt {P _ {n} (h ^ {(m)} - h ^ {(m) *}) ^ {2}} + \sqrt {P _ {n} (h ^ {(m) *} - \hat {h} ^ {(m)}) ^ {2}}\right) ^ {2} \\ \leq \left(\sqrt {P _ {n} (h ^ {(m)} - h ^ {(m) *}) ^ {2}} + \sqrt {c ^ {\prime} r}\right) ^ {2}. \\ \end{array}
$$
Again, applying Corollary 2.2 of [20] to  $\mu_{\ell}\mathcal{H}^{(m)}$  as before, but now for  $r\geq \psi_m(r)$ , it follows that with probability at least  $1 - 4e^{-\eta}$
$$
\begin{array}{l} \{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max  P (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq r \} \\ = \bigcap_ {m = 1} ^ {2} \left\{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P \left(h ^ {(m)} - h ^ {(m) *}\right) ^ {2} \leq r \right\} \\ \subseteq \bigcap_ {m = 1} ^ {2} \left\{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P _ {n} \left(h ^ {(m)} - h ^ {(m) *}\right) ^ {2} \leq 2 r \right\} \\ \subseteq \bigcap_ {m = 1} ^ {2} \left\{h ^ {(m)} \in \mathcal {H} ^ {(m)}: \mu_ {\ell} ^ {2} P _ {n} \left(h ^ {(m)} - \hat {h} ^ {(m)}\right) ^ {2} \leq \left(\sqrt {2 r} + \sqrt {c ^ {\prime} r}\right) ^ {2} \right\} \\ = \left\{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max  P _ {n} \left(h ^ {(m)} - \hat {h} ^ {(m)}\right) ^ {2} \leq c _ {3} r \right\}, \\ \end{array}
$$
where  $c_{3} = (\sqrt{2} + \sqrt{c^{\prime}})^{2}$ . Combining this with Lemma A.4 of [20] leads to the following inequality: with probability at least  $1 - 5e^{-x}$
$$
\begin{array}{l} \psi (r) = \frac {c _ {1}}{2} \Re \left\{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max  P \left(h ^ {(m)} - h ^ {(m) *}\right) ^ {2} \leq r \right\} + \frac {\left(c _ {2} - c _ {1}\right) \eta}{n} \\ \leq c _ {1} \hat {\Re} _ {S} \{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max  P (h ^ {(m)} - h ^ {(m) *}) ^ {2} \leq r \} + \frac {c _ {2} \eta}{n} \\ \leq c _ {1} \hat {\Re} _ {S} \{h \in \mathcal {H}: \mu_ {\ell} ^ {2} \max P _ {n} (h ^ {(m)} - \hat {h} ^ {(m)}) ^ {2} \leq c _ {3} r \} + \frac {c _ {2} \eta}{n} \\ = \hat {\psi} (r). \\ \end{array}
$$
Letting  $r = r^{*}$  and using Lemma 4.3 of [20], we obtain  $r^{*} \leq \hat{r}^{*}$ , thus proving the statement.
Under Assumption 4.2, we obtain the following excess risk bound for the proposed model class using Corollary D.3. The proof is based on [20].
Theorem D.4. Let  $\hat{h}$  be any element of  $\mathcal{H}$  satisfying  $P_{n}\ell(y, \hat{h}(\boldsymbol{x})) = \inf_{h \in \mathcal{H}} P_{n}\ell(y, h(\boldsymbol{x}))$ . Suppose that Assumption 4.2 is satisfied, then there exists a constant  $c$  depending only on  $\mu_{\ell}$  such that for any  $\eta > 0$ , with probability at least  $1 - 5e^{-\eta}$ ,
$$
\begin{array}{l} P (y - \hat {h} (\boldsymbol {x})) ^ {2} - P (y - h ^ {*} (\boldsymbol {x})) ^ {2} \\ \leq c \left(\min  _ {0 \leq \kappa_ {1}, \kappa_ {2} \leq n} \left\{\frac {\kappa_ {1} + \kappa_ {2}}{n} + \left(\frac {1}{n} \sum_ {j = \kappa_ {1} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(1)} + \sum_ {j = \kappa_ {2} + 1} ^ {n} \hat {\lambda} _ {j} ^ {(2)}\right) ^ {\frac {1}{2}} \right\} + \frac {\eta}{n}\right). \\ \end{array}
$$
Theorem D.4 is a multiple-kernel version of Corollary 6.7 of [20], and a data-dependent version of Theorem 2 of [46] which considers the eigenvalues of the Hilbert-Schmidt operators on  $\mathcal{H}$  and  $\mathcal{H}^{(m)}$ . Theorem D.4 concerns the eigenvalues of the Gram matrices  $K^{(m)}$  computed from the data.