SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {H} = \left\{x \mapsto g _ {1} (\boldsymbol {f} _ {\boldsymbol {s}}) + g _ {2} (\boldsymbol {f} _ {\boldsymbol {s}}) g _ {3} (\boldsymbol {x}) \mid g _ {j} \in \mathcal {G}, j = 1, 2, 3 \right\}.
$$
In particular, the following discussion in this section assumes that  $g_{1}, g_{2}$ , and  $g_{3}$  are represented by linear functions on the RKHSs.
4.1 Generalization Bound
The optimization problem is expressed as follows:
$$
\min  _ {\alpha , \beta , \gamma} P _ {n} \ell \left(y, \langle \alpha , \Phi_ {1} \rangle_ {\mathcal {H} _ {1}} + \langle \beta , \Phi_ {2} \rangle_ {\mathcal {H} _ {2}} \langle \gamma , \Phi_ {3} \rangle_ {\mathcal {H} _ {3}}\right) + \lambda_ {\alpha} \| \alpha \| _ {\mathcal {H} _ {1}} ^ {2} + \lambda_ {\beta} \| \beta \| _ {\mathcal {H} _ {2}} ^ {2} + \lambda_ {\gamma} \| \gamma \| _ {\mathcal {H} _ {3}} ^ {2}, \tag {4}
$$
where  $\Phi_1 = \Phi_1(\pmb{f}_s(\pmb{x}))$ ,  $\Phi_2 = \Phi_2(\pmb{f}_s(\pmb{x}))$  and  $\Phi_3 = \Phi_3(\pmb{x})$  denote the feature maps. Without loss of generality, it is assumed that  $\| \Phi_i \|_{\mathcal{H}_i}^2 \leq 1$  ( $i = 1, 2, 3$ ) and  $\lambda_\alpha, \lambda_\beta, \lambda_\gamma > 0$ . Hereafter, we will omit the suffixes  $\mathcal{H}_i$  in the norms if there is no confusion.
Let  $(\hat{\alpha},\hat{\beta},\hat{\gamma})$  be a solution of Eq. (4), and denote the corresponding function in  $\mathcal{H}$  as  $\hat{h}$ . For any  $\alpha$ , we have
$$
\lambda_ {\alpha} \| \hat {\alpha} \| ^ {2} \leq P _ {n} \ell (y, \langle \hat {\alpha}, \Phi_ {1} \rangle + \langle \hat {\beta}, \Phi_ {2} \rangle \langle \hat {\gamma}, \Phi_ {3} \rangle) + \lambda_ {\alpha} \| \hat {\alpha} \| ^ {2} + \lambda_ {\beta} \| \hat {\beta} \| ^ {2} + \lambda_ {\gamma} \| \hat {\gamma} \| ^ {2} \leq P _ {n} \ell (y, \langle \alpha , \Phi_ {1} \rangle) + \lambda_ {\alpha} \| \alpha \| ^ {2},
$$
where we use the fact that  $\ell(\cdot, \cdot)$  and  $\|\cdot\|$  are non-negative, and  $(\hat{\alpha}, \hat{\beta}, \hat{\gamma})$  is the minimizer of Eq. (4). Denoting  $\hat{R}_s = \inf_{\alpha} \{P_n \ell(y, \langle \alpha, \Phi_1 \rangle) + \lambda_\alpha \|\alpha\|^2\}$ , we obtain  $\|\hat{\alpha}\|^2 \leq \lambda_\alpha^{-1} \hat{R}_s$ . Because the same inequality holds for  $\lambda_\beta \|\hat{\beta}\|^2, \lambda_\gamma \|\hat{\gamma}\|^2$  and  $P_n \ell(y, \hat{h})$ , we have  $\|\hat{\beta}\|^2 \leq \lambda_\beta^{-1} \hat{R}_s$ ,  $\|\hat{\gamma}\|^2 \leq \lambda_\gamma^{-1} \hat{R}_s$  and  $P_n \ell(y, \hat{h}) \leq \hat{R}_s$ . Moreover, we have  $P \ell(y, \hat{h}) = \mathbb{E} P_n \ell(y, \hat{h}) \leq \mathbb{E} \hat{R}_s$ . Therefore, it is sufficient to consider the following hypothesis class  $\tilde{\mathcal{H}}$  and loss class  $\mathcal{L}$ :
$$
\tilde {\mathcal {H}} = \left\{x \mapsto \langle \alpha , \Phi_ {1} \rangle + \langle \beta , \Phi_ {2} \rangle \langle \gamma , \Phi_ {3} \rangle \Big | \| \alpha \| ^ {2} \leq \lambda_ {\alpha} ^ {- 1} \hat {R} _ {s}, \| \beta \| ^ {2} \leq \lambda_ {\beta} ^ {- 1} \hat {R} _ {s}, \| \gamma \| ^ {2} \leq \lambda_ {\gamma} ^ {- 1} \hat {R} _ {s}, P \ell (y, h) \leq \mathbb {E} \hat {R} _ {s} \right\},
$$
$$
\mathcal {L} = \left\{(x, y) \mapsto \ell (y, h) \mid h \in \tilde {\mathcal {H}} \right\}.
$$
Here, we show the generalization bound of the proposed model class. The following theorem is based on [11], showing that the difference between the generalization error and empirical error can be bounded using the magnitude of the relevance of the domains.
Theorem 4.1. There exists a constant  $C$  depending only on  $\lambda_{\alpha}, \lambda_{\beta}, \lambda_{\gamma}$  and  $L$  such that, for any  $\eta > 0$  and  $h \in \tilde{\mathcal{H}}$ , with probability at least  $1 - e^{-\eta}$ ,
$$
P \ell (y, h) - P _ {n} \ell (y, h) = \tilde {O} \bigg (\bigg (\sqrt {\frac {R _ {s}}{n}} + \frac {\mu_ {\ell} ^ {2} C ^ {2} + \sqrt {\eta}}{n} \bigg) \big (C + \sqrt {\eta} \big) + \frac {C ^ {2} + \eta}{n} \bigg),
$$
where  $R_{s} = \inf_{\alpha}\{P\ell (y,\langle \alpha ,\Phi_{1}\rangle) + \lambda_{\alpha}\| \alpha \|^{2}\}$ .
Because  $\Phi_1$  is the feature map from the source feature space  $\mathcal{F}_s$  into the RKHS  $\mathcal{H}_1$ ,  $R_s$  corresponds to the true risk of training in the target domain using only the source features  $f_s$ . If this is sufficiently small, e.g.,  $R_s = \tilde{O}(n^{-1/2})$ , the convergence rate indicated by Theorem 4.1 becomes  $n^{-1}$ , which is an improvement over the naive convergence rate  $n^{-1/2}$ . This means that if the source task yields feature representations strongly related to the target domain, training in the target domain is accelerated. Theorem 4.1 measures this cross-domain relation using the metric  $R_s$ .
Theorem 4.1 is based on Theorem 11 of [11] in which the function class  $g_{1} + g_{3}$  is considered. Our work differs in the following two points: the source features are modeled not only additively but also multiplicatively, i.e., we consider the function class  $g_{1} + g_{2} \cdot g_{3}$ , and we also consider the estimation of the parameters for the source feature combination, i.e., the parameters of the functions  $g_{1}$  and  $g_{2}$ . In particular, the latter affects the resulting rate. With fixed the source combination parameters, the resulting rate improves only up to  $n^{-3/4}$ . The details are discussed in Section D.2.
4.2 Excess Risk Bound
Here, we analyze the excess risk, which is the difference between the risk of the estimated function and the smallest possible risk within the function class.
Recall that we consider the functions  $g_{1}, g_{2}$  and  $g_{3}$  to be elements of the RKHSs  $\mathcal{H}_{1}, \mathcal{H}_{2}$  and  $\mathcal{H}_{3}$  with kernels  $k_{1}, k_{2}$  and  $k_{3}$ , respectively. Define the kernel  $k^{(1)} = k_{1}$ ,  $k^{(2)} = k_{2} \cdot k_{3}$  and  $k = k^{(1)} + k^{(2)}$ . Let  $\mathcal{H}^{(1)}, \mathcal{H}^{(2)}$  and  $\mathcal{H}$  be the RKHS with  $k^{(1)}, k^{(2)}$  and  $k$  respectively. For  $m = 1, 2$ , consider the normalized Gram matrix  $K^{(m)} = \frac{1}{n} (k^{(m)}(\pmb{x}_i, \pmb{x}_j))_{i,j=1,\dots,n}$  and its eigenvalues  $(\hat{\lambda}_i^{(m)})_{i=1}^n$ , arranged in a nonincreasing order.
We make the following additional assumptions:
Assumption 4.2. There exists  $h^* \in \mathcal{H}$  and  $h^{(m)*} \in \mathcal{H}^{(m)}$  ( $m = 1, 2$ ) such that  $P(y - h^*(\pmb{x}))^2 = \inf_{h \in \mathcal{H}} P(y - h(\pmb{x}))^2$  and  $P(y - h^{(m)*}(\pmb{x}))^2 = \inf_{h \in \mathcal{H}^{(m)}} P(y - h(\pmb{x}))^2$ .
Assumption 4.3. For  $m = 1,2$ , there exist  $a_{m} > 0$  and  $s_m \in (0,1)$  such that  $\hat{\lambda}_j^{(m)} \leq a_m j^{-1 / s_m}$ .
Assumption 4.2 is used in [20] and is not overly restrictive as it holds for many regularization algorithms and convex, uniformly bounded function classes. In the analysis of kernel methods, Assumption 4.3 is standard [21], and is known to be equivalent to the classical covering or entropy number assumption [22]. The inverse decay rate  $s_m$  measures the complexity of the RKHS, with larger values corresponding to more complex function spaces.
Theorem 4.4. Let  $\hat{h}$  be any element of  $\mathcal{H}$  satisfying  $P_{n}(y - \hat{h} (\pmb {x}))^{2} = \inf_{h\in \mathcal{H}}P_{n}(y - h(\pmb {x}))^{2}$ . Under Assumptions 4.2 and 4.3, for any  $\eta >0$  with probability at least  $1 - 5e^{-\eta}$ ,
$$
P (y - \hat {h} (\pmb {x})) ^ {2} - P (y - h ^ {*} (\pmb {x})) ^ {2} = O \Big (n ^ {- \frac {1}{1 + \max \{s _ {1}, s _ {2} \}}}.
$$
Theorem 4.4 suggests that the convergence rate of the excess risk depends on the decay rates of the eigenvalues of two Gram matrices  $K^{(1)}$  and  $K^{(2)}$ . The inverse decay rate  $s_1$  of the eigenvalues of  $K^{(1)} = \frac{1}{n} (k_{1}(\pmb{f}_{s}(\pmb{x}_{i}),\pmb{f}_{s}(\pmb{x}_{j})))_{i,j = 1,\dots,n}$  represents the learning efficiency using only the source features, while  $s_2$  is the inverse decay rate of the eigenvalues of the Hadamard product of  $K_{2} = \frac{1}{n} (k_{2}(\pmb{f}_{s}(\pmb{x}_{i}),\pmb{f}_{s}(\pmb{x}_{j}))_{i,j = 1,\dots,n}$  and  $K_{3} = \frac{1}{n} (k_{3}(\pmb{x}_{i},\pmb{x}_{j}))_{i,j = 1,\dots,n}$ , which addresses the effect of combining the source features and original input. While rigorous discussion on the relationship between the spectra of two Gram matrices  $K_{2},K_{3}$  and their Hadamard product  $K_{2}\circ K_{3}$  seems difficult, intuitively, the smaller the overlap between the space spanned by the source features and by the original input, the smaller the overlap between  $\mathcal{H}_2$  and  $\mathcal{H}_3$ . In other words, as the source features and original input have different information, the tensor product  $\mathcal{H}_2\otimes \mathcal{H}_3$  will be more complex, and the decay rate  $1 / s_2$  is expected to be larger. In Section B.1, we experimentally confirm this speculation.
We demonstrate the potential of the affine model transfer through two case studies: (i) the prediction of feed-forward torque at seven joints of the robot arm [23], and (ii) the prediction of review scores and decisions of scientific papers [24]. The experimental details are presented in Section C. Additionally, two case studies in materials science are presented in Section B. The Python code is available at https://github.com/mshunya/AffineTL.
5.1 Kinematics of the Robot Arm
We experimentally investigated the learning performance of the affine model transfer, compared to several existing methods. The objective of the task is to predict the feed-forward torques, required to follow the desired trajectory, at seven different joints of the SARCOS robot arm [23]. Twenty-one features representing the joint position, velocity, and acceleration were used as the input  $\mathbf{x}$ . The target task is to predict the torque value at one joint. The representations encoded in the intermediate layer of the source neural network for predicting the other six joints were used as the source features  $\mathbf{f}_s \in \mathbb{R}^{16}$ . The experiments were conducted with seven different tasks (denoted as Torque 1-7) corresponding to the seven joints. For each target task, a training set of size  $n \in \{5, 10, 15, 20, 30, 40, 50\}$  was randomly constructed 20 times, and the performances were evaluated using the test data.
Table 1: Performance on predicting the torque values at the first and seventh joints of the SARCOS robot arm. The mean and standard deviation of the RMSE are reported for varying numbers of training samples. For each task and  $n$ , the case with the smallest mean RMSE is indicated by the bold type. An asterisk indicates a case where the RMSEs of 20 independent experiments were significantly improved over Direct at the  $1\%$  significance level, according to the Welch's t-test.  $d$  represent the dimension of the original input  $x$  (i.e.,  $d = 21$ ).
Target	Model	Number of training samples
n < d	n ≈ d	n > d
5	10	15	20	30	40	50
Torque 1	Direct	21.3 ± 2.04	18.9 ± 2.11	17.4 ± 1.79	15.8 ± 1.70	13.7 ± 1.26	12.2 ± 1.61	10.8 ± 1.23
Only source	24.0 ± 6.37	22.3 ± 3.10	21.0 ± 2.49	19.7 ± 1.34	18.5 ± 1.92	17.6 ± 1.59	17.3 ± 1.31
Augmented	21.8 ± 2.88	19.2 ± 1.37	17.8 ± 2.30	15.7 ± 1.53	13.3 ± 1.19	11.9 ± 1.37	10.7 ± 0.954
HTL-offset	23.7 ± 6.50	21.2 ± 3.85	19.8 ± 3.23	17.8 ± 2.35	16.2 ± 3.31	15.0 ± 3.16	15.1 ± 2.76
HTL-scale	23.3 ± 4.47	22.1 ± 5.31	20.4 ± 3.84	18.5 ± 2.72	17.6 ± 2.41	16.9 ± 2.10	16.7 ± 1.74
AffineTL-full	21.2 ± 2.23	18.8 ± 1.31	18.6 ± 2.83	15.9 ± 1.65	13.7 ± 1.53	12.3 ± 1.45	11.1 ± 1.12
AffineTL-const	21.2 ± 2.21	18.8 ± 1.44	17.7 ± 2.44	15.9 ± 1.58	13.4 ± 1.15	12.2 ± 1.54	10.9 ± 1.02
Fine-tune	25.0 ± 7.11	20.5 ± 3.33	18.6 ± 2.10	17.6 ± 2.55	14.1 ± 1.39	12.6 ± 1.13	11.1 ± 1.03
MAML	29.8 ± 12.3	22.5 ± 3.21	20.8 ± 2.12	20.3 ± 3.14	16.7 ± 3.00	14.4 ± 1.85	13.4 ± 1.19
L2-SP	24.9 ± 7.09	20.5 ± 3.30	18.8 ± 2.04	18.0 ± 2.45	14.5 ± 1.36	13.0 ± 1.13	11.6 ± 0.983
PAC-Net	25.2 ± 8.68	22.7 ± 5.60	20.7 ± 2.65	20.1 ± 2.16	18.5 ± 2.77	17.6 ± 1.85	17.1 ± 1.38
Torque 7	Direct	2.66 ± 0.307	2.13 ± 0.420	1.85 ± 0.418	1.54 ± 0.353	1.32 ± 0.200	1.18 ± 0.138	1.05 ± 0.111
Only source	2.31 ± 0.618	*1.73 ± 0.560	*1.49 ± 0.513	*1.22 ± 0.269	*1.09 ± 0.232	*0.969 ± 0.144	*0.927 ± 0.170
Augmented	2.47 ± 0.406	1.90 ± 0.515	1.67 ± 0.552	*1.31 ± 0.214	1.16 ± 0.225	*0.984 ± 0.149	*0.897 ± 0.138
HTL-offset	2.29 ± 0.621	*1.69 ± 0.507	*1.49 ± 0.513	*1.22 ± 0.269	*1.09 ± 0.233	*0.969 ± 0.144	*0.925 ± 0.171
HTL-scale	2.32 ± 0.599	*1.71 ± 0.516	1.51 ± 0.513	*1.24 ± 0.271	*1.12 ± 0.234	*0.999 ± 0.175	0.948 ± 0.172
AffineTL-full	*2.23 ± 0.554	*1.71 ± 0.501	*1.45 ± 0.458	*1.21 ± 0.256	*1.06 ± 0.219	*0.974 ± 0.164	*0.870 ± 0.121
AffineTL-const	*2.30 ± 0.565	*1.73 ± 0.420	*1.48 ± 0.527	*1.20 ± 0.243	*1.04 ± 0.217	*0.963 ± 0.161	*0.884 ± 0.136
Fine-tune	*2.33 ± 0.511	*1.62 ± 0.347	*1.35 ± 0.340	*1.12 ± 0.165	*0.959 ± 0.12	*0.848 ± 0.0824	*0.790 ± 0.0547
MAML	2.54 ± 1.29	1.90 ± 0.507	1.67 ± 0.313	1.63 ± 0.282	1.28 ± 0.272	1.20 ± 0.199	1.06 ± 0.111
L2-SP	*2.33 ± 0.509	*1.65 ± 0.378	*1.35 ± 0.340	*1.12 ± 0.165	*0.968 ± 0.114	*0.858 ± 0.0818	*0.802 ± 0.0535
PAC-Net	2.24 ± 0.706	*1.61 ± 0.394	*1.43 ± 0.389	*1.24 ± 0.177	*1.18 ± 0.100	1.13 ± 0.0726	1.100 ± 0.0589
The following seven methods were compared, including two existing HTL procedures:
Direct Train a model using the target input  $x$  with no transfer.
Only source Train a model  $g(\pmb{f}_s)$  using only the source feature  $\pmb{f}_s$ .
Augmented Perform a regression with the augmented input vector concatenating  $x$  and  $f_{s}$ .
HTL-offset [10] Calculate the transformed output  $z_{i} = y_{i} - g_{\mathrm{only}}(\pmb{f}_{s})$  where  $g_{\mathrm{only}}(\pmb{f}_{s})$  is the model pre-trained using Only source, and train an additional model with input  $x_{i}$  to predict  $z_{i}$ .
HTL-scale [12] Calculate the transformed output  $z_{i} = y_{i} / g_{\mathrm{only}}(\pmb{f}_{s})$ , and train an additional model with input  $\pmb{x}_i$  to predict  $z_{i}$ .
AffineTL-full Train the model  $g_{1} + g_{2}\cdot g_{3}$
AffineTL-const Train the model  $g_{1} + g_{3}$
Kernel ridge regression with the Gaussian kernel  $\exp(-\|\pmb{x} - \pmb{x}'\|^2 / 2\ell^2)$  was used for each procedure. The scale parameter  $\ell$  was fixed to the square root of the dimension of the input. The regularization parameter in the kernel ridge regression and  $\lambda_{\alpha}, \lambda_{\beta}$ , and  $\lambda_{\gamma}$  in the affine model transfer were selected through 5-fold cross-validation. In addition to the seven feature-based methods, four weight-based TL methods were evaluated: fine-tuning, MAML [25],  $L^2$ -SP [26], and PAC-Net [27].
Table 1 summarizes the prediction performance of the seven different procedures for varying numbers of training samples in two representative tasks: Torque 1 and Torque 7. The joint of Torque 1 is located closest to the root of the arm. Therefore, the learning task for Torque 1 is less relevant to those for the other joints, and the transfer from Torque 2-6 to Torque 1 would not work. In fact, as shown in Table 1, no method showed a statistically significant improvement to Direct. In particular, Only source failed to acquire predictive ability, and HTL-offset and HTL-scale likewise showed poor prediction performance owing to the negative effect of the failure in the variable transformation. In contrast, the two affine transfer models showed almost the same predictive performance as Direct, which is expressed as its submodel, and successfully suppressed the occurrence of negative transfer.
Because Torque 7 was measured at the joint closest to the end of the arm, its value strongly depends on those at the other six joints, and the procedures with the source features were more effective than in the other tasks. In particular, AffineTL achieved the best performance among the other feature-based methods. This is consistent with the theoretical result that the transfer capability of the affine model transfer can be further improved when the risk of learning using only the source features is sufficiently small.
In Table S.3 in Section C.1, we present the results for all tasks. In most cases, AffineTL achieved the best performance among the feature-based methods. In several other cases, Direct produced the best results; in almost all cases, Only source and the two HTLs showed no advantage over AffineTL. Comparing the weight-based and feature-based methods, we noticed that the weight-based methods showed higher performance with large sample sizes. Nevertheless, in scenarios with extremely small sample sizes (e.g.,  $n = 5$  or 10), AffineTL exhibited comparable or even superior performance.
The strength of our method compared to weight-based TLs including fine-tuning is that it does not degrade its performance in cases where cross-domain relationships are weak. While fine-tuning outperformed our method in cases of Torque 7, the performance of fine-tuning was significantly degraded as the source-target relationship became weaker, as seen in Torque 1 case. In contrast, our method was able to avoid negative transfer even for such cases. This characteristic is particularly beneficial because, in many cases, the degree of relatedness between the domains is not known in advance. Furthermore, weight-based methods can sometimes be unsuitable, especially when transferring knowledge from large models, such as LLMs. In these scenarios, fine-tuning all parameters is unfeasible, and feature-based TL is preferred. Our approach often outperforms other feature-based methods.
5.2 Evaluation of Scientific Documents
Through a case study in natural language processing, we compare the performance of the affine model transfer with that of ordinary feature extraction-based TL and show the advantage of being able to estimate domain shift and domain-specific factors separately.
We used SciRepEval [24], a benchmark dataset of scientific documents. The dataset consists of abstracts, review scores, and decision statuses of papers submitted to various machine learning conferences. We focused on two primary tasks: a regression task to predict the average review score, and a binary classification task to determine the acceptance or rejection status of each paper. The original input  $\mathbf{x}$  was represented by a two-gram bag-of-words vector of the abstract. For the source features  $\mathbf{f}_s$ , we utilized text embeddings of the abstract generated by the pre-trained language models; BERT [28], SciBERT [29], T5 [30], and GPT-3 [31]. In the affine model transfer, we employed neural networks with two hidden layers to model  $g_1$  and  $g_2$ , and a linear model for  $g_3$ . For comparison, we also evaluated the performance of the ordinary feature extraction-based TL using a two-layer neural network with  $\mathbf{f}_s$  as inputs. We used 8,166 training samples and evaluated the performance of the model on 2,043 test samples.
Table 2 shows the root mean square error (RMSE) for the regression task and accuracy for the classification task. In the regression tasks, the RMSEs of the affine model transfer were significantly improved over the ordinary feature extraction for the four types of text feature embedding. We also observed the improvements in accuracy for the classification task even though the affine model transfer was derived on the basis of regression settings. While the pre-trained language models have the remarkable ability to represent text quality and structure, their representation ability to perform prediction tasks for machine learning documents is not sufficient. The affine model transfer effectively bridged this gap by learning the additional target-specific factor via the target task, resulting in improved prediction performance in both regression and classification tasks.
Table 3 provides a list of phrases that were estimated to have a positive or negative effect on the review scores. Because we restricted a network to output positive values for  $g_{2}$ , the influence of each phrase could be inferred from the estimated coefficients of the linear model  $g_{3}$ . Specifically, phrases such as "tasks including" and "new state" were estimated to have positive influences on the predicted score. These phrases often appear in contexts such as "demonstrated on a wide range of tasks including" or "establishing a new state-of-the-art result," suggesting that superior experimental results tend to yield higher peer review scores. In addition, the phrase "theoretical analysis" was also identified to have a positive effect on the review score, reflecting the significance of theoretical validation in machine learning research. On the contrary, general phrases with broader meanings such as "recent advances" and "machine learning," contributed to lower scores. This observation suggests the importance of explicitly stating the novelty and uniqueness of research findings and refraining from using generic terminologies.
Table 2: Prediction performance of peer review scores and acceptance/rejection for submitted papers. The mean and standard deviation of the RMSE and accuracy are reported for the affine model transfer (AffineTL) and feature extraction (FE). Definitions of asterisks and boldface letters are the same as in Table 1.
Regression	Classification
BERT	FE	1.3086 ± 0.0035	0.6250 ± 0.0217
AffineTL	*1.3069 ± 0.0042	0.6252 ± 0.0163
SciBERT	FE	1.2856 ± 0.0144	0.6520 ± 0.0106
AffineTL	*1.2797 ± 0.0122	0.6507 ± 0.0124
T5	FE	1.3486 ± 0.0175	0.6344 ± 0.0079
AffineTL	*1.3442 ± 0.0030	0.6366 ± 0.0065
GPT-3	FE	1.3284 ± 0.0138	0.6279 ± 0.0181
AffineTL	*1.3234 ± 0.0140	*0.6386 ± 0.0095
Table 3: Phrases with the top and bottom ten regression coefficients for  $g_{3}$  in the affine transfer model for the regression task with SciBERT.
Positive	Negative
1	tasks including	recent advances
2	new state	novel approach
3	high quality	latent space
4	recently proposed	learning approach
5	latent variable	neural architecture
6	number parameters	machine learning
7	theoretical analysis	attention mechanism
8	policy gradient	reinforcement learning
9	inductive bias	proposed framework
10	image generation	descent sgd
As illustrated in this example, integrating modern deep learning techniques and highly interpretable transfer models through the mechanism of the affine model transfer not only enhances prediction performance, but also provides valuable insights into domain-specific factors.
5.3 Case Studies in Materials Science
We conducted two additional case studies, both of which pertain to scientific tasks in the field of materials science. One experiment aims to examine the relationship between qualitative differences in source features and learning behavior of the affine model transfer. In the other experiment, we demonstrate the potential utility of the affine model transfer as a calibration tool bridging computational models and real-world systems. In particular, we highlight the benefits of separately modeling and estimating domain-specific factors through a case study in polymer chemistry. The objective is to predict the specific heat capacity at constant pressure of any given organic polymer with its chemical structure in the polymer's repeating unit. Specifically, we conduct TL to bridge the gap between experimental values and physical properties calculated from molecular dynamics simulations. The details are shown in Section B in Supplementary Material,
In this study, we introduced a general class of TL based on affine model transformations, and clarified their learning capability and applicability. The proposed affine model transformation was shown to be an optimal class that minimizes the expected squared loss in the HTL procedure. The model is contrasted with widely applied TL methods, such as re-using features from pre-trained models, which lack theoretical foundation. The affine model transfer is model-agnostic; it is easily combined with any machine learning models, features, and physical models. Furthermore, in the model, domain-specific factors are involved in incorporating the source features. From this property, the affine transfer has the ability to handle domain common and unique factors simultaneously and separately.