Transfer Learning with Affine Model Transformation
Authors: Shunya Minami
Kenji Fukumizu
Yoshihiro Hayashi
Ryo Yoshida
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine learning
Transfer learning
TL;DR: We present a general class of transfer learning regression, following the principle of expected-square loss minimization.

Abstract: Supervised transfer learning has received considerable attention due to its potential to boost the predictive power of machine learning in scenarios where data are scarce. Generally, a given set of source models and a dataset from a target domain are used to adapt the pre-trained models to a target domain by statistically learning domain shift and domain-specific factors. While such procedurally and intuitively plausible methods have achieved great success in a wide range of real-world applications, the lack of a theoretical basis hinders further methodological development. This paper presents a general class of transfer learning regression called affine model transfer, following the principle of expected-square loss minimization. It is shown that the affine model transfer broadly encompasses various existing methods, including the most common procedure based on neural feature extractors. Furthermore, the current paper clarifies theoretical properties of the affine model transfer such as generalization error and excess risk. Through several case studies, we demonstrate the practical benefits of modeling and estimating inter-domain commonality and domain-specific factors separately with the affine-type transfer models.

Transfer Learning with Affine Model Transformation
Shunya Minami
Kenji Fukumizu
The Institute of Statistical Mathematics mshunya@ism ac. jp
The Institute of Statistical Mathematics fukumi zu@ism.@emmmm ac
Yoshihiro Hayashi
Ryo Yoshida
The Institute of Statistical Mathematics yhayashi @ism. ac jp
The Institute of Statistical Mathematics yoshidar@ism.oo ac. jp
Abstract
Supervised transfer learning has received considerable attention due to its potential to boost the predictive power of machine learning in scenarios where data are scarce. Generally, given set of source models and a dataset from a target domain are used adapt the pre-trained models to target domain by statistically learning domain shift and domain-specific factors. While such procedurally and intuitively plausible methods have achieved great success in a wide range of real-world applications, the lack of a theoretical basis hinders further methodological development. This paper presents a general class of transfer learning regression called affine model transfer, following the principle of expected-squant loss minimization. It is shown that the affine model transfer broadly encompasses various existing methods, including the most common procedure based neural feature extractors. Furthermore, the current paper clarifies theoretical properties of the affine model transfer such as generalization eтror and excess risk. Through several case studies, we demonstrate the practical benefits of modeling and estimating inter-domain commonality and domain-specific faccoors separately with the affine-type transfer models.
Introduction
Transfer learning (TL) is a methodology to improve the predictive performance of machine learning in target domain with limited data by reusing knowledge gained from training related source domains. Its great potential has been demonstrated in various real-world problems, including computer vision [ 2), natural language processing most of the outstanding successes of TL to date have relied on the feature extraction ability of deep neural networks. For example, conventional method reuses feature representations encoded in an intermediate layer of pre-trained model as an input for the target task or uses samples from the target domain fine-tune the parameters of the pre-trained source model (9. While such methods are operationally plausible and intuitive, they lack methodological principles and remain theoretically unexplored in terms of their learning capability for limited data. This study develops principled methodology generally applicable to various kinds of TI..
3 biology (5), and materials science [6] g Notably,
In this study, focus on supervised TL settings. In particular, deal with settings where, given feature representations obtained from training in the source domain, we use samples from the target domain model and estimate the domain shift to the target. This procedure called hypothesis transfer learning (HTL); several methods have been proposed, such as using a linear transformation function [T and considering a general class of continuous transformation functions [2]. If the transformation function appropriately captures the functional relationship between the source and target domains, only the domain-speed factors need be additionally learned, which can be done efficiently even with limited sample size. In other words, the performance of HTL depends strongly
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

on whether the transformation function appropriately represents the cross-domain shift. However, the general methodology for modeling and estimating such domain shifts has been less studied.
This study derives a theoretically optimal class of supervised TL. that minimizes the expected l2 loss function of the HTTL. The resulting function class takes the form of affine coupling 91 (ff) + 92(fe) (x) of three functions 92.92 and f3. where the shift from a given source feature fs the target domain is represented by the functions 9i and 92, and the domain-specific factors are represented by (z()) for any given input æ. These functions can be estimated simultancously using conventional supervised learning algorithms such as kernel methods or deep neural networks. Hereafter, we refer to this framework as the affine model transfer. As described later, we can formulate a wide variety of TL algorithms within the affine model transfer, including the widely used neural feature extractors, offset and scale HTLs [OO] [T [1]. and Bayesian TL [[3]. We clarify theoretical properties of the affine model transfer such as generalization error and excess risk.
To summarize, the contributions of our study are as follows:
The affine model transfer is proposed to adapt source features to the target domain by separately estimating cross-domain shift and domain-specific factors.
The affine form derived theoretically as an optimal class based on the squared loss for the target task.
The affine model transfer encompasses several existing TL methods, including neural feature extraction. It can work with any type of source model, including non-machine learning models such as physical models as well as multiple source models.
For each of the three functions 91 92. and 93. we provide an efficient and stable estimation algorithm when modeled using the kernel method.
Two theoretical properties of the affine transfer model are shown: the generalization and the excess risk bound.
With several applications, we compare the affine model transfer with other TL algorithms, discuss its strengths, and demonstrate the advantage of being able to estimate cross-domain shifts and dpmain-specific factors separately.
2 Transfer Learning via Transformation Function
2.1 Affine Model Transfer
This study considers regression problems with squared loss. We assume that the output of the target domain y E y C R. follows y = f((x)) + where ft X' ++ R the true model on the target domain, and the observation noise has mean zero and variance x². We are given samples {x,,, y:) } E = ()) from the target domain and the feature representation fa (x) E F, from one or more source domains. Typically, fa is given as vector, including the output of the source models, observed data in the source domains, or learned features in a pre-trained model, but it can also be a non-vector feature such as tensor, graph, or text. Hereafter, Se is referred to as the source features.
In this paper, we focus on transfer learning with variable transformations as proposed in [12. For an illustration of the concept, consider the case where there exists relationship between the true and f (ar) E IR such that fi (ax) = f (xx) + x: 0" E Rd with an unknown parameter 0 E idd If f non-smooth, large number of training samples is needed learn fi directly. However, since the difference f? f is a linear function with respect to the unknown 0", can be learned with fewer samples if prior information about f available, For example, a target model can be obtained by adding f. to the model g trained for the intermediate variable 2 y (x))
functions f (x)
The following is a slight generalization of TL procedure provided in 12
1. With the source features, perform a variable transformation of the observed outputs as 2; Φ(ys, fa (2:)), using the data transformation function ø y x F, + R.
2. Train intermediate model ((x:) using the transformed sample set { 2) to predict the transformed output z for any given æ.
3. Obtain target model (ax) (ar)) (x)) using the model transformation function &
R x F. y that combines and fa to define predictor.

---

In particular, [12] considers the case where the model transformation function is equal to the inverse of the data transformation function. We consider more general case that eliminates this constraint.
The objective of step is to identify a transformation Φ that maps the output variable to the intermediate variable making the variable suitable for learning. In step 2, predictive model for is constructed. Since the data is limited in many TL setups, simple model, such as a linear model, should be used as g. Step 3 is to transform the intermediate model g into predictive model for the original output
This class of TL includes several approaches proposed in previous studies. For example, 10 pro- posed learning algorithm consisting of linear data transformation and linear model transformation: -=y- (0. fo) and e = g(x) + (0, fa) with pre-defined weights O. In this case, factors unexplained by the linear combination of source features are learned with g, and the target output is predicted additively with the common factor (0, f*) and the additionally learned g. In [13, is shown that a type of Bayesian TL is equivalent to using the following transformation functions; for Fs R, and = pg( (1)(=))((pp))))))))) with two varying hyperparameters т 1 and 0 0Spe This includes TL using density ratio estimation [14] and neural network- fine-tuning as special cases when the two hyperparameters belong specific regions.
é= (V-TT))))
The performance of this TL strongly depends on the design of the two transformation functions ø and eb. In the sequel, we theoretically derive the optimal form of transformation functions under the squared loss scenario. For simplicity, we denote the transformation functions as Φ5,(-) (-.()=oo (-)== fa) on ) and y (-) vb(:, fx) RR. To derive the optimal class of & and vh, note first that the TL procedure described above can be formulated in population as solving two successive least square problems;
(i) 9 = arg min Ep. , y,, (Y) - 3
(ii) min Ep, I|Y v. (g"0(((()) (X))1I?
Since the regression function that minimizes the mean squared error is the conditional mean, the first problem is solved by g% (x) = (Y) = æ]. which depends on ø. We can thus consider the optimal transformation functions & and v by the following minimization:
min Ep. Y v. (x ))?.. 4.
(1)
It is easy to see that Eq. is equivalent the following consistency condition:
v. (z)(z)) Ep, Y xz).
From the above observation, we make three assumptions to derive the optimal form of th and d: Assumption 2.1 (Differentiability). The data transformation function é is differentiable with respect to the first argument.
Assumption 2.2 (Invertibility). The model transformation function is invertible with respect the first argument, i.e., its inverse v1 exists.
Assumption 2.3 (Consistency). For any distribution on the target domain pe (x, y), and for all X,
vy (g) Ep [Y| x).
where 9" (a:) Ep. fo (Y) x=).
Assumption 2.2 is commonly used in most existing HTL settings, such as 10 and 12] It assumes one-to-one correspondence between the predictive value fe(2:) and the output of the intermediate model ((x). this assumption does not hold, then multiple values ý correspond to the same predicted value fü, which unnatural. Note that Assumption 2.3 corresponds to the unbiased condition of [12].
now derive the properties that the optimal transformation functions must satisfy.
Theorem 2.4. Under Assumptions 2.3 the transformation functions & and ib satisfy the following two properties:
i
1 Os.
ii (5) = gi(f)) + + +2(l))
where 91 and g2 are some functions.
3

---

The proof is given in Section DJ Supplementary Material. Despite not initially assuming that the two transformation functions are inverses. Theorem 2.4 implies they must indeed be inverses. Furthermore, the mean squared error is minimized when the data and model transformation functions are given by an affine transformation and its inverse, respectively. In summary, under the expected squared loss minimization with the HTL procedure, the optimal class for HTL model is expressed as follows:
H = gi (f&) + g2 go E ク 1,2,3}
where G1. G2 and G3 are the arbitrarily function classes. Here, each of 91 and 92 is modeled as a function of f. that represents common factors across the source and target domains, Is is modeled as a function æ, order to capture the domain-specific factors unexplainable by the source features.
We have derived the optimal form of the transformation functions when the squared loss is employed. Even for general convex loss functions, (i) of Theorem 2.4 still holds. However, (ii) of Theorem [..4] does not generally hold because the optimal transformation function depends on the loss function. Extensions to other losses are briefly discussed in Section A. I but the establishment of a complete theory is a future work.
Here, the affine transformation is found to be optimal in terms of minimizing the mean squared error. We can also derive the same optimal function by minimizing the upper bound of the estimation error in the HTL procedure, discussed in Section
One of key principles for the design of gl, 92, and da is interpretability. In our model, 9i and ga primarily facilitate knowledge transfer, while the estimated fa is used to gain insight on domain- specific factors. For instance, in order to infer cross-domain differences. we could design 9I and 92 by the conventional neural feature extraction, and simple, highly interpretable model such linear model could be used for 93. Thus, observing the estimated regression coefficients in ga, one can statistically infer which features of are related to inter-domain differences. This advantage of the proposed method is demonstrated in Section 5.2 and Section B.3
2.2 Relation to Existing Methods
The affine model transfer encompasses some existing TL procedures. For example, by setting 9: (fx) 0 and (ff) 1, the prediction model estimated without using the source features, which corresponds to an ordinary direct learning, i.e., learning scheme without transfer. Furthermore, various kinds of HTLs can be formulated by imposing constraints on 91 and g2. prior work, [TO] employs two-step procedure where the source features are combined with pre-defined weights, and then the auxiliary model is additionally learned for the residuals unexplainable by the source features. The affine model transfer can represent this HTL as a special case by setting //2 1. [12] uses the transformed output 2; = yt/ fs with the output value fs E of a source model, and this cross-domain shift is then regressed onto using target dataset. This HTL corresponds 91 0 and 92 = fs.
When a pre-trained source model is provided as a neural network, TL is usually performed with the intermediate layer as input to the model the target domain. This is called feature extractor or frozen featurizer and has been experimentally and theoretically proven to have strong transfer capability as the de facto standard for TL 9 15). The affine model transfer encompasses the neural feature extraction as a special subclass, which is equivalent to setting (f2(f))g3(a) performance comparison of the affine model transfer with the neural feature extraction is presented in Section and Section B.2 The relationships between these existing methods and the affine model transfer are illustrated in Figure I and Figure S.I
0. A
The affine model transfer can also be interpreted as generalizing the feature extraction by adding a product term (x). This additional term allows for the inclusion of unknown factors in the transferred model that are unexplainable by source features alone. Furthermore, this encourages the avoidance of negative transfer, phenomenon where prior learning experiences interfere with training in a new task. The usual TL based only on fa attempts to explain and predict the data generation process in the target domain using only the source features. However, in the presence of domain-speciff factors, a negative transfer can occur owing to a lack of descriptive power The additional term compensates for this shortcoming. The comparison behavior for the case with the non-relative source features is described in Section 5.1

---

Algorithm 1 Block relaxation algorithm 19
Initialize: an. bp # 0, Co #0 repeat
d++1 arg min F(a, b Ct)
b arg min, F(a b, Ce)
Figure 1: Architectures of (a) feature extraction, (b) HTL in (10), and affine model transfer.
C++1 = arg min F ar++1 bet1> C)
until convergence
The affine model transfer can be naturally expressed as an architecture of network networks, This architecture, called affine coupling layers, widely used for invertible neural networks flow-based generative modeling [16 [7]. Neural networks based on affine coupling layers have been proven have universal approximation ability [88]. This implies that the affine transfer model has the potential to represent wide range of function classes, despite its simple architecture based on the affine coupling of three functions.
Modeling and Estimation
In this section, we focus on using kernel methods for the affine transfer model and provide the estimation algorithm. Let H1, H2 and Hg be reproducing kernel Hilbert spaces (RKHSs) with positive-definite kernels 11, 22 and k3. which define the feature mappings 4 :F: - D2 iF H2
and Q3 :2: H3, respectively. Denote L = 4 (fa (x))), 22 = the proposed model class, the tg-regularized empirical risk with the squared loss is given follows:
23, L3(x;). For
n n Σ {-- (a, 11,)) (B, 22,) 3 A1llall + >211 + >ally (2)
Fa,
where A1. 2 are hyperparameters for the regularization. According to the representer theorem, the minimizer Fa. with respect to the parameters a H1, 1 Hz, and 7 E Hg reduces to
El= 01 = = b. 7 Σx= c, with the n--dimensional unknown parameter vectors a, b, c C R". Substituting this expression into Eq. (2) we obtain the objective function as
Fa,3, by Kia - (K2))((K)))
A K1a+>bb Kyb+ Age K3c= F(a, b, c). (3)
Here, the symbol o denotes the Hadamard product. Ky is the Gram matrix associated with the kernel ky for / E (1, kl [ #1) k )n)] denotes the i-th column of the Gram matrix.
The n x n matrix (1) given by the tensor product ( = ( @ 1 of k and k(").
Because the model is linear with respect to parameter a and bilinear for and the optimization of Eq. B can be solved using esta-esta techniques for the low-rank tensor regression. In this study, we use the block relaxation algorithm [19] as described in Algorithm [ It updates b, and e by repeatedly fixing two the three parameters and minimizing the objective function for the remaining one. Fixing two parameters, the resulting subproblem can be solved analytically because the objective function is expressed in quadratic form for the remaining parameter.
Algorithm can be regarded as repeating the HTL procedure introduced in Section alternately estimates the parameters (a, b) the transformation function and the parameters of the model for the given transformed data { 2a)) } The function F in Algorithm is not jointly convex in general. However, when employing methods like kernel methods or generalized linear models, and fixing two parameters, F exhibits convexity with respect to the remaining parameter. According to 19, when each sub-minimization problem convex, Algorithm D is guaranteed to converge to stationary point. Furthermore, [19] showed that consistency and asymptotic normality hold for the alternating minimization algorithm.
4 Theoretical Results
In this section, present two theoretical properties, the generalization bound and excess risk bound.
5

---

Let (2, P) be an arbitrary probability space, and { - be independent random variables with distribution P. For function f: R. let the expectation of f with respect P and its empirical counterpart denote respectively by Pʃ- = Erf(=), P.. = Em= [z;). We use non-negative loss fly, y²) such that it is bounded from above by L > 0 and for any fixed y eV, y y e Ely, 3/) is je-Lipschitz for some Pt > O.
Recall that the function class proposed in this work is
H. [z - 91 (f+) 92 (f)))g((()) gو E G.j 1.2,3).
In particular, the following discussion in this section assumes that 91 92, and ga are represented by linear functions on the RKHSs.
4.1 Generalization Bound
The optimization problem is expressed follows:
min Py (a, 11)H, + (3, 12) (y, Hs)H)) + Aallalla, + AollBl1 + AyIbilisss (4) a.8.y
where 1 21 (z)) 222 (2(f(( (x) and 43 1z((x) denote the feature maps. Without loss -i= - and da λB, > 0. Hereafter, we will omit the suffixes H4 the norms if there is no confusion.
of generality, it is assumed that |
Let (å, (, be solution of Eq. 4 and denote the corresponding function in H. as h. For any a. we have
Na [|]] < P,E (3, (|
(3,
where we use the fact that ((.-)) and || are non-negative, and (å, (d, 7) is the minimizer of Eq. ( Denoting R = inf (P, Ey, (a, 21)) + da [|]a]]]]] we obtain || S >- Because the same inequality holds for A A, ||2l2 and P,l(y, h), we have || < A Rs ||5112 < x R, and Py h) s R.. Moreover, we have Pe(y, h) = EP, h) S ER,. Therefore, it is sufficient to consider the following hypothesis class H and loss class L:
H =
{ x (1 + (B, 22h .a) = | RR, | M(y,
{ (z. v) + Ely, h) h E H..
Here, we show the generalization bound the proposed model class. The following theorem is based on [Л]] showing that the difference between the generalization error and empirical error can be bounded using the magnitude of the relevance of the domains.
Theorem 4.1. There exists a constant depending only on No. As, dy and L such that, for any 7 > and E H. with probability at least 1 - -
PÊly, h) 100-P,,C00 Prl(y, h) 0 ((VV R
12022
VI n
C2
C²+1 +1
(C+ VT)
where R, = inf (Pey, (a, 41)) + >..
Because +1 is the feature map from the source feature space F, into the RKHS Hi. R, corresponds to the true risk of training in the target domain using only the source features f.. If this sufficiently small, e.g., R. = Ö( -1/2), the convergence rate indicated by Theorem 4.1 becomes n- which is an improvement over the naive convergence rate n 11/2 This means that if the source task yields feature representations strongly related to the target domain, training in the target domain is accelerated. Theorem 4.1 measures this cross-domain relation using the metric R..
Theorem 4.1 is based Theorem 11 of [1] in which the function class øi + g3 is considered. Our work differs in the following two points: the source features are modeled not only additively but also multiplicatively, i.e., we consider the function class 91 + 92 92-99, and we also consider the estimation of the parameters for the source feature combination, i.e., the parameters of the functions Φ1 and g2- In particular, the latter affects the resulting rate. With fixed the source combination parameters, the resulting rate improves only up to n 3/1 The details are discussed in Section D.2

---

4.2 Excess Risk Bound
Here, we analyze the excess risk, which is the difference between the risk of the estimated function and the smallest possible risk within the function class.
Recall that consider the functions 91: 92 and 93 be elements of the RKHSs H1, Hzz and H3 with kernels k 22 and k3, respectively. Define the kernel k(2) = 1: k(() k2 ky and k: k + (2))
Let H()) H(2) and H. be the RKHS with ((2)) and respectively. For m = 2, consider the
normalized Gram matrix K(m) = (A/(m) (a:; ";))e.= and its eigenvalues (A, (m) )== arranged nonincreasing order.
We make the following additional assumptions:
Assumption 4.2. There exists h E H and h(m)) E H(m) (m = 1, such that P(y - (xx)) =
infnem Ply - -((=))? and Ply - h(m)* (x))² = inf nEN P(y h(x)))
Assumption 4.3. For 1, there exist um > 0 and 8m E (0, 1) such that X.m) knnj-
Assumption is used in [20] and is not overly restrictive as holds for many regularization algorithms and convex, uniformly bounded function classes. In the analysis of kernel methods, Assumption 4.3 is standard [2I], and is known to be equivalent to the classical covering or entropy number assumption [22] The inverse decay rate 8m measures the complexity of the RKHS, with larger values corresponding to more complex function spaces.
Theorem 4.4. Let h be any element of H. satisfying P, (y--h(x) h(x))² == infRem (n hx))) Under Assumptions 4.2 and 4.3 for any r7 > with probability at least 1 5e
Ply - h(x))² P(y - h² (x))) = 0 TFimax 1871og7
Theorem 4.4 suggests that the convergence rate of the excess risk depends on the decay rates of the eigenvalues of two Gram matrices (1)) and (22), The inverse decay rate 81 of the eigenvalues of k(') = features, while 82 is the inverse decay rate of the eigenvalues of the Hadamard product of K₂ = 1 fzx)
(f (kg (xx))
represents the learning efficiency using only the source
and K3 1 n. (kag(1; (x,, 2.)))..=....... which addresses the effect of combining the source features and original input. While rigorous discussion on the relationship between the spectra of two Gram matrices K2, K3 and their Hadamard product K2 o K3 seems difficult, intuitively, the smaller the overlap between the space spanned by the source features and by the original input, the smaller the overlap between H2 and H3. other words, as the source features and original input have different information, the tensor product H2 @ H3 will be more complex, and the decay rate 1/32 is expected to be larger. In Section B.1 we experimentally confirm this speculation.
5 Experimental Results
We demonstrate the potential of the affine model transfer through two case studies: (i) the prediction of feed-forward torque at seven joints of the robot arm [23] and (ii) the prediction of review scores and decisions scientific papers (24). The experimental details are presented in Section C Additionally, two case studies in materials science are presented in Section B The Python code available at
https://gi http///tttttu.././ com/ wom//mhunya//iiammmmammaa in
5.1 Kinematics of the Robot Arm
We experimentally investigated the learning performance of the affine model transfer, compared to several existing methods. The objective of the task is to predict the feed-forward torques, required to follow the desired trajectory, at seven different joints of the SARCOS robot arm 23 Twenty-on features representing the joint position, velocity, and acceleration were used as the input æ. The target task is to predict the torque value at one joint. The representations encoded in the intermediate layer of the source neural network for predicting the other six joints were used as the source features fa E R6 The experiments were conducted with seven different tasks (denoted as Torque 1-7) corresponding to the seven joints. For each target task, a training set of size n (5, 10, 15, 20, 30, 40, 50) was randomly constructed 20 times, and the performances were evaluated using the test data.
7

---

Table 1: Performance predicting the torque values at the first and seventh joints of the SARCOS robot arm. The mean and standard deviation of the RMSE are reported for varying numbers of training samples. For each task and n, the with the smallest mean RMSE is indicated by the bold type. An asterisk indicates a case where the RMSES of 20 independent experiments were significantly improved over Direct at the 1% significance level, according the Welch's 's t-test. d represent the dimension of the original input x: (i.e., d 21).
training samples
Target
Model
R <
It
10
15
20
50
Direct
21.3 2.04
189 2.11
17.4
1.29
15.8 1.20
13.7 1.26
12.2 1.61
10.8 1.23
Only source
24.0
6.37
22.3
5.10
21.0
2.40
19.7
+ 1.34
18.5 + 1.92
17.6 + 1.59
17.3 + 1.31
Augmented
21.8 4: 2.88
19.2
1.37
17.8 + 2.30
15.7 +: 1,53
13.3 + 1.19
11.9 +: 1.37
10.7 +: 0,954
HTL offset
23.7 6.50
21.2 3.85
19.8
17.8 2.35
16.2 4 3.31
15.0 4 3,16
15.1 4 2.76
23.3 13.3.144447
18.5 2.72
2.41
16,9
1.71
Torque
AffineTLLfull
21.2
188 1.31
186 2,83
15.9 1.68
13.7 1.53
12.3+ 1.45
11.1 1.12
AllinTLL-ooms
21.2 2.21
188 1: 1.44
17.7 1 2.44
15.9 1: 1,58
13,4 1: 1.15
12.2 1: 1.54
10,9 1: 1.02
Fine-lume
25.0
20.5 3.33
18.6 1. 2.10
17.6 2:55
14.1 1.39
12.6 1 1.13
11.1 1.03
MAMI
29.8 12.3
22.5 +
20.8 + 2.12
20.3 + 3.14
16.7 +: 3.00
14.4 + 1.85
13.4 1.19
L"-SP
24.9
20.5
18.8 2.04
18.0
14.5 1.36
13.0 1.13
11.6
PAC-Net
25.2 8.68
22.7 5.60
20.7 1. 2.65
20.1 2,16
18.5 2.77
17.6
1.85
17.1 1.38
Direct
2.66 0.307
2.13 0.420
1.85 + 0.418
1.54 0.353
1.32 0.200
1.18 + 0.138
1,05 + 0.111
Only source
2.31 +: 0.618
*177 + 0.560
11499 + 0.513
1.22 + 0.269
11.09 + 0.232
*0.969 +: 0.144
*0.927 + 0.170
Augmented
2.47 1. 0.406
1,90 0.515
1.67 1: 0.552
*1.31 0.214
1.16 0.225
*0,984 1 0.149
*0,897 1 0. 138
HTL-olbstt
2.29
40,969
40,925
HTL-seale
2.52 0.599
*1.71 0.516
1.51 1: 0.513
*1.24 0.271
.1.12 0.234
40,999 0.175
0,948 0.172
Torque
AfaleTTL-fall
+2.23 0.554
.1.71 + 0.501
*1.45 + 0.458
.1.21 0.256
+1.06 + 0.219
*0.974 # 0.164
10,870 + 0.121
AffinTL const
+2.30 42 0.565
*1.73 à 0.420
*1.48 à: 0,527
*1220 + 0.243
*1.04 = 0.217
+0.963 à: 0.161
*0.884 1 0,136
Fine-tune
0.,55
*1 62 0.347
$1.35 + 0.340
$1.12 + 0.165
40,959 +: 0.12
50,848 + 0,0824
40.790 +: 0.0547
MAMI.
2.54
0.507
166
1.28 0.272
1,20
1.06 D.111
17-SP
*2.33 0.509
*1 11.65.1 0.378
+1.35 1. 0.340
*1.12 1. 0.165
*0,968 1: 0.114
*0.85K 1. 0,0818
40,812 1. 0.0535
PAC-Net
2.24 + 0.706
*1.61 + 0.394
+1,43 +: 0.389
*1.24 + 0.177
*1.18 # 0,100
1.13 + 0.0726
1.100 + 0.0589
The following seven methods were compared, including two existing HTL procedures:
Direct Train model using the target input æ with no transfer.
Only source Train model ((fx) using only the source feature fs-
Augmented Perform regression with the augmented input vector concatenating æ and fa. HTL-offset [0O] Calculate the transformed output 2; =
Gonly (f+) (only(f.) where Jonly (fe) is the model pre-trained using Only source, and train an additional model with input æ; to predict 24-
HTL-scale [12] Calculate the transformed output 2; yt/ Gonly (fs), and train additional
model with input æ, to predict 2;-
AL-ineTLffull Train the model 91 + 92 93- AffineTL-cons' Train the model 91 g3-
Kernel ridge regression with the Gaussian kernel exp(- exp(-1[x-xx'] 22222 was used for each procedure. The scale parameter l was fixed the square root of the dimension of the input. The regularization parameter in the kernel ridge regression and Aa, Ag, and A,, in the affine model transfer were selected through 5-fold cross-validatio validation. In addition to the seven feature- based methods, four weight-based TL methods were evaluated: fine-tuning, MAML (25) SP (26). and PAC-Net (27]
Table summariizs the prediction performance of the seven different procedures for varying numbers of training samples in two representative tasks: Torque and Torque 7. The joint of Torque is located closest to the root of the arm. Therefore, the learning task for Torque 1 is less relevant to those for the other joints, and the transfer from Torque 2-6 to Torque would not work. In fact, as shown in Table T no method showed statistically significant improvement to Direct. In particular, Only source failed to acquire predictive ability, and HTL-offset and HTL-scale likewise showed poor prediction performance owing the negative effect of the failure in the variable transformation. In contrast, the two affine transfer models showed almost the same predictive performance as Direct, which is expressed as its submodel, and successfully suppressed the occurrence of negative transfer.
Because Torque was measured at the joint closest to the end of the arm, its value strongly depends on those at the other six joints, and the procedures with the source features were more effective than the other tasks. In particular, AffineTL achieved the best performance among the other feature-based methods. This is consistent with the theoretical result that the transfer capability of the affine model transfer can be further improved when the risk of learning using only the source features is sufficiently small.
8

---

In Table S.3 in Section C.I we present the results for all tasks. In most cases, AffineTTL achieved the best performance among the feature- based methods. In several other cases, Direct produced the best results; in almost all cases, Only source and the two HTLL showed no advantage over AffineTT.. Comparing the weight-based and feature- based methods, we noticed that the weight-based methods showed higher performance with large sample sizes. Nevertheless, in scenarios with extremely small sample sizes (e.g., n n= or 10), AffineTL exhibited comparable or even superior performance.
The strength of our method compared to weight-based TLs including fine-tuning is that it does not degrade its performance in cases where cross- dossin relationships are weak. While fine-tuning outperformed our method in cases of Torque 7, the performance of fine-tuning was significantly degraded as the source-targer relationship became weaker, as seen in Torque 1 case. In contrast, our method was able to avoid negative transfer even for such cases. This characteristic is particularly beneficial because, in many cases, the degree of relatedness between the domains is not known in advance. Furthermore, weight-based methods can sometimes be unsuitable, especially when transfer- ring knowledge from large models, such as LLMs. In these scenarios, fine-tuning all parameters unfeasible, and feature-based TL is preferred. Our approach often outperforms other based methods.
5.2 Evaluation of Scientific Documents
Through a case study in natural language processing, we compare the performance of the affine model transfer with that of ordinary feature extraction-based TL and show the advantage of being able to estimate domain shift and domain-specifie factors separately.
We used SciRepEval (24), a benchmark dataset of scientific documents. The dataset consists of abstracts, review scores, and decision statuses of papers submitted to various machine learning conferences. We focused on two primary tasks: regression task to predict the average review score, and a binary classification task to determine the acceptance or rejection status of each paper. The original input æ was represented by a two-gram bag-of-worde vector of the abstract. For the source features fa. we utilized text embeddings of the abstract generated by the pre-trained language models; BERT (28). SciBERT (2)) T5 (30), and GPT-3 [I] In the affine model transfer, we employed neural networks with two hidden layers to model 91 and g2, and a linear model for gs- For comparison, we also evaluated the performance of the ordinary feature extraction- based TL using a two-layer neural network with fa as inputs. We used 8,166 training samples and evaluated the performance of the model on 2,043 test samples.
Table 2 shows the root mean square error (RMSE) for the regression task and accuracy for the classification task. In the regression tasks, the RMSEs of the affine model transfer were significantly improved over the ordinary feature extraction for the four types of text feature embedding. We also observed the improvements in accuracy for the classification task even though the affine model transfer was derived on the basis of regression settings. While the pre-trained language models have the remarkable ability to represent text quality and structure, their representation ability to perform prediction tasks for machine learning documents is not sufficient. The affine model transfer effectively bridged this gap by learning the additional target-specifer factor via the target task, resulting in improved prediction performance in both regression and classification tasks.
Table 3 provides a list of phrases that were estimated have positive or negative effect on the review scores. Because we restricted a network to output positive values for g2, the influence of each phrase could be inferred from the estimated coefficients of the linear model g3. Specifically, phrases such as "tasks including" and "new state" were estimated to have positive influences on the predicted score. These phrases often appear in contexts such as "demonstrated on a wide range of tasks including" or "establishing a new state-of- of thethe-rrt result," suggesting that superior experimental results tend to yield higher peer review scores. In addition, the phrase "theoretical analysis" " was also identified to have positive effect on the review score, reflecting the significance of theoretical validation in machine learning research. On the contrary, general phrases with broader meanings such as "recent advances" and "machine learning." " contributed to lower scores. This observation suggests the importance of explicitly stating the novelty and uniqueness of research findings and refraining from using generic terminologies.

---

Table Prediction performance of peer review scores and acceptanccerejeccion for submitted papers. The mean and standard deviation of the RMSE and accuracy are reported for the affine model transfer (AffineTT.) and feature extraction (FE). Definitions of asterisks and boldface letters are the same as in Table I
Table 3: Phrases with the top and bottom ten regression coefficients for ga in the affine transfer model for the regression task with SciBERT.
Positive
Negative
1
lasks including
recent advances novel approach latent space
2
new state
Regression
Classification
3
high quality
FE AftincTL
1.3086 + 0.0035
0.6250 = 0.0217
4
recently proposed
learning approach neural architecture machine learning attention mechanism reinlorcement learning proposed framework descent sgd
BERT
*1.3069 + 0.0042
0.6252
0.0163
5
latent variable
SciBERT
FE AffineTI.
1.2797 + 0.0122
1.2856 1 0.0144
0.6507
0.6520
0.0124
0.0106
7
number parameters theoretical analysis
1.3486 +: 0.0175
0.6344
0.0079
8
policy gradient
15
AffinTTL
11.344 = 0.0030
0.6366
0.0068
9
inductive bias
GPT-3
AffineTI.
FE
11,234 + 0.0140
1.3284 +
"0.6386 0.0095
0.6279
0.0181
10
image generation
As illustrated in this example, integrating modern deep learning techniques and highly interpretable transfer models through the mechanism of the affine model transfer not only enhances prediction performance, but also provides valuable insights into domain-specifie factors.
5.3 Case Studies in Materials Science
We conducted two additional case studies, both of which pertain to scientific tasks in the field of materials science. One experiment aims to examine the relationship between qualitative differences in source features and learning behavior of the affine model transfer. In the other experiment, we demonstrate the potential utility of the affine model transfer as calibration tool bridging computational models and real-world systems. In particular, we highlight the benefits of separately modeling and estimating domain- specifcc factors through case study in polymer chemistry. The objective is to predict the specific heat capacity at constant pressure of any given organic polymer with its chemical structure the polymer's ss repeating unit. Specifically, we conduct TL. to bridge the gap between experimental values and physical properties calculated from molecular dynamics simulations. The details are shown in Section B in Supplementary Material,
6 Conclusions
In this study, we introduced a general class of TL based on affine model transformations, and clarified their learning capability and applicability. The proposed afline model transformation was shown to be an optimal class that minimizes the expected squared loss in the HTL procedure. The model is contrasted with widely applied TL methods, such as re-using features from pre-trained, models, which lack theoretical foundation. The affine model transfer model-agnostic; is easily combined with any machine learning models, features, and physical models. Furthermore, in the model, domain- specific factors are involved in incorporating the source features. From this property, the affine transfer has the ability to handle domain common and unique factors simultaneously and separately.
The advantages of the model were verified theoretically and experimentally in this study. We showed theoretical results on the generalization bound and excess risk bound when the regression tasks are solved by kernel methods. It is shown that if the source feature is strongly related to the target domain, the convergence rate of the generalization bound is improved from naive learning. The excess risk of the proposed TL evaluated using the eigen-decay of the product kernel, which also illustrates the effect of the overlap between the source and target tasks. In our numerical studies, the affine model transfer generally outperforms in test errors when the target and source tasks have a similarity. We have also seen in the example of NLP that the proposed affine model transfer can identify the (non-)vate mnon-vvaluable phrases for high-quality papers. This can be done by the affine representation of cross-domain shift and domain-speciff factors in our model.
10

---

Acknowledgments and Disclosure of Funding
This work supported by JST SPRING Grant No. JPMJSP2104, JST CREST Grants No. JP- MJCR2203 and No. JPMJCR 1913, MEXT KAKENHI Grant-in-Aid for Scientific Research on Innovative Areas (Grant No. 19H50820), the Grant-in-Aid for Scientific Research (A) (Grant No. 19H01 132) and Grant-in-Aid for Research Activity Start-up (Grant No. 23K19980) from the Japan Society for the Promotion of Science (JSPS), and the MEXT Program for Promoting Researches on the Supercomputer Fugaku (No. hp210264).
References
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, pp. 84-90, 2012.
[2] G. Csurka, "Domain adaptation for visual applications: A comprehensive survey," arXiv, vol. abs/1702.05374, 2017.
[3] S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf, "Transfer learning in natural language pro- cessing," Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv, vol. abs/ boos/1810..m 810.04805, 2019.
[5] R. K. Sevakula, V. Singh, N. K. Verma, C. Kumar, and Y. Cui, "Transfer learning for molecular cancer classification using deep neural networks," IEEE/ACM Transactions Computational Biology and Bioinformatics, vol. 16, pp. 2089-2100, 2019.
[6] H. Yamada, C. Liu, S. Wu, Y. Koyama, S. Ju, J. Shiomi, J. Morikawa, and R. Yoshida, "Predicting materials properties with little data using shotgun transfer learning," ACS Central Science, vol. 5, pp. 1717-1730, 2019.
[7] S. Wu, Y. Kondo, M. Kakimoto, B. Yang. H. Yamada, I. Kuwajima, G. Lambard, K. Hongo, Y. Xu, J. Shiomi, et al., leaching--erriing---ee assisted discovery of polymers with high thermal conductivity using a molecular design algorithm," npj Computational Materials, vol. 5, no. 1, pp. 1- 2019.
[8] S. Ju, R. Yoshida, C. Liu, K. Hongo, T. Tadano, and J. Shiomi, "Exploring diamond-like lattice thermal conductivity crystals via feature-based transfer learning," Physical Review Materials, vol. 5, no. 5, 053801, 2021.
[9] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," Advances in Neural Information Processing Systems, vol. 27, 2014.
[10] I. Kuzborskij and F. Orabona, "Stability and hypothesis transfer learning," International Confer- ence on Machine Learning. pp. 942-950, 2013.
[11] Kuzborskij and F. Orabona, "Fast rates by transferring from auxiliary hypotheses," Machine Learning, vol. 106, no. 2, pp. 171-195, 2017.
[12] S. Du, J. Koushik, A. Singh, and B. Péczos, "Hypothesis transfer learning via transformation functions." Advances in Neural Information Processing Systems, vol. 30, 2017.
[13] S. Minami, S. Liu, S. Wu, K. Fukumizu, and R. Yoshida, "A general class of transfer learn- ing regression without implementation cost," Proceedings of AAAI Conference on Artificial Intelligence, vol. 35, pp. 8992-8999, 2021.
[14] S. Liu and K. Fukumizu, "Estimating posterior ratio for classification: Transfer learning from probabilistic perspective," " Proceedings of the 2016 SIAM International Conference on Data Mining, pp. 747-755, 2016.
[15] N. Tripuraneni, M. Jordan, and C. Jin, "On the theory of transfer learning: The importance of task diversity," Advances in Neural Information Processing Systems, vol. 33, pp. 7852-7862, 2020.
11

---

[16] L. Dinh, D. Krueger, and Y. Bengio, "Nice: Non-linear independent components estimation," " arXiv, vol. abs/ 1bs/1410.8516, 2014.
[17] L. Dinh, J. N. Sohl- Sick-Dickstein, and Bengio, "Density estimation using real nvp." International Conference Learning Representations, 2017.
[18] T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama, "Coupling- baased invertible neural networks are universal diffeomorphism approximators." Advances in Neural Information Processing Systems, vol. 33, pp. 3362-3373, 2020.
[19] H. Zhou, L. Li, and H. Zhu, "Tensor regression with applications in neuroimaging data analysis." Journal of the American Statistical Association, vol. 108, no. 502, pp. 540-552, 2013.
[20] P. L.. Bartlett, O. Bousquet, and S. Mendelson, "Local Rademacher complexities," Annals of Statistics, vol. 33, pp. 1497-1537, 2005.
[21] I. Steinwart and A. Christmann, Support vector machines. Springer science & business media, 2008.
[22] I. Steinwart, D. R. Hush, and C. Scovel, "Optimal rates for regularized least squares regression," " Proceedings of the 22nd Annual Conference Learning Theory, pp- 79-93, 2009.
[23] C. K. Williams and C. E. Rasmussen, Gaussian processes for machine learning, vol. 2. MIT press Cambridge, MA, 2006.
[24] A. Singh, M. D' Arcy, A. Cohan, D. Downey, and S. Feldman, "SciRepEval: A multi-format benchmark for scientific document representations." " ArXiv, vol.
2022.
[25] C. Finn, P. Abbeel, and S. Levine, aModel-agnostic meta-learning for fast adaptation of deep networks," International Conference on Machine Learning, 2017.
[26] I.. Xuhong, Y. Grandvalet, and F. Davoine, "Fixpicit inductive bias for transfer learning with convolutional networks," International Conference on Machine Learning, pp. 2825-2834, 2018.
[27] S. Myung, I. Huh, W. Jang, J. M. Choe, J. Ryu, D. Kim, K.-E. Kim, and C. Jeong, "PAC-Net: model pruning approach to inductive transfer learning," International Conference on Machine Learning, pp. 16240-16252, 2022.
[28] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04480 2018
[29] I. Beltagy, K. Lo, and A. Cohan, "SciBERT: A pretrained language model for scientific text," in Conference on Empirical Methods in Natural Language Processing, 2019.
[30] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Li, and J. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020.
[31] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Phyym, G. Sastry, A. Askell, el al., "Language models few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[32] R. Vershynin, ffigh-ddmensionall probability: An introduction with applications in data science, vol. 47. Cambridge university press, 2018
[33] S. Wu, G. Lambard, C. Liu, H. Yamada, and R. Yoshida, "iQSPR in XenonPy: A Bayesian molecular design algorithm," " Molecular Informatics, vol. 39, no. 1-2, p. 1900107, 2020.
[34] C. Liu, K. Kitahara, A. Ishikawa, T. Hiroto, A. Singh, E. Fujita, Y. Katsura, Y. Inada, R. Tamura, K. Kimura, and R. Yoshida, "Quasicrystals predicted and discovered by machine learning." " Phys. Rev. Mater, vol. 7, p. 093805, 2023.
[35] C. Liu, E. Fujita, Y. Katsura, Y. Inada, A. Ishikawa, R. Tamura, K. Kimura, and R. Yoshida, "Machine learning to predict quasicrystals from chemical compositions," Advanced Materials, vol. 33, no. 36, p. 2102507, 2021.
12

---

[36] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." International Confer- ence for Learning Representations, 2015.
[37] Wang, R. M. Wolf, W. Caldwell, A. Kollman, and D. A. Case, "Development and testing of general amber force field," Journal of Computational Chemistry, vol. 25, no. 9, pp. 1177-1174, 2004.
[38] Y. Hayashi, J. Shiomi, J. Morikawa, and R. Yoshida, "Radonpy: automated physical property calculation using all-atom classical molecular dynamics simulations for polymer informatics." npj Computational Materials, vol. 8, no, 222, 2022.
[39] S. Otsuka, I. Kuwajima, J. Hosoya, Y. Xu, and M. Yamazaki, "PoLyInfo: Polymer database for polymeric materials design." 2011 International Conference on Emerging Intelligent Data and Web Technologies, pp. 22 29, 2011.
[40] M. Kusaba, Y. Hayashi, C. Liu, A. Wakiuchi, and R. Yoshida, "Representation of materials by kernel mean embedding," " Physical Review B, vol. 108, p. 134107, 2023.
[41] Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization."" Journal of machine learning research, vol. 12, no. 7, 2011.
[42] D. Kingma and J. Ba, "Adam: A method for stochastic optimization." International Confer- ence for Learning Reprexentations, 2015.
[43] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825 2830, 2011.
[44] M. Mohri, A. Rostamizadeh, and A. S. Talwalkar, Foundations of Machine Learning. MIT press, 2018.
[45] N. Srebro, K. Sridharan, and A. Tewari, "Smmoothness, low noise and fast rates." Advances in Neural Information Processing Systems, vol. 23, 2010.
[46] M. Kloft and G. Blanchard, "The local Rademacher complexity of nn-rorm multiple kernel learning," Advances in Neural Information Processing Systems, vol. 24, 2011.
13

---

Supplementary Material
Transfer Learning with Affine Model Transformation
A Other Perspectives on Affine Model Transfer
A.1 Transformation Functions for General Loss Functions
Here discuss the optimal transformation function for general loss functions.
Let fly. 3/) 2 0 be a convex loss function that returns zero and only if y = y', and let g (a) be the optimal predictor that minimizes the expectation of l with respect to the distribution pæ followed by a and transformed by Ø:
ga()) arg min Ep, [((g(2), (y)))
The function that minimizes the expected loss
Ep. [((g(z). /ʃ elg(z), (U))), yddady
should be solution to the Luler-Lagrange equation
0g(x) /
8 ((((x), Φy
[(g(a), (v)) yddy Øg(x))
= 0. (S.1)
Denote the solution of Eq. (S.I) by ((=;;,,)) While G depends on the loss and distribution Pt- we omit those from the argument for notational simplicity. Using this function, the minimizer of expected loss function.
v)] can be expressed as G(æ;; id), where id represents the identity
Here, we consider the following assumption hold, which generalizes Assumption 2.3 in the main text:
Assumption 2.3(b). For any distribution on the target domain pe(ar, u) and all X, the following relationship holds:
(g" (x:) arg min E. [((g(()), y)]. Equivalently, the transformation functions 6s, and v. satisfy
5. (G( o, G(æ;id).
(S.2)
Assumption 2.3(b) states that if the optimal predictor G(æ; o5,)) for the data transformed by é given the model transformation function vh, it is consistent with the overall optimal predictor G(æ;; id) in the target region in terms of the loss function &. We consider all pairs of sb and á that satisfy this consistency condition.
Here, let us consider the following proposition:
Proposition A.1. Under Assumption 2./ 2.2 and 2.3(b)
17. 05...
Proof. The proof is analogous to that of Theorem 2.4 in Section D.I For any yo E V, let pr(y/z) = 8y. Combining this with Eq. (S.I) leads to
(g((x), or (un) = 0 (Vyo E )).
0g(x))
Because l'y.y') returns the minimum value zero if and only if = y', we obtain G(æ; Φr.) d (yo). Similarly, we have G(ær; id) = Yo- From these two facts and Assumption 2.3(b) we have y (0 (vo) yo. proving that the proposition is true.
14

---

Proposition A.1 indicates that the first statement of Theorem 2.4 holds for general loss functions. However, the second claim of Theorem 2.4 generally depends on the type of loss function. Through the following examples, we describe the optimal class of transformation functions for several loss functions.
Example (Squared loss). Let E(y, v/) = ly y' As solution of Eq. (S.I) we can see that the optimal predictor is the conditional expectation Ep, (x) = z]. As discussed in Section 2.1 the transformation functions Øy, and y,y, should be affine transformations.
Example 2 (Absolute loss). Let fly, y²) = yy - 5//. Substituting this into Eq. (.I we have
dg(x) ((x) o, (v) (v x)dy
(ggn(g(>) Φz. (v) p. (y/a)dy
Pa (olla)dd - /
pr(vla)dy-
(1)
(v)<(()))))
Assuming that oy, is monotonically increasing, we have
/
0
poly)a)dy - (007) 12 0207 (0(7)(((m)) (m(m))
pe(vlx)dy
This yields
Pr(yla)dy (g(2))
6, (o(x))
The same result obtained even if Φs, monotonically decreasing. Consequently,
(g(x)) Median mr).
which results in
G( )1,) = Us. (Median[Y| X =+).
This implies that Eq. S.2) holds for any Φy. including an affine transformation, and the function form cannot be identified. from this analysis.
Example 3 caaponnntiallsquaaeee loss). As an example where the affine transformation is not optimal, consider the loss function ((y, ) = je" - 12 Substituting this into Eq. (S.1), we have
f 0g(x)
0 eep(((()) exp(0,, (v)) (((lx)dy
0
= 2 (xx)( /
(exp(e(x)) - exp(os,
Therefore,
G(æ; = log E [exp((s, (Y)) m].
Consequently, Eq. S.2
becomes
log (()) = o,, (log E (Yx)())))
Even if o is an affine transformation, this equation does not generally hold.
A.2 Analysis of the Optimal Function Class Based on the Upper Bound of the Estimation Error
Here, we discuss the optimal class for the transformation function based on the upper bound of the estimation error.
In addition to Assumptions 2.1 and 2.2 we assume the following in place of Assumption 2.3
Assumption A.2. The transformation functions ó and i are Lipschitz continuous with respect the first argument, i.e., there exist constants as and no such that,
ó(a.c) é(a', c) S polla a' d'l12, (bb,c) (ib.' c) s Hwllb 6 b' 1|2 for any a' e and b, b' I with any given e F..
15

---

Note that each Lipschitz constant is a function of the second argument for i.e., ps = (o(fe) and My (fx)..
Under Assumptions
2.1
2.2 and A.2
the estimation error is upper bounded as follows:
E
[f(())) - f((x))]]
E
(g(a)) fa) ((0(()
( (()(f()))))((()) g(x)
m.
E8 a.
Us (fa(a)))) ((g(z) ((I(<))
- o(v. + - fo()))
+lo(y, -(())
< 33 E a.y
Me [x (Un())
+3 Me ( yl
a.
+ +3
2.1
E 2.y
us
(
+ 30
E
My
(fo (x)) (f)
a.
E 2.y
pa (fa
(()
The derivation of this inequality is based on [2]. We use the Lipschitz property of eb and ø for the first and third inequalities, and the second inequality comes from the inequality (a d)² s 3(a- - 6)2 + 3(b - c)² + 3(c - d)² for a, b, d E R.
According this inequality, the upper bound of the estimation error is decomposed into three terms: the discrepancy between the two transformation functions, the variance of the noise, and the estimation error for the transformed data. Although it is intractable find the optimal solution of é, v. that minimizes all these terms together, it is possible to find a solution that minimizes the first and second terms expressed as the functions of and v only. Obviously, the first term, which represents the discrepancy between the two transformation fuunction, reaches its minimum (zero) when o, = 7 The second term, which is related to the variance of the noise, is minimized when the differential coefficient
σy thy, (u) is constant, i.e., when y is linear function. This is verified as follows. From ob. = / and the continuity of s it follows that
@
i Ou v. (w) Me max
i 0u 1. (1))
b max
min
a Bu vy. (u)l
and thus the product Halpy takes the minimum value (one) when the maximum and minimum of the differential coefficient are the same. Therefore,
write
@uy, fa) 3 - -g((fa)
(o(()) gf()) + go(f))((m),
92(fx)
where 91, .2 F. + I are arbitrarily functions. Thus, the minimization of the third term in the upper bound of the estimation error can expressed as
min Ely 91 (f.) 92 (ff))(()))) 91-92-9 s.y
As result, the suboptimal function class for the upper bound of the estimated function is given
H = (-++ (Ia))) + 92(fa) ga(x) 91 E G1, g2 E G2, ga Ga).
This is the same function class derived in Section 2.1
16

---

(a) Direct learning
(b) Feature extraction
(c) HTL-offset
(d) Affine model transfer
Figure S.I: Model architectures for the affine model transfer and related procedures. (a) Direct learning predicts outputs using only the original inputs z, while (b) feature extraction-based neural transfer predicts outputs using only the source features fs. (c) The HTL procedure proposed in
10 (fTL-offseet constructs the predictor as the sum of 91 (f+)) and (z)).. (d) The affine model
transfer encompasses these procedures, computing and O2 as functions of the source features and constructing the predictor as an affine combination with f3-
B
Additional Experiments
B.1 Eigenvalue Decay of the Hadamard Product of Two Gram Matrices
We experimentally investigated how the decay rate 82 in Theorem 4.4 is related to the overlap degree in the spaces spanned by the original imput æ and source features fs.
For the original input a E R2000 we randomly constructed a set of 10 orthonormal bases, and then generated 100 samples from their spanning space. For the source features f. E 10: we selected d bases randomly from the 10 orthonormal bases selected for æ and the remaining their orthogonal complement space. We then generated 100 samples of fa from the space spanned by these bases. The overlap number can regarded as the degree of overlap two spaces spanned by the samples of and f.- We generated the 100 different sample sets of æ and f..
- bases from
We calculated the Hadamard product of the Gram matrices K₂ and K3 using the samples of æ and fe. respectively. For the computation of K2 and Kg, all combinations of the following five kernels were tested:
x. x
Linear kernel k(æ, x²)
2n2
-1-v (V20 x 112
(v20/1 1/20la0 x'112 for 2 oc,
1 3
Matérn kernel k<x x²)= T(v)
Kg
7
where (-(-) a modified Bessel function and I(.) the gamma function. Note that for y = co, the Matérn kernel is equivalent to the Gaussian RBF kernel. The scale parameter of both kernels was set to 7 /iim æ>) v10. For given matrix K, the decay rate of the eigenvalues was estimated as the smallest value that satisfies X; S || i-! where |[ denotes the Frobenius norm. Note that this inequality holds for any matrices K with
32
Figure S.2 shows the change of the decay rates with respect varying for various combinations of the kernels. In all cases, the decay rate of K2 o K3 showed clear trend of monotonically decreasing as the degree of overlap increases. In other words, the greater the overlap between the spaces spanned by æ and f., the smaller the decay rate, and the smaller the complexity of the RKHS H2 H3.
B.2 Lattice Thermal Conductivity of Inorganic Crystals
Here, we describe the relationship between the qualitative differences in source features and the learning behavior of the affine model transfer, in contrast to ordinary feature extraction using neural networks.
17

---

Kernel function Ao
Linoar
Matér- 1/2
Matern 3/2
Matórn 5/2
Gaussian
Kernel lunciton As
0i1ii01 Marérr-SS Mattrm-32
Caussion
Figure S.2: Decay rates of eigenvalues K₂ (blue lines), K3 (green lines) and K₂ K3 (red lines) for all combinations of the five different kernels. The vertical axis represents the decay rate, and the horizontal axis represents the overlap dimension in the space where and fa are distributed.
The target task is predict the lattice thermal conductivity (LTC) of inorganic crystalline materials, where the LTC is the amount of vibrational energy propagated by phonons in a crystal. In general, LTC can be caleulated ab initio by performing many-body electronic structure caleulations based quantum mechanics. However. it is quite time-consuming to perform the first-principles calculations for thousands of crystals. which will be used as a training sample set create a surrogate statistical model. Therefore, we perform TL with the source task of predicting an alternative, computationally tractable physical property called scattering phase space (SPS), which known be physically related to LTC.
B.2.1 Data
used the dataset from S that records SPS and LTC for 320 and inorganic compounds, respectively. The input compounds were translated to 290-dimensional compositional descriptors
using XenonPy 6 33 34
B.2.2 Model Definition and Hyperparameter Search
Fully connected neural networks used for both the source and target models, with LeakyRel.U activation function with a = 0.01. The model training was conducted using the Adam optimizer Hyperparameters such as the width of the hidden layer, learning rate, number of epochs, and regularization parameters were adjusted with 5-fold cross-validation. For more details on the experimental conditions and procedure, refer the provided Python code.
33).
Source Model For the preliminary step, neural networks with three hidden layers that predict SPS were trained using 80% of the 320 samples. 100 models with different numbers neurons were randomly generated and the top 10 source models that showed the highest generalization performance the source domain were selected. The hidden layer width L was randomly chosen from the range [50, 100), and we trained neural network with structure of (input)-L-L---I. Each of the three hidden layers of the source model was used as an input to the transfer models, and we examined the difference in prediction performance for the three layers.
Target Model In the target task, an intermediate layer of source model was used as the feature extractor. A model trained using 40 randomly chosen samples of LTC, and its performance was evaluated with the remaining 5 samples. For each the 10 source models, we performed the
18

---

Feature extractor
Atline traneler Without tranefer
Fine-tuning
25
mean
1et 1e
2nd lnyer Extracted bayor
3rd bayer
Figure S.3: Change of RMSE values between the affine transfer model and the ordinary feature extractor when using different levels of intermediate layers as the source features. The line plot shows the mean and 95% confidence interval. As baseline, RMSE values for direct learning without transfer and fine-tuned neural networks are shown as dotted and dashed lines, respectively.
training and testing 10 times with different sample partitions and compared the mean values of RMSE among four different methods: (i) the affine model transfer using neural networks to model the three functions 91, 92 and g3. (ii) a neural network using the XenonPy compositional descriptors input without transfer, (iii) neural network using the source features as input, and (iv) fine-tuning of the pre-trained neural networks. The width of the layers of each neural network, the number of training epochs, and the dropout rate were optimized during 5-fold cross-validation looped within each training set. For the affine model transfer, the functions gl, 92. and ga were modeled by neural networks. We used neural networks with one hidden layer for 9l, 92 and 93.
B.2.3 Results
shows the change in prediction performance of TL models using source features obtained from different intermediate layers from the first to the third layers. The affine transfer model and the ordinary feature extractor showed opposite patterns. The performance of the feature extractor improved when the first intermediate layer closest to the input layer was used as the source features and gradually degraded when layers closer to the output were used. When the third intermediate layer was used, a negative transfer occurred in the feature extractor as its performance became worse than that of the direct learning. In contrast, the affine transfer model performs better the second and third intermediate layers closer to the output were used. The affine transfer model using the third intermediate layer reached a level of accuracy slightly better than fine-tuning, which intuitively uses more information to transfer than the extracted features.
Figure S33
In general, the features encoded in intermediate layer of neural network more task insk-independent as the layer is closer to the input, and the features are more task specific as the layer is closer to the output [9] Because the first layer does not differ much from the original input, using both and fa in the affine model transfer does not contribute much to performance improvement. However, when using the second and third layers the feature extractors, the of both ær and fa contributes to improving the expressive power of the model, because the feature extractors have acquired different representational capabilities from the original input. In contrast, a model based only on f. from a source task-specific feature extractor could not account for data in the target domain, so its performance would become worse than direct learning without transfer, i.e., negative transfer would occur.
B.3 Heat Capacity of Organic Polymers
We highlight the benefits of separately modeling and estimating domain-specine factors through case study in polymer chemistry. The objective is to predict the specific heat capacity at constant pressure Cp of any given organic polymer with its chemical structure in the polymer's repeating unit. Specifically, we conduct TL to bridge the gap between experimental values and physical properties calculated from molecular dynamics (MD) simulations.
19

---

IDD-calclaadd beat capacity (log, 3746.5
100
1900
Correlation 0.7788 3000
abs ebe
Exporimentally obsorved heat capacity (log 2/1g
Figure S.4: MD-calculated (vertical axis) and experimental values (horizontal axis) the specific heat capacity at constant pressure for various amorphous polymers.
Table S.I: Force field parameters that form the General AMBER force field 37 version (GAFF2), and their detailed descriptions.
Parameter mass
Description
Atomic mass
Equilibrium radius of van der Waals (vdW) interactions
Depth of the potential well of vdW interactions
charge
Atomic charge Gasteiger model
ro
Equilibrium length chemical bonds
boond polar
Force constant of bond stretching
Bond polarization defined by the absolute value of charge difference between atoms in a bond Equilibrium angle of bond angles
Oo
K Kangle Kaib
Force constant of bond bending
Rotation barrier height of dihedral angles
As shown in Figure S.4 there was large systematic bias between experimental and calculated values; the MD-calculated properties CM exhibited an evident overestimation with respect to their experimental values. As discussed in [38), this bias is inevitable because classical MD calculations do not reflect the presence of quantum effects in the real system. According to Einstein's theory for the specific heat in physical chemistry, the logarithmic ratio between CExp and CMD can be calibrated by the following equation:
fiw k
exp
huu KnT
log C = log Cp + log
+ log
(S.3)
2
exp
K
where KB is the Boltzmann constant, h is the Planck constant, is the frequency of molecular vibrations, and T' is the temperature. The bias is monotonically decreasing function of frequency w, which is described as black-box function of polymers with their molecular features. Hereafter, we consider the calibration of this systematic bias using the affine transfer model.
B.3.1 Data
Experimental values of the specific heat capacity of the 70 polymers were collected from PoLyInfo 39]. The MD simulation was also applied to calculate their heat capacities. For models to predict the log-transformed heat capacity, a given polymer with its chemical structure was translated into the 190-dimensional force field descriptors, using RadonPy [38
The force field descriptor represents the distribution of the ten different force field parameters ( t T (mass, a, c, charge, ro, K Kboud, polar, f, K Kangle, Kath) that make up the empirical potential (i.e., the General AMBER force field [77 version 2 (GAFF2)) of the classical MD simulation. The
httt::://ggtaab..cc/aaaaaaa////
20

---

Algorithm S.1 Block relaxation algorithm for the model in Eq. (S.4).
Initialize: f âo,ott f oot, B <- 0, 7 - Vair
repeat
a f arg mina Fa,p,7
8 - arg ming Fa.3,7
7 arg miny 8,,877
unfil convegence
detailed descriptions for each parameter are listed in Table SS. For each f, pre-defined values are assigned their constituent elements polymer, such individual atoms (mass, charge, a, and e), bonds (ro. Kbond. and polar), angles (0g and Kangle) or dihedral angles (Kaih). respectively. The probability density function of the assigned values of is then estimated and discretized into 10 points corresponding to 10 different element species such as hydrogen and carbon for mass, and 20 equally spaced grid points for the other parameters. The details of the descriptor calculations are described in [40]
The source feature Js given as the oog-transformed value of CMD. Therefore, is longer function of æ; this modeling was intended for calibrating the MD-caleulated properties.
We randomly sampled 60 training polymers and tested the prediction performance of trained model on the remaining 10 polymers 20 times. The PoLyInfo sample identifiers for the selected polymers are listed in the code.
B.3.2 Model Definition and Hyperparameter Search
As described above, the 190-dimensional force field descriptor consists of ten blocks corresponding to different types of features. The J; features that make up block t represent discretized values of the density function of the force field parameters assigned to the atoms, bonds, or dihedral angles that constitute the given polymer. Therefore, the regression coefficients of the features within a block should be estimated smoothly. To this end, we imposed fused regularization the parameters as
Au 1 + A2
(7.j - 1
LET 1-2
where T = {mass, charge, e, a, Kbond, To: Kangle 0, Kaij. , and Jr 10 for = mass and J, = 20 otherwise. The regression coefficient "t. corresponds to the j-th feature of block t.
Ordinary Linear Regression The experimental heat capacity log Cp was regressed on the MD-calculated property, without regularization, ox + a where ÿ denotes the conditional expectation and fa log CAI..
Learning the Log- Log-Difference We calculated the log- digffefeeeenc log Cpxp log CM and trained the linear model with the ridge penalty. The hyperparameters ê₁ and A2 for the scale- and smoothness- regularizers were determined based 5-fold cross validation across 25 equally space grids in the interval [10-2, 10 for x and across the set (50, 100, 150} for X2-
Affine Transfer
The bog-transformed value of C is modeled as
y log Op
ao aife (8f. 1) x TT tco
(S.4)
where e, represents observation noise, and oo, 0n, 3 and y are unknown parameters to be estimated. When o1 and = 0, Eq. (S.4) consistent with the theoretical equation Eq. (S.3) which the quantum effect is linearly modeled as ag + x TTy.
21

---

Table S.2: Mean and standard deviation of RMSE of three prediction models.
Model
RMSE (log J/kg K)
y 0g + +111s to
0.1403 0.0461
y fa x 7 Eg
0.1368 0.04265
y 00 + aif.
(3fa + 1900 6e
0.1357 + 0.04173
In the model training, the objective function was given as follows:
12
Fa,B,r =
Σ 71
{yr - (a0 + aa - (3Sa, + 1)xx >)}
+ A0 + >>.11 + Xy.2
(7.. )))-))
j=2
where |00 a1]". With fixed X3 1, the remaining hyperparameters X,, and Xy, were optimized through 5-fold cross validation over 25 equally space grids in the interval [10-3 10 for Xy.1 and across the set {50, 100, 150} for 2,,2-
The algorithm to estimate the parameters a, 8 and is described in Algorithm S.I where og and Q1,oir are the estimated parameters of the ordinary linear regression model, and YYur the estimated parameter of the log-difie cog-difference model. For each step, the full conditional minimization of Fa, with respect to each parameter can be made analytically as
arg min Pa,p, (P,, F.)) F. (y + (8fa.1m 1) (X7)), ( arg min Fa.9.y = - (f diag( X X7)² fx, + (m2)2= fa, diag( Xx)(y F5a + X7),
arg min F Fa.B,y -xx" (iag(Bf... ff 1)2X + A) -1 x diag fo fl + 1) F,a),
where X denote the matrix in which the i-th row is [y ............ yn [T Sa = [Sa,1 ... Ss,n]
F. = =ss,11n 1), and 190. a matrix including the two regularization parameters xy, and
Xy.2 as
1
A = D. where D
] M 2
dy,1d
m-th rows
where m {10, 30, 50, 70, 90, 110, 130, 150, 1 170). Note that the matrix M is the same the matrix (0 J189] - [759 0) except that the m-th row is all zeros. Note also that M E TR 18188x190 and therefore D 2279x190 and A IR 1900x190
The stopping criterion of the algorithm was set as
max; 0
(new) (old)
10
(S.5)
AE (o.
max,
10 (lll))
where denotes the i-th element of the parameter 0. This convergence criterion is employed in several existing machine learning libraries, e.g., scikit-leam
B.3.3 Results
Table S.2 summarizes the prediction performance (RMSE) of the three models. The ordinary linear model y = au + a1fs co. which ignores the force field descriptors, exhibited the lowest prediction
wwwpaaaann..nnnnmonoeaeeeea sociarrmoodl..o Lasso.
htm
22

---

Figure S.5: Bar plot of regression coefficients y of linear calibrator filling the discrepancy between
experimental and MD-calculated specific heat capacity of amorphous polymers.
performance. The other calibration models y = S. + x: ca and the full model in Eq. S..4) reached almost the same accuracy, but the latter had achieved slightly better prediction accuracy. The estimated parameters the full model were a1 0.889 and / -0.004. The model form highly consistent with the theoretical equation in Eq. (S.3) as well as the restricted model (a) = 1, = 0). This supports the validity of the theoretical model
is expected that physicochemical insights can be obtained by examining the estimated coefficient y which would capture the contribution of the force field parameters the quantum effects. The magnitude of the quantum effect is a monotonically increasing function of the frequency w. and known to be highly related the descriptors e, Kain. Kbond, K Knngle and mass. According to physicochemical intuition, it is considered that as Khond. Kangle, and Kaih decrease, their potential energy surface becomes shallow, which leads to the decrease of w, and in turn the decrease of quantum effects. Furthermore, because the molecular vibration of light-weight atoms is faster than that of heavy atoms, w and quantum effects should theoretically increase with decreasing mass.
Figure S.5 shows the mean values of the estimated parameter r for the full calibration model. The physical relationships deseribed above be captured consistently with the estimated coefficients. The coefficients lower regions of c, Kbond Kagge and Kaih showed large negative values, indicating that polymers containing more atoms, bonds, angles, and dihedral angles with lower values will have smaller quantum effects. Conversely, the coefficients in lower regions of mass showed positive large values, meaning that polymers containing more atoms with smaller masses will have larger quantum effects. As illustrated in this example, separate inclusion of the factors in the affine transfer model enables infer the features relevant to the cross-domain differences.
and domain domain-specifie
C Experimental Details
Instructions for obtaining the datasets used in the experiments are described in the code.
C.1 Kinematics of the Robot Arm
C.1.1 Data
We used the SARCOS dataset in (33 The task is to predict the feed-forward torque required to follow the desired trajectory in the seven joints of the SARCOS anthropomorphic robot arm. The twenty one features representing the joints' position, velocity, and acceleration were used observed values of six torques other than the torque at the joint in the target domain were given the source features fa- The dataset includes 44,484 training samples and 4,449 test samples. We selected (5. 10, 15, 20, 30, 40, 50} samples randomly from the training set. The prediction performances of the trained models were evaluated using the 4,449 test samples. Repeated experiments were conducted 20 times with different independently sampled datasets.
The
C.1.2 Model Definition and Hyperparameter Search
Source model For each target task, multi-task neural network was trained to predict the torque values of the remaining six source tasks. The source model shares four layers (256- 12--6433) up to the final layer, and only the output layer task specifei We used training data and Adagrad 41 with learning rate of 0.01.
23

---

Algorithm S.2 Block relaxation algorithm for AffineTL-full.
Initialize: ao (Kı A1 bo NO, [n), ~ NOO, In), do 0.5
repeat
(Ki + A1 - (Kyb 1) (Kzc) - d1)
Diag K3c) (,,c))KK1 + ^2 (Kg
aia- K3c- d1))
c (Diag(Kzb K3 )342) (K2b++) Kia d1))
d - (y Kia- (K 1
until convergence
Direct, Only source, Augmented, HTL-offset, HCL-e For each procedure, used kernel ridge regression with the RBF kernel k(x,a' x²) exp( 22 | -a--ra x' The scale parameter & was set to the square root of the input dimension as l = 221 for Direct, HTL-offset and HTL-scale, = /6 for Only souree and & = v27 for Augmented. The regularization parameter À was selected in 5-fold cross cross- validation in which the grid search was performed over 50 grid points in the interval
[10-+, +,10).
Affine TL-full, Affine TL-const
We considered the following kernels:
(f) exp(
26 [f(()) fa())11 (e V6),
(a(f fo fz()) exp(
[fe())) fo(x')11) (e= v6),
2/2
ka exp
- -
x'
x²133) (e V27),
for 91 and 93 in the affine transfer model, respectively.
Hyperparameters be optimized are the three regularization parameters Å A2 and Ag. We per- formed 5-fold cross-validation to identify the best hyperparameter set from the candidate points; (10-3, 100 10 1} for and {10 2, 10 10} for each of X2 and λ3.
To learn the AffineTTLL TL-full and AffineLL-come we used the following objective functions:
AffineTL-full - (Kia+ (s0 +1)) ( Aaa Kia+ dab K2b+ Age Kyc,
AffineTL-const
ll (Kia + K3c + d1) Aia Kia + A3e Kzc.
n
Algorithm S.2 summarizes the block relaxation algorithm for AffineTL-full For AlfineTL-const. we found the optimal parameters as follows:
à à
((
K1
A1
1) 0
K 3
=
K3 1
K1
K3
1
1
Ag
y
d
The stopping criterion for the algorithm was the same Eq.
S.5
Fine-tuning The target network was constructed by adding a one-dimensional output layer to the shared layers of the source network. As initial values for the training. used the weights of the source neural network for the shared layer and the average of the multidimensional output layer of the source network for the output layer. Adagrad (41 used for the optimization. The learning at 0.01 and the number of training epochs was selected from (1, 3, 4, 5, 6, 7, 8,8, 9, 10, 20, 50, 100) through 5-fold cross-validation.
was fixed
MAML A fully connected neural network with 356-64-32-16 16-1 layer width was used, and the initial values were searched through MAML 25 using the six source tasks. The obtained base model was fine-tuned with the target samples. Adam 42 with fixed learning rate of 0.01 was used for the opti- mization. The number of training epochs was selected from (1, 3, 4, 5, 6, 7, 9, 10, 20, 50, 100) through 5-fold cross-validation.
24

---

LP-SP L2-SP is regularization method proposed by 226 in which the following regularization term added so that the weights of the target network are estimated in the neighborhood of the weights of the source network:
X 1/ee - w2 2
((w)
(S.6)
where w and ws are the weights of the target and source model, respectively, and > hyperparameter. We used the weights of the source network as the initial point for the training of the target model, and added a regularization parameter as in Eq. (S.6). Adagrad [41] were used for the optimizer, and the regularization parameters and learning rate were fixed at 0.01 and 0.001, respectively. The number of training epochs was selected from 21, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100} through 5-fold cross-validation.
PAC-Net PAC-Net, proposed in 27 is TL method that leverages pruning of the weights of the source network. Its training strategy consists of three steps: identifying the important weights in the source model, fine-tuning them using the source samples, and updating the remaining weights using the target samples.
Firstly, we pruned the bottom 10% of weights, based on absolute value, from the pre-trained source network. Following this, the remaining weights were retrained using the stochastic gradient descent (SGD). Finally, the pruned weights were retrained using target samples. For the final training phase, SGD with learning rate 0.01, was employed, and the number of training epochs was selected from {1, 2, 4, 6, 8, 9, 10, 20, 50, 100} through 5-fold cross-validation.
25

---

Table S.3: Performance on predicting the torque values at the first and seventh joints of the SARCOS robot arm. Definitions of an asterisk and bold type are the same as in Table
Number of miinnigg samples
Target
Model
d
1 2.
20
30
50
Direct
2.11
17.4 1.79
1.70
10.8 1: 1.23
Only source
24.0 6.37
22.3 3.10
21.0
2.49
19.7
1.34
18.5
1.92
17.6 1.59
17.3 1 1.31
Augmented
21.8 1:2:::2. 2.88
19.2 1.37
17. 1 2.30
15.7
1: 1.53
13.3
1.19
11.9
1.37
10.7 - 0.954
Hts-effset
23.7 1: 6.50
19.8 4 3.23
17.8
2: 2.35
16.2
15.0 3.16
15.1 1 2.76
HTL scale
23.3 4,47
22.1 5.31
20.4 3.84
18.5 2.72
17.6 2.41
2.10
16,7 1.71
Torque
AftineTT--til
2.28
18.8 1.31
18.6 2,83
15.9
1.53
12.3 1.45
11.1 1.12
AflinTLL-coms
2:21
17.7 2.44
15.9
109 1.02
25.0+ 7.11
20.5
18.6 2.10
2.55
1.39
126- 1.13
11.1 1.03
MAML
29.8
22.5 + 3.21
20.8 + 2.12
20.3 3.14
16.7 3.00
14.4 1.85
13.4 1.19
SP-SP
24.9
20.5 1: 3.30
18.8 4: 2.04
18.0 2.45
14.5 2: 1.36
13.0 1.13
11.6 0.83
PAC-Net
25.2
22.7 5.60
20.7 2.65
20.1 1. 2,16
18.5 2.77
17.6 1.85
17.1 1.38
Direct
1..8
1,41
11.5 0:985
10.4 0,845
9,20-0827
8:35
7.78
Only sonrce
14.9+1.7
13.6 2.51
12.3 1.77
1.16
9.74
Augmented
15..2:
12.3 1: 0.923
11.4 1.48
10.2 0.813
9.07 0.983
8,06 0.862
7:23 0.629
Hol-ef
14.8+1.71
13.4 2.41
12.2 + 1.81
10.9 1.29
10.4 1.37
9.32 1.11
8 0829
HTL-scale
14.8
13.4 2,47
12.2 1.82
11.0
10.5 1,28
9.39 1.01
8.1 0.946
Torque
ArtineTL-tiil
1277733
119
11.39
9.38 - 0634
R 0.912
7:44 0008
AfhneTT--const
14.6
1.09
11.5 0.807
0.828
8.35
7.33 0.57
Fine-tune
2587
1
13.6
1:211
MAML
2.3
14.8 4.511
13.1 4: 2.69
11.5 2:2.4
9.77 1 1.24
1.10
7,89 0.733
12-SP
5.87
15.1 2.02
13.6 + 2.29
12.0 1.22
10.8 0.866
9.70 0.78
8.68 0.868
PAC-Net
24.0 6.94
16.7 + 4,14
13.7 2,36
13.2 + 2.49
12.4 2.05
11.6 + 0.844
11.2 0,706
Direct
1.91 1.65
8.15 1 1,01
7.39
1.21
6,84 0.878
5.90 0.800
5.26 0.774
4.66 0.523
source
7.51
6.90
Augmented
7.72 1.05
6.99
1,25
0.967
0.988
4.76 0.826
4.30
offee
8,96 1. 1.42
7,47 1: 1,06
6.88 1 1.15
6,39 1: 0.952
5.58 0.356
5.18 0.221
4.83 0.600
HTL-scale
9.05 + 1.40
7,49 + 1.08
6.89 + 1.18
6.63 + 1.03
5.00 0955
5.21 = 0.836
4.6 0.503
Torque 3
AfinnTL--ful
9.24 1.46
7.45 1.25
6.85 1 1.23
6.28 0,930
5.54 L.5
4.89 1 0.907
4.46 _ 0.733
9,08 1,21
7,55 1. 0,974
6.67 1,00
6.17 0.916
5,42 0.971
4,85 0.752
4.42 0614
Fine-tune
9.00 2.14
7.38 1,09
6.72 1.01
*5.91
0.734
26.26-1 0541
4 0.488
4.41 0.325
MAML
1:50::44:4
#7.11
1.01
9.992:1
0.743
0626
0.539
4.79
12-SP
9,00 2.14
7.39 1: 1.08
6,73 4. 1.02
95.91 0.73
5.39 0.63
4.89 +: 0.493
4,46 0.319
PAC-Net
9.14++2.11
#7.31 + 1,03
*6.33 + 0.841
"5 96
= 0.926
5.34 0,63
5.17 + 0.474
5.05 0.371
Direct
14.2+
11.1 2.28
9.49 + 2.19
7.78 + 1.02
6.86 - 0.768
6.13 + 0.714
5.88 - 0.592
Only
13.1.1
9.62 1: 2.05
8.38 2,06
7,06 1 1.32
6.36 1 1.24
5.79 0.768
5:37 0.997
Augmented
13.5
1,89
1.84
-600
HTl-offeet
13 3.38
9.62 2.05
8.34+ 2.00
1.24 7.01
7.02
5.76 0764
5.3
Hale
13.0 1.0.1... 3.35
9.63 1. 2.07
8,30 1.95
1.16
6.30 1 1.17
5.77 1 0.758
5:37 0,002
Torque 4
AftlneT--full
13:0 + 2.69
9.48 + 2.10
8.38 + 1.85
7.14 1.62
"5.91
003
*5.45 0.777
*4.4 0.03
AlincTLL-coms
1322113.11
+9.32 1: 1,99
8.39 1 1.84
66.88
1: 1,00
0.710 "5.8
55.55 1 0.679
14.94 449-0050 0.581
Fine-tune
*11.7=
*8.24 + 1,31
76.71 + 1.02
"S 90.900-58 0.971
"5.17 - 0.785
*4.59 0.442
"421 = 0.376
MAML
7.75
10.9 3.44
9.55 1,99
9.41
7.98 2.36
6,70 1.25
6.18 1.35
SP
1.177:1..700
1.11
"5.92
0.959
"5.22 0.765
*4.67 0.45
0,33
PAC-Net
11.2
+8.84 2.75
*7.64 1 1.17
7.34 L 1.56
6.77 0.966
6.29 1 0.536
600 0.16
Direct
1.07
0.993 + 0.0903
0.10 + 0.119
0.847
0.129
0.744 0.113
0 0.0996
6.613 + 0,0944
Only source
1,15 115.2:0..10 0,214
L.4 + 0.0775
0.998
+ 0.145
0
= 0.133
0363 0.111
0,826 = 0.155
0,775 = 0.106
Auguented
1.04
0.967 0,109
0,907
4 0,120
0.874
0.136
0,755 0.130
0,710 0.110
0.37 0.0993
Hfl-effset
1.14 0.221
1.02 0.0864
0.157
0.141
0X37 0.104
0,500 0.156
0.101
HTL-scale
0.194
1.01 0.0786
0.980
0.177
0.14 0.132
0330 0.114
0.844 0.171
Torque
"Do8o 0.175
69907
+ 0.
66176
"054 -617
T 00024
"634 6,076
AffneTTL-zoust
1.05 4 0.106
0.974 + 0.102
0.899 + 0.123
0.1220
0,756 2000-0.000 0.106
0,200 = 0.0809
0.638 = 0.0996
Fine-iume
0.356
L.4 0.105
0.976 1 0,0878
09131001
0.749 0.111
0,688 1 103
0.598 1 0.0697
MAMI.
0.479
L18 + 0.183
1.07 + 0.208
0.999
= 0.193
0816 0211
07703 0 124
0613 + 0.334
17-SP
0.385
0.973
0.0873
0.17
0.133
0.55 0.109
0.06 00044
PAC-Net
0.319
1.03 +
0,985
0.145
0881
0.151
Direct
1,85 0.244
1.7 1 0.192
1,50 1 0.162
1.36 0.159
1.22 0.163
1.12 1 0,102
1.05 0.0916
Only sounce
1.91 1: 0.250
1.87 1 0.357
1.76 1 0.179
1.64 = 0.190
1.53 0.255
1.36 1: 0. 153
1.26 2 08883
Augmented
1.85 2: 0.180
L66 1: 0.17
L.55 4 0.219
LA45 = 0.231
1 0.265
111 2 0.130
1.1 1 0,0944
ofl-et
1.88 0.214
1881 1 0.369
L67 1 0.246
1.51 0.239
1.16 0.267
1.33 0.142
1.21 0.133
HTL-scale
0.649
1.71 0.241
1.60 0,308
0.474
1.25 0.0222
Torque
AfiinTL--fuli
0.232
1.74
1.55 0.242
0.231
0.230
1 0.122
1.06
AlineTLL-cems
1,83 0.173
1.68 1: 0.207
1,55 1: 0.236
1.40 0.227
2323 0.198
1.12 1 0.113
1.02 2 0.0053
tue-ttune
2.41 0.375
2.03 + 0.387
1.71 + 0.463
1.49 0.297
1.27 0227
1.15 = 0.110
1.00 = 0.0817
MAML
2.69 2: 0,676
2,17 1: 0.511
1.96 4: 0.526
1.68 =: 0.373
L42 = 0.265
1.23 1: 0.111
1.16 2 0.0629
1P-SP
2.41 0.375
2,03 0.371
1.72 1. 0.155
1.19 0.299
1.28 0.251
1.16 0.108
1.09 1 0.0777
Net
0.3385
2.22
2.22
0.372
0.355
0.0595
Direct
2.66+ 0.307
2.13 +
1,85 +
1.54=
0.200
1.05
Oaly source
2,31 4: 0.618
1173 +: 0.560
*1.48 4: 0.513
"1.22 0.269
*1.09 02232
Y0969
0.144
*0927 = 0.170
Augmentad
2,47 + 01,406
1.90 + 0.515
1.67 + 0.552
*1.31 + 0.214
1.16 0.225
*0.844
0. 149
10,07 0.138
HTL-offeet
2.29 1.00++0..00 (1622
*1.69 + 0.507
*1.49 + 0.513
*1.22 0.269
*1.09 0.238
0.144
10925
0.171
2.32 0.599
.1,71 1: 0.516
1.51 1. 0.513
11.24 0,271
-1.12 0.234
*0,999
0.1775
0948
0.172
Torque
AftineTL-fuli
0.554 12.23
11.7 0.501
*1.45 0.458
.1.21 0.256
"1.06-0211
*0.974
0. 164
10.570 0.121
AfhneTL-comss
+2.30
0.565
*1.73 0.420
+:
0.243 11.20
"L.N4 0217
0.161
Fire-lume
2..33:10051
*1.62 1: 0.347
*1.35 0.340
11.12:1 0.165
"0.999 0.12
*0.848
0.0344
10,700 2 0.0577
MAML
2.56 2: 1.29
1.90 1: 0.507
1.67 4: 0.313
1.63
0.282
1.28 0.22
1.20 0.199
1.06 0.111
17-SP
+2.33 01.00
*1.65 + 0.378
*1.35 + 0.340
*1.12
1096 0.114
*0,58 0.0818
*0.000
= 0.0335
PAC-Net
2.24 2.24.1.0.00
*1.61 1. 0.394
+1,43 1. 0.389
0.177
-1.18 0.100
1.13 0.0726
1,100 1 0.0599
26

---

C.2
Evaluation of Scientific Papers
C.2.1
Data
We used SciRepEval benchmark dataset for scientific documents, proposed in 23 This dataset com- prises abstracts, review scores, and decision statuses of papers submitted to various machine learning conferences. We conducted two primary tasks: predicting the average review score (a regression task) and determining the acceptance or rejection status of each paper (a binary classification task). The original input was represented for source features fe. we employed text embeddings from the abstracts, which were generated by various pre-trained language models, including BERT (28). SciBERT [29]. T5 [30], and GPT-3 [3]]. When building the vocabulary for the bag-of-wordd strictly higher than 0.9 strictly lower than 0.01. Additionally,
two-gram bag-of- -oodd vector derived from the abstract. As
we ignore phrases with document frequencies eliminated certain stop-words using the default settings in scikit-learn 43 The sentences containing URLs were removed from the abstracts because accepted papers tend include GitHub links in their abstracts after acceptance, which may cause leakage of information to the prediction. The models were trained on a dataset comprising 8, 1666 instances, and their performance were subsequently evaluated on test dataset of 2,043 instances.
C.2.2 Model Definition and Hyperparamter Search
For both the affine model transfer and feature extraction, employed neural networks with ReLU activation and dropout layer with 0.1 dropout rate. The parameters were optimized using Adagrad 41 with 0.01 learning rate.
Affine Model Transfer For functions 91 and 92 in the affine model transfer. we used neural network composed of layers with widths 128, 64, 32, and 16, wherein the source features fs were used as inputs. The number of layers of each width was determined based on Bayesian optimization. Sigmoid activation was employed to the output of (22 in order to facilitate the interpretation of 9a- For Ga. we employed a linear model with the input œ. To prevent overfitting and promote model simplicity, applied ε regularization to the parameters of 9s with regularization parameter of 0.01. The final output of the model was computed as 91 + 92 93. In the case the binary classification task, we applied sigmoid activation function to this final output.
Feature Extraction As in the affine model transfer, we used a neural network composed of layers with widths 128, 64, 32, and 16, wherein the source features fa were used as inputs. The number of layers of each width was determined based on Bayesian optimization. In the case of the binary classification task, we applied sigmoid activation function to the final output.
D Proofs
D.1 Proof of Theorem 2.4
Proof. According Assumption 2.3 holds that for any pe (r(lax))
( of (w)naaoooa)) = up (v) (vlxdy.d..
(S.7)
(i) Let 6y be the Dirac delta function supported on yo- Substituting Pr(y/a)) 5yn into Eq. (S.7). we have
v. (20) = yo (Vyo E )).
Under Assumption
2.3 this implies the property (i).
(ii) For simplicity, we assume the inputs æ are fixed and ()(1)))) Applying the property (i) to Eq. (S.7) yields
f (ww.wlll or. (ʃ vp.coladdy)
27

---

We consider a two-component mixture pt (v(())) (1 - ()(() + ch(y)æ) with a mixing rate e [0, 1], where q and h denote arbitrary probability density functions. Then, we have
/ 00, (1) + ch(ylz)) = Φs. (/ ([(l--))g(uaa) ch(sk=))dy)
Taking the derivative at c 0, we have
f or. (1) ((vl=)) = o',, (ʃ (alule)aa)) (f
(fo( ((b(vll)) ((vl))
which yields
/ (1(ulz))
-cul)) - 86, ( = =xl 0.
(S.8)
For Eq. (S.8) to hold for any and h, Φ5. (v) - 4'/, (E V = æ=)y must be independent of
Thus, the function Φ,, and its inverse v) 6, limited to affine transformations with respect to
y. Since & depends on y and fa(),, takes the form ø(y, fa())) = (f)(())) (2(((x))) O
D.2 Proof of Theorem 4.1
To bound the generalization error, we use the empirical and population Rademacher complexity RFF and RFF) of hypothesis class F, defined as:
Rs(F) E, sup JEF n
1
Σ az.()))
R(F) (sF)
where {a,) a set of Rademacher variables that are independently distributed and each take one of the values in f-1, +1} with equal probability, and S denotes a set of samples. The following proof is based on the one of Theorem 11 shown in II
Proof of Theorem 4.1. For any hypothesis class F with feature map + where || < 1, the following inequality holds:
n Σ
E2 sup
((.:)
(x:))
S
||0|!<<>
1=1
28

---

The proof is given, for example, in Theorem 6.12 of [44]. Thus, the empirical Rademacher complexity of H. is bounded as
Rs(H)) = Eg
sup
Σ
oi
(a,
(ffam())) + +3, (z)
|alli A= Rg
n
1=1
||| hg >g R..
lg ^ R.
E
sup
Σ i=1
(a, 4 (fa(w))))
||a | Saa R
sup l S Ap 1 Ra. 1rlig S 2 R
7, & ((a,)))s00
n
sup
σ: (a,, Φ (fa(x:)))Hz
| Smn Rt 1=1
sup
o; (3 @ 7, (2(f((a))) @ ((.:))
1 2A 2 R2 1=1
Rss A
R2
+
V
(S.9)
Aadym
Rs n
L
+ VV
Agdyy f
Aa
The first inequality follows from the subadditivity of supremum. The last inequality follows from the fact that R. S Pxl(y, (0, 21)) + Aa ||112 < L.
L and applying Talagrand's 's lemma Let C = V + A
44
and Jensen's 's inequality, we obtain
Rs S Cuey
ERR
R(C) (L) S (EE
CueBE
n
n
To apply Corollary 3.5 of 20 we should solve the equation
r= ==Cpe
(S.10)
and obtain r" =² C" > 0 that satisfies
Thus, for any n > O, with probability at least 1 - - there exists constant
Pally. S
EE, C
RRC²
n +++)
R, =
+ !
(S.11)
n 7
Note that, for the last inequality, because R, S P.l(u, (a,, +1)) + >alla for any a, taking the expectation of both sides yields ER, I Pl(y, (a, 41)) >a ballax12 and this gives ER, s infa (PE(y, (a, (1)) + {a]lax] = Rg, Consequently, applying Theorem of [45), we have
Pf(y, M(x)) < Pal(y, M(x)) ō ((V
R.
MºC vn) ( (VIC+
C² Ln n
n
n
(S.12)
Here, use RH) S CV Be S SOVE.
Remark D.I. As in of Eq. (S.9) becomes
without the estimation of the parameters ar and B,, the right-hand side
Ci + 02 / R,,) with some constant ei and e2 0, and Eq. (S.10)
Vm
becomes
n (a C2Vr).
r
29

---

This yields the solution
2
02
C2
2
+
2
2
n
n
Vn
where we use the inequality v + V + y S v4r + 2y.
Thus, Eq. S..1
becomes
Pae(y, h) s C'
R. c
ci
_
n
vn
Consequently, have the following result:
Pł(y, h(x=)) S Pally, h(x))
+ δ R
Vai 3/4
C2 vT)
(c VLn)
(a + c2VL)2 Lm
+
n
n
This means that even if Rg = Ön ((n-)),, the resulting rate only improves to (3/4).
D.3 Proof of Theorem 4.4
Recall that loss function ((:-) is assumed to be pe-Lipschitz for the first argument. In addition, impose the following assumption.
Assumption D.2. There exists a constant B 2 such that for every h H. P(h h²) S
BP(((y, - E(y, h))).
Because we consider E(y.3') = (y- - y²)² in Theorem 4.4 Assumption D.2 holds as stated in 220
First, we show the following corollary, which is slight modification of Theorem 5.4 of (20).
Corollary D.3. Let h be any element of satisfying Py i)) infnem Pnl(y, h), and let (mm)
be any element of (m)) satisfying P,,((y, h(m)) infneHt)) P.l(y. h). Define
((r) Rs H. max
P. (h (m)) < car)
C2n
mE(1,2)
n
where C1, 02 and c3 are constants depending only on B and pe. Then, for any nj > O, with probability at least 5e ""
705..
B p* + (1l + +2BB)
Płly, h)) -P(y,
where f is the fixed point of &b
Proof. Define the function sb as
Ci Rihh E H:: 1 max (m) - (m))) < r) (C2 - c1)n
(r)
2
Because H. ())) and (2)2 are all convex and thus star-shaped around each of its points, Lemma 3.4 of (0 implies that &i is sub-root. Also, define the sub-root function eim
4m (r) = ci (m)
(m)) E H(m). p2 (m))) < r3 + (c2 c1)n
2
Let r be the fixed point of vmm (Tm). Now, for Tm 2 e'm (rm)). Corollary 5.3 20 and the condition on the loss function imply that, with probability at least
11 P(h (2)))) < Bpe hmm)) Ely. (m)))))) = 705 (11p + B7B)
Denote the high side by Sms and define = max fm and s max Sm. Because 2 8m 2 I' r we obtain 2 ui (s) according Lemma 3.2 of 20). and thus,
> 10y20R(h! H(m) Ph - (m)))) )2 s 8s + 11
n
30

---

Therefore, applying Corollary 2.2 of (20) to the class me it follows that with probability at least
1 1_ (1(m)) E H((m) p. - n.(m))y3 - 8) C (1(m))
E H(m)
Pr - 11 - 20}.
This implies that with probability least 1 - 2e
(11p + 27B)
Pa (m(him) - ) s
705r +
n
(1lpe + 27BB
S
705 +
r
n
where the second inequality follows from 2 (b(r) 2 ezn. Define (005- (11u+t78B88) d.
"
According to the triangle inequality in L2(Px), ), it holds that
(n( - = (v (h(mm)) h(m)))) + -Pr(hhlm)--htnm h(m)))) 2
1 V (h(mm)) - h2mm))2 h160))2 + vcr)
Again, applying Corollary 2.2 of (20 (m as before, but now for r 2 t/ (r), it follows that
with probability at least 4e
(h E 1 max P()(m) - M())))) sr}
2
n (m) H(m) : P() - -homma3 sr}
=
m=1
c n (m) E H())) : pp - 1m < 2r
m=1
(h
(m) H(m) Pr (h((mm (m)) - h((m))2 S (V2r + Vdr)
m=1
(h H / max P., (h((m) (2)) S carl.
where c = (V2 + 2 Combining this with Lemma A.4 of 20 leads to the following inequality:
with probability at least - 5e
(c2 01
(r) - Rith H 12 max P(h(m) (22) rr} +
n
S Rg E H u max P()(m)) - 11(m)))3 S sr} + C27
n
I <q E H. 2 max Pa (h - -2mm)2 s car) + C27
= (r))
Letting r = r* and using Lemma 4.3 of 20 we obtain r" St" thus proving the statement.
Under Assumption 4.2,
obtain the following excess risk bound for the proposed model class using Corollary D.3 The proof is based on 20
Theorem D.4. Let le be any element of H satisfying Pal(y, M(x) = inl P., Ely, h(x)). Suppose that Assumption 4.2 satisfied, then there exists constant depending only on pe such that for any 7 > O, with probability at least 1 - 50,
P(y - i(x))) P(y - 1²))))
Sc c
min OSH
16 + K2
+
Σ
(2)
+ 1
n
j=m4+1
1=
31

---

Theorem D.4 is multiple-kernel version of Corollary 6.7 of [20], and a data-dependent version of Theorem 2 of [46] which considers the eigenvalues of the Hilbert-Schmidt operators on H. and Hm(m),
Theorem D.4 concerns the eigenvalues of the Gram matrices ((m) computed from the data.
Proof Theorem D.4 Define R maxm SUPREH(()) Px(y-h(21) h For any h H(m), we obtain
(x) < (,),
S
sup NgH(m)
Pa(y-h(x)) < 4R.
From the symmetry of the oj and the fact that H(m) is convex and symmetric, we obtain the following:
Rs H max Pa(h - mmm 2 4R}
n
2
Egg
sup
Σ qi
Σ m=1
(m))
(z:)
A (m)) cH(m) Pa (m h ( SAR
π
i=1
n
2
Egg
sup (mm) cx((m) Pa (m) h Pan
(h (x;)) - (mm) (x:))
m=1
2 (1(m)(zz) - gom)
S. Eg
sup hlm) gf ( (a( (h -g (m) SAR
m=
2
2Eg
sup
h (m) (x:)
(m)) (m) [=1 P.h (m))2 SR
m=1
2
< 2
E.g
sup
Σ 1=1
o (m)) (xx,)
(m) (m( n1
m=
P. (mm)2R 2R
2 Σ
min ( R, {(m))
<
72 j=l
m=
16
2
a
min { R, {/m)
The second inequality comes from the subadditivity of supremum and the third inequality follows
from Theorem 6.6 of (20). To obtain the last inequality, we use vi + Vy I /2(x + y). Thus, we have
2c, Rg E H. max (m(m())) - -m(m))² < 4R] + (c2 + 2)m
n
16
min { RR, X(m)) } }
(oz + 2)m
< 4cg
j=1
for some constants Ci and 0z. To apply Corollary D.3
we should solve the following inequality for
16
2
min fr. 2 (m)
r v 4c
m=1
j=1
32

---

For any integers Km E [0, rl]. the right-hand side is bounded as
16 2 Σ
min (r, 2 (mm) }
16
(m)
4c1
S 4c1
m= j=1
m=l
/=1
J M
+1
256
Σ Kmm
2
256
2
i (m)
r
n
m=1
m.m
j Nor +1
and we obtain the solution r*
128
128c
2560
(m)
Kim
+
Kym
m=1
m=1
m= j= +1
256c
256c
(m)
m=1
j=Km +1
Optimizing the right-hand side with respect to X1 and Ki2, we obtain the solution as
256c
256c
1
(m))
min On
Km +
n
n
Furthermore, according to Corollary D.3 there exists constant such that with probability at least
5e "
Ply Mx)))² Ply 10" (x))²
1
71
(10))
11
S min Og
Km +
n2
71
j=
With Theorem D.4 and Assumption 4.3 we prove Theorem D.4 as follows.
Proof of Theorem 4.4. Using the inequality V + S / x + for 2 0. 2 0, we have
Pσy - h(z))² pΦy 2x))²
Ki K2
Σ
Σ
X
Σ
72
113)
min K5
+
+
+
71
71
71
j=
j=N2+!
Ki + K2
n2
72 Σ
5O
min 05k11825m
j
n
n
72
n
j=Ki+1
j=Nz+1
0o
min OSK
Ki + K2
Σ
n2
+
1
+
n
n
n1
71
j=Ri+1
j=Kg
Because it holds that
00
m
j
Nm
nm dr <
8m
j=
8m
33

---

for m = 1,2, we should solve the following minimization problem:
min ME
K1 + K2 +
1
11
1
21
+
1
81
K2
.1
= p(K).
n1
1
Taking the derivative, we have
0g(k)
Ki
+
1i
0R1
n
n
-11-81
81
n2
Setting this to zero, find the optimal
41
81 4
1441
K1
81
11
Similarly, we have
22
82
4
K2
82 11
and
P(y - h(x)))² - PPy - (x)))²
x 00 ( 81
82
1t9g
1 - $1
82 12
+ 2 1--1
81
1
1-e2
92
14
1+
+
1+22
+
1 82
=0
TTनl + n
=0 11
mmm 11:
34


Reviews:
[{"review_id": "SzTrBCT6I0", "reviewer": "Reviewer_DK8u", "summary": "The authors focus on the hypothesis transfer learning (HTL) problem. They show that under some hypothesis the data and model transformation is given by an affine transformation. They consider an associated kernel regression problem to identify the transformation and then characterise the generalisation bound which reflects the importance of the target-source relation. Finally, the method is benchmarked in 2 settings.", "questions": "* In the formulation of sec.2.1, it seems that $f_t(x)$ and $f_s(x)$ have the same dimension of $y$, i.e. they both belong to $\\mathbb R$. However, in the examples below $y$ has the same dimension as the scalar product $\\langle \\theta, f_s(x) \\rangle$. Can you clarify?", "limitations": "I find assumption 2.2 a limitation. The authors seemed not concerned by this and pointed out that this is standard in the field. However, the intuition is that if the target representation is not included (or equal) to the source representation we shouldn't be able to invert the function.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed method allows for the theoretical characterisation of generalisation and excess risk bounds.\n* The method is versatile and can be used in very different settings.", "weaknesses": "* The model is not clear (sec.2.1). Maybe adding a scheme or introducing the linear models as a first example before moving to the general version would help. Given this first setting, the rest flows nicely.\n* I find assumption 2.2 a strong assumption. The authors comment saying that this is common in HTLs. However, it implies no information compression which is hard to believe in many practical contexts. \n* In tables 1 and 2, the improvements are only marginally better than the other methods even if significant according to the authors' test.\n\nMinor comments: \n* assumption 2.2 (Invetribility) is instead (Invertibility)\n* fig.1 is misplaced. \n* proposition A.1 refer to the proof of Th.2.4 in the main paper, but this is instead in the SM sec.D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for providing insightful comments. Below is our point-by-point response to each of the comments and questions.\n\n> **[W1]** The model is not clear (sec.2.1). Maybe adding a scheme or introducing the linear models as a first example before moving to the general version would help. Given this first setting, the rest flows nicely.\n\nTo make our setups easier to understand, we will add the following sentences to the beginning of Section 2.1 in the revised version:\n\nIn this paper, we focus on transfer learning with variable transformations as proposed in Du et al. (2017). For an illustration of the concept, consider the case where there exists a relationship between the true functions $f_s^* (x) \\in \\mathbb{R}$ and $f_t^* (x) \\in \\mathbb{R}$ such that $f_t^* (x)=f_s^* (x)+x^\\top \\theta^*, x \\in \\mathbb{R}^d$ with an unknown parameter $\\theta^* \\in \\mathbb{R}^d$. If $f_t^*$ is non-smooth, a large number of training samples is needed to learn $f_t^*$ directly. However, since the difference $f_t^*-f_s^*$ is a linear function with respect to the unknown $\\theta^*$, it can be learned with fewer samples if prior information about $f_s^*$ is available. For example, a target model can be obtained by adding $f_s$ to the model $g$ trained for the intermediate variable $z=y-f_s (x)$.\n\n> **[W2]** I find assumption 2.2 a strong assumption. The authors comment saying that this is common in HTLs. However, it implies no information compression which is hard to believe in many practical contexts.\n\nThere may be a misunderstanding about Assumption 2.2. It assumes the invertibility of the function $\\psi_{f_s} ∶ \\mathbb{R} \\to \\mathbb{R}$ for a fixed $f_s (x)$. In other words, it assumes a one-to-one correspondence between the predictive value $\\hat{f}_t (x)$ and the output of the intermediate model $\\hat{g}(x)$, rather than between $x$ and $f_t$, or the source representation and target representation. If this assumption does not hold, then multiple values of $\\hat{g}$ correspond to the same predicted value $\\hat{f}_t$. Such modeling is unnatural. For example, if we set $\\psi = g(x)^2+f_s (x)$, Assumption 2.2 does not hold. In this case, we get the same predicted value even if the sign of $\\hat{g}(x)$ is reversed.  \nIn the revised manuscript, we will add the above explanation to clarify that Assumption 2.2 is never restrictive.\n\n\n> **[Q1]** In the formulation of sec.2.1, it seems that $f_t (x)$ and $f_s (x)$ have the same dimension of $y$, i.e. they both belong to $\\mathbb{R}$. However, in the examples below $y$ has the same dimension as the scalar product $\\langle \\theta, f_s (x) \\rangle$. Can you clarify?\n\nIn our setup, $f_t (x)$ is a scalar, while $f_s (x)$ is a vector. Since the current notation is misleading, we will revise the manuscript to represent vectors in bold.\n\n**[Minor comments]**  \nThank you very much for pointing out our typos. We will fix the errors.\n\n[References]  \nS. S. Du, et al. “Hypothesis transfer learning via transformation functions,” NeurIPS, 2017."}, {"review_id": "CJZngWVXH9", "reviewer": "Reviewer_tCYn", "summary": "This paper focuses on the theoretical perspective on the affine model transfer learning. Specifically, this paper presents the optimal class of models on transfer learning with l2 loss takes the form of an affine decoupling. The decoupling consists of three functions, \\$g_1 + g_2g_3\\$, where \\$g_1, g_2\\$ encodes the domain gaps between source and target domains, and \\$g_3\\$ encodes the domain-specific information. Further, for each of the functions, this paper proposes a kernel method to approximate. Additionally, the generalization and risk bounds are presented. \n\n**___________________________________________________ Post Rebuttal __________________________________________________**\n\n\nThe reviewer has read the rebuttal. \n\nIn the rebuttal, parts of the theorems are restricted to \\$ \\mathbb{R} \\$. This may degrade the contribution of the paper since we are handling the high-dimension data for most of the cases.\n\n The reviewer will maintain the score of 7 since there appear to be no technical flaws to the reviewer, and the reviewer gained some insights from the paper. However, the reviewer won't champion the paper, since I am not familiar to the literature. ", "questions": "1. All the discussions are limited to the \\$\\mathbb{R}\\$. It remains elusive whether the analysis can be extended to higher dimensional spaces naturally. \n2. What is d in table 1, is it the source domain feature dimension?", "limitations": "While the paper's analysis is primarily limited to the regression task, it provides valuable insights with a commendable depth of exploration, despite limiting its general applicability. The complexity of a broader analysis would undoubtedly be a challenging undertaking, but the paper sets a strong foundation for future exploration.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents universal results to the transfer learning on linear regression, which looks interesting to the reviewer. Specifically, it is shown that the optimal solution under l2 loss falls into a certain class of models. And the analysis is not limited to machine learning models.\n2. This paper also presents the generalization and risk bounds, making the analysis more solid. \n2. Empirically, this paper experiments on two datasets with extensive ablation and baselines, making the paper complete and thorough.", "weaknesses": "1. Since the reviewer is not from the area, it is hard for the reviewer to judge the novelty and significance of the work. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to receive a positive evaluation from the reviewer. Below is our point-by-point response to each question.\n\n> **[Q1]** All the discussions are limited to the $\\mathbb{R}$. It remains elusive whether the analysis can be extended to higher dimensional spaces naturally.\n\nWhen $y$ and $f_t$ are multi-dimensional, the claims in Theorem 2.4 would hold under some modifications. However, it is not obvious whether the theoretical analyses of Section 4 can be straightforwardly extended to the multi-dimensional scenario.\n\n> **[Q2]** What is d in table 1, is it the source domain feature dimension?\n\n$d$ is the dimension of the original input $x$. We will add the definition of $d$ in the caption of Table 1."}, {"review_id": "9Tgf3qBSjw", "reviewer": "Reviewer_XSgT", "summary": "This paper presents a simple yet effective approach to transfer learning in regression problems by introducing an affine transform model. The proposed model incorporates the source feature function and the product of the source feature function and the function associated with the target input augmentation. Through the utilization of a least square loss function and a transformation function, this paper establishes the optimality of the affine model as the architecture of choice for addressing transfer learning challenges. Furthermore, the paper extends upon previous research by deriving generalization bounds and excess risk bounds in the proposed framework.", "questions": "1. Would it not be an overstatement to claim that the proposed algorithm encompasses all existing algorithms? This assertion arises from the fact that weight-based algorithms (such as L2-SP and PAC-NET) are not incorporated into the proposed algorithm.\n\n2. Could we possibly expand the experimentation to include a broader range of experimental datasets?", "limitations": "This reviewer thinks that the algorithm proposed in this paper has limitations that can only be applied to regression problems.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is impeccably written and exceptionally comprehensible.\n\n2. The proposed method is rigorously derived and its performance is systematically evaluated from a theoretical standpoint.\n\n3. The proposed algorithm can incorporate and generalize previously proposed algorithms while showing better performance.", "weaknesses": "1. The existence of inverse functions appears to be of significance. However, it can be considered a potential limitation as it may not always be feasible to obtain inverse functions in practice.\n\n2. I would suggest that the authors conduct evaluations of the proposed method using widely recognized benchmark datasets. It is particularly crucial to assess the algorithm's performance based on the proximity or disparity between the source and target, in order to ascertain its effectiveness under varying conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in writing in-depth feedback. Please find our response to your concerns below.\n\n> **[W1]** The existence of inverse functions appears to be of significance. However, it can be considered a potential limitation as it may not always be feasible to obtain inverse functions in practice.\n\nThere may be a misunderstanding about Assumption 2.2. It assumes the invertibility of the function $\\psi_{f_s} ∶ \\mathbb{R} \\to \\mathbb{R}$ for a fixed $f_s (x)$. In other words, it assumes a one-to-one correspondence between the predictive value $\\hat{f}_t (x)$ and the output of the intermediate model $\\hat{g}(x)$, rather than between $x$ and $f_t$, or the source representation and target representation. If this assumption does not hold, then multiple values of $\\hat{g}$ correspond to the same predicted value $\\hat{f}_t$. Such modeling is unnatural. For example, if we set $\\psi = g(x)^2+f_s (x)$, Assumption 2.2 does not hold. In this case, we get the same predicted value even if the sign of $\\hat{g}(x)$ is reversed.  \nIn the revised manuscript, we will add the above explanation to clarify that Assumption 2.2 is never restrictive.\n\n> **[W2, Q2]** I would suggest that the authors conduct evaluations of the proposed method using widely recognized benchmark datasets. It is particularly crucial to assess the algorithm's performance based on the proximity or disparity between the source and target, in order to ascertain its effectiveness under varying conditions.\n\nTransfer learning for regression has been less studied than for classification tasks. Hence, commonly used benchmark datasets have not been established. Therefore, we conducted two real-world experiments with our original datasets in the field of physical chemistry in Supplementary Material. We aim to provide these two datasets to the machine learning community as new public datasets for transfer learning regression. It should also be noted that in one of the experiments, we investigated the effect of the proximity between the source features and target domain on the learning behavior of the affine model transfer.  \nDue to the time constraints of this rebuttal, additional experiments can not be conducted. We will take your suggestion into consideration for future work.\n\n> **[Q1]** Would it not be an overstatement to claim that the proposed algorithm encompasses all existing algorithms? This assertion arises from the fact that weight-based algorithms (such as L2-SP and PAC-NET) are not incorporated into the proposed algorithm.\n\nAs you pointed out, our method does not encompass weight-based transfer learning such as fine-tuning. Several misleading phrases, such as \"including neural network-based TL\" in line 53, will be corrected (e.g., \"including the neural feature extraction\")."}, {"review_id": "CB2oFtN4T3", "reviewer": "Reviewer_3FfK", "summary": "The authors consider the transfer learning setting by affine models adapting source features to the target domain by separately estimating cross-domain shift and domain-specific factors. They motivate the choice of such models deriving the affine form as an optimal class based on the squared loss for the target task. The authors show that the considered affine model encompasses several existing transfer learning methods, including neural network based transfer learning. The authors provide an estimation algorithm when modeled by kernel methods. The authors provide a generalization bound and an excess risk bound for their method. The bound show that when the source feature is strongly related to the target domain, their transfer learning approach results to be advantageous in comparison to the vanilla learning approach. Such an advantage is confirmed also by numerical experiments, in which the proposed method outperforms widely applied transfer learning methods lacking of theoretical foundation, such as re-using features from pre-trained models. More specifically, they consider two experimental settings: (i) the prediction of feed-forward torque at seven joints of the robot arm, and (ii) the prediction of review scores and decisions of scientific papers. Additionally, two case studies in materials science are presented in the appendix.", "questions": "The algorithm the authors propose applies a coordinate descent minimization procedure on the objective function F. Is F jointly convex? Is the algorithm globally convergent? Usually, the study of global convergence for coordinate descent algorithms is problematic.\n\nThe authors require the Lipschitzness of the loss function, but the squared loss is Lipschitz if and only if the labels are bounded and this is quite restrictive.\n\nFrom Thm. 4.1 it seems the authors get an improvement w.r.t. n by transfer learning. This seems to be strange, since, usually, the advantage observed in previous work by transfer learning is by constant factors. Could you please comment about this with reference to the papers 'Stability and hypothesis transfer learning' and 'Fast rates by transferring from auxiliary hypotheses'?", "limitations": "I do not see any potential negative societal impact related to this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors consider an interesting problem in a rigorous and formal way.\n\nThe authors provide both theoretical and experimental validation for their method. \n\nThe derivation/motivation of the affine form as an optimal class based on the squared loss for the target task is interesting.\n\nThe affine model transfer is model-agnostic and it can be easily combined with pre-existent models. In addition, since domain specific factors are involved in incorporating the source features, the affine transfer can also deal with domain common and unique factors simultaneously and separately.\n\nOverall, the paper is written in a clear way.", "weaknesses": "Considering only the squared loss is quite limiting. The authors comment about this, but it would be nice to understand if there is possibility to extend their framework to more general loss functions. For instance, what about the optimality of the class of affine functions with different loss functions? The authors say they can also derive the same optimal function by minimizing the upper bound of the estimation error in the TL procedure, as discussed in the appendix. I guess this could be related to my question.\n\nSome technical aspects could be better explained and intuitively motivated in my opinion, see for instance my comment below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the valuable comments. Below we respond to each concern / question raised in your review.\n\n> **[W1]** Considering only the squared loss is quite limiting. The authors comment about this, but it would be nice to understand if there is possibility to extend their framework to more general loss functions. For instance, what about the optimality of the class of affine functions with different loss functions? The authors say they can also derive the same optimal function by minimizing the upper bound of the estimation error in the TL procedure, as discussed in the appendix. I guess this could be related to my question.\n\nIn Section A.1, we discussed the case of general convex loss functions; it was found that the optimality of the affine function does not hold for other losses.  \nIn addition, further assumptions would be needed if the current framework is extended to include other losses based on the upper bound on the estimation error, as you suggest.\n\n> **[Q1]** The algorithm the authors propose applies a coordinate descent minimization procedure on the objective function F. Is F jointly convex? Is the algorithm globally convergent? Usually, the study of global convergence for coordinate descent algorithms is problematic.\n\nIn Algorithm 1, $F$ is not jointly convex in general. However, the key consideration for running Algorithm 1 is that when two parameters are fixed, $F$ is convex with respect to the remaining parameter. This condition holds for kernel methods and generalized linear models.  \nAccording to Zhou et al. (2013), when each sub-minimization problem is convex, Algorithm 1 is guaranteed to converge to a stationary point. Furthermore, Zhou et al. (2013) showed the consistency and asymptotic normality to hold for the alternating minimization algorithm.  \nThis discussion will be included in the revised version of the paper. \n\n> **[Q2]** The authors require the Lipschitzness of the loss function, but the squared loss is Lipschitz if and only if the labels are bounded and this is quite restrictive.\n\nIndeed, the assumption that the labels are bounded does not always hold. However, without this assumption, theoretical analysis becomes more challenging. For example, Mohri et al. (2018) states \"The analysis of unbounded regression problems is technically more elaborate and typically requires some other types of assumptions.\" We believe that the labels are often bounded in many real-world problems.\n\n> **[Q3]** From Thm. 4.1 it seems the authors get an improvement w.r.t. n by transfer learning. This seems to be strange, since, usually, the advantage observed in previous work by transfer learning is by constant factors. Could you please comment about this with reference to the papers 'Stability and hypothesis transfer learning' and 'Fast rates by transferring from auxiliary hypotheses'?\n\nTheorem 4.1 in our study corresponds to Theorem 8 in Kuzborskij and Orabona (2017). The structure of the bounds derived in the two studies is identical. Both results indicate that the rate improves with respect to $n$ when the source features have high descriptive power for the target domain.  \nAn example of a bound with a constant factor can be found in Theorem 3 of Ben-David et al. (2010), where a constant factor appears as the $\\mathcal{H}$-divergence. In that study, the authors considered a loss function of $(1-\\alpha) \\cdot (\\text{error on the target task})+\\alpha \\cdot (\\text{error on the source task})$. Note that our setting corresponds to $\\alpha=0$. It is then found that the constant factor vanishes in the bound derived in Ben-David et al. (2010).\n\n[Reference]  \nH. Zhou, et al. \"Tensor regression with applications in neuroimaging data analysis,” Journal of the American Statistical Association, 2013.  \nM. Mohri, et al. Foundations of machine learning. MIT press, 2018.  \nI. Kuzborskij and F. Orabona, “Fast rates by transferring from auxiliary hypotheses,” Machine Learning, 2017.  \nS. Ben-David, et al. \"A theory of learning from different domains.\" Machine Learning, 2010.  "}, {"review_id": "2sYNuRecJB", "reviewer": "Reviewer_t1vu", "summary": "The paper provides a comprehensive investigation of transfer learning and proposes an affine model transfer framework. The authors summarize various transfer learning approaches under the class of affine model transformation and establish theoretical properties such as generalization error and excess risk.\nIn the experimental evaluation, the proposed methods demonstrate comparable performance to other baseline methods on tasks such as robot arm trajectory prediction and scientific document score prediction.\n", "questions": "I would like to know that why were the experiments not conducted on commonly adopted transfer learning datasets, such as MIT Indoors, Caltech 256, and CelebA? \n\nConsidering that the proposed method performs worse than Fine-tuning in the case of Torque 7 prediction according to table 1, it is important to clarify the advantages of the proposed method over Fine-tuning. What unique benefits or improvements does the proposed method offer in comparison?\n\nAre there any suggestions provided regarding the design of $g_1, g_2$, and  $g_3$ functions, or any specific neural network architectures? Providing guidance on the selection and design of these components would greatly assist readers in implementing the proposed method effectively.\n", "limitations": "See [Weaknesses] for limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies an important topic: the theoretical properties of transfer learning.\n\nThe proposed theoretical framework for affine model transfer and the derived generalization bound provide valuable insights into transfer learning, domain generalization, and model adaptation.\n\nOverall, the paper is well-organized and effectively conveys the intricacies of the proposed theoretical framework.\n", "weaknesses": "My main concerns revolve around the experimental evaluation conducted in the paper. The evaluation is narrower in scope compared to most transfer learning studies. It would be beneficial to validate the proposed method on standard transfer learning benchmarks, such as the MIT Indoors dataset, Caltech 256 dataset, or other commonly used regression datasets. This kind of evaluation can be found in references [1,2].\nFurthermore, in the experimental evaluation, the observed improvement is limited. Table 1, specifically in the Torque 7 prediction, shows that the proposed method performs worse than Fine-tuning in most cases.\n\nWhile Theorem 2.4 provides a nice property of transformation functions, the paper lacks a discussion on the specific properties of $g_1, g_2$, and $g_3$ functions. It would be helpful to provide suggestions or insights into the types of $g_1, g_2$, and  $g_3$ functions.\n\n[1] L. Xuhong, et al, “Explicit inductive bias for transfer learning with convolutional networks,” ICML, 2018.    \n[2] S. Myung, et al, “PAC-Net: A model pruning approach to inductive transfer learning,” ICML, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for providing us with valuable feedback. Below is our point-by-point response to each question.\n\n> **[Q1]** I would like to know that why were the experiments not conducted on commonly adopted transfer learning datasets, such as MIT Indoors, Caltech 256, and CelebA?\n\nThe MIT Indoors, Caltech 256, and CelebA datasets are primarily designed for classification and object recognition tasks, not for regression tasks which are the main focus of this study. Transfer learning for regression has been less studied than for classification tasks. Hence, common benchmark datasets have not been established. For example, Li et al. (2018) limited their experiments to object recognition and scene classification. Myung et al. (2022) conducted experiments on regression tasks, but used only simplistic toy examples rather than real-world datasets.  \nTherefore, we conducted two real-world experiments with our original datasets in the field of physical chemistry in Supplementary Materials. We aim to provide these two datasets to the machine learning community as new public datasets for transfer learning regression.\n\n> **[Q2]** Considering that the proposed method performs worse than Fine-tuning in the case of Torque 7 prediction according to table 1, it is important to clarify the advantages of the proposed method over Fine-tuning. What unique benefits or improvements does the proposed method offer in comparison?\n\nThe strength of our method compared to fine-tuning is that it does not degrade its performance in cases where cross-domain relationships are weak. While the experiment in Section 5.1 revealed that fine-tuning outperformed our method in cases of Torque 7, the performance of fine-tuning was significantly degraded as the source-target relationship became weaker, as seen in Torque 1 case. In contrast, our method was able to avoid negative transfer even for such cases. This characteristic is particularly beneficial because, in many cases, the degree of relatedness between the domains is not known in advance.  \nAnother distinctive feature of our method is its theoretical foundation of optimality for the squared loss case. This aspect would encourage further development of transfer learning for regression.  \nIn the revised manuscript, we will provide additional discussion comparing our method to weight-based transfer learning, as replied above.\n\n> **[Q3]** Are there any suggestions provided regarding the design of $g_1,g_2$, and $g_3$ functions, or any specific neural network architectures? Providing guidance on the selection and design of these components would greatly assist readers in implementing the proposed method effectively.\n\nOne of our key principles for the design of $g_1,g_2$, and $g_3$ is interpretability. In our model, $g_1$ and $g_2$ primarily facilitate knowledge transfer, while the estimated $g_3$ is used to gain insight on domain-specific factors. For instance, in order to infer cross-domain differences, we could design $g_1$ and $g_2$ by the conventional neural feature extraction, and a simple, highly interpretable model such as a linear model could be used for $g_3$. Thus, observing the estimated regression coefficients in $g_3$, one can statistically infer which features of x are related to inter-domain differences. This advantage of the proposed method is demonstrated in Section 5.2 and in Section B.3.  \nIn the revised manuscript, we will provide such additional guidance on the design of each component of the model. \n\n[Reference]  \nX. Li, et al. “Explicit inductive bias for transfer learning with convolutional networks,” ICML, 2018.  \nS. Myung, et al. “PAC-Net: A model pruning approach to inductive transfer learning,” ICML, 2022.  "}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the area chair's personal evaluation, and the post-rebuttal discussion. 

This paper provides new theoretical understandings and practical insights into transfer learning in the considered affine model transformation setting. Most reviewers find the results sufficiently novel and convincing. The authors’ rebuttal has successfully addressed the major concerns of reviewers. Therefore, I recommend acceptance of this submission. I also expect the authors to include the new results and discussion during the rebuttal phase to the final version.
Decision: Accept (poster)