SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The advantages of the model were verified theoretically and experimentally in this study. We showed theoretical results on the generalization bound and excess risk bound when the regression tasks are solved by kernel methods. It is shown that if the source feature is strongly related to the target domain, the convergence rate of the generalization bound is improved from naive learning. The excess risk of the proposed TL is evaluated using the eigen-decay of the product kernel, which also illustrates the effect of the overlap between the source and target tasks. In our numerical studies, the affine model transfer generally outperforms in test errors when the target and source tasks have a similarity. We have also seen in the example of NLP that the proposed affine model transfer can identify the (non-)valuable phrases for high-quality papers. This can be done by the affine representation of cross-domain shift and domain-specific factors in our model.
This work was supported by JST SPRING Grant No. JPMJSP2104, JST CREST Grants No. JPMJCR22O3 and No. JPMJCR19I3, MEXT KAKENHI Grant-in-Aid for Scientific Research on Innovative Areas (Grant No. 19H50820), the Grant-in-Aid for Scientific Research (A) (Grant No. 19H01132) and Grant-in-Aid for Research Activity Start-up (Grant No. 23K19980) from the Japan Society for the Promotion of Science (JSPS), and the MEXT Program for Promoting Researches on the Supercomputer Fugaku (No. hp210264).
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, pp. 84-90, 2012.
[2] G. Csurka, "Domain adaptation for visual applications: A comprehensive survey," arXiv, vol. abs/1702.05374, 2017.
[3] S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf, "Transfer learning in natural language processing," Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv, vol. abs/1810.04805, 2019.
[5] R. K. Sevakula, V. Singh, N. K. Verma, C. Kumar, and Y. Cui, "Transfer learning for molecular cancer classification using deep neural networks," IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 16, pp. 2089-2100, 2019.
[6] H. Yamada, C. Liu, S. Wu, Y. Koyama, S. Ju, J. Shiomi, J. Morikawa, and R. Yoshida, "Predicting materials properties with little data using shotgun transfer learning," ACS Central Science, vol. 5, pp. 1717-1730, 2019.
[7] S. Wu, Y. Kondo, M. Kakimoto, B. Yang, H. Yamada, I. Kuwajima, G. Lambard, K. Hongo, Y. Xu, J. Shiomi, et al., "Machine-learning-assisted discovery of polymers with high thermal conductivity using a molecular design algorithm," npj Computational Materials, vol. 5, no. 1, pp. 1-11, 2019.
[8] S. Ju, R. Yoshida, C. Liu, K. Hongo, T. Tadano, and J. Shiomi, "Exploring diamond-like lattice thermal conductivity crystals via feature-based transfer learning," *Physical Review Materials*, vol. 5, no. 5, p. 053801, 2021.
[9] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," Advances in Neural Information Processing Systems, vol. 27, 2014.
[10] I. Kuzborskij and F. Orabona, "Stability and hypothesis transfer learning," International Conference on Machine Learning, pp. 942-950, 2013.
[11] I. Kuzborskij and F. Orabona, "Fast rates by transferring from auxiliary hypotheses," Machine Learning, vol. 106, no. 2, pp. 171-195, 2017.
[12] S. S. Du, J. Koushik, A. Singh, and B. Póczos, "Hypothesis transfer learning via transformation functions," Advances in Neural Information Processing Systems, vol. 30, 2017.
[13] S. Minami, S. Liu, S. Wu, K. Fukumizu, and R. Yoshida, “A general class of transfer learning regression without implementation cost,” Proceedings of AAAI Conference on Artificial Intelligence, vol. 35, pp. 8992–8999, 2021.
[14] S. Liu and K. Fukumizu, "Estimating posterior ratio for classification: Transfer learning from probabilistic perspective," Proceedings of the 2016 SIAM International Conference on Data Mining, pp. 747-755, 2016.
[15] N. Tripuraneni, M. Jordan, and C. Jin, "On the theory of transfer learning: The importance of task diversity," Advances in Neural Information Processing Systems, vol. 33, pp. 7852-7862, 2020.
[36] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference for Learning Representations, 2015.
[37] J. Wang, R. M. Wolf, J. W. Caldwell, P. A. Kollman, and D. A. Case, "Development and testing of a general amber force field," Journal of Computational Chemistry, vol. 25, no. 9, pp. 1157-1174, 2004.
[38] Y. Hayashi, J. Shiomi, J. Morikawa, and R. Yoshida, "Radonpy: automated physical property calculation using all-atom classical molecular dynamics simulations for polymer informatics," npj Computational Materials, vol. 8, no. 222, 2022.
[39] S. Otsuka, I. Kuwajima, J. Hosoya, Y. Xu, and M. Yamazaki, "PoLyInfo: Polymer database for polymeric materials design," 2011 International Conference on Emerging Intelligent Data and Web Technologies, pp. 22-29, 2011.
[40] M. Kusaba, Y. Hayashi, C. Liu, A. Wakiuchi, and R. Yoshida, “Representation of materials by kernel mean embedding,” Physical Review B, vol. 108, p. 134107, 2023.
[41] J. Duchi, E. Hazan, and Y. Singer, "Adaptive subgradient methods for online learning and stochastic optimization," Journal of machine learning research, vol. 12, no. 7, 2011.
[42] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference for Learning Representations, 2015.
[43] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
[44] M. Mohri, A. Rostamizadeh, and A. S. Talwalkar, Foundations of Machine Learning. MIT press, 2018.
[45] N. Srebro, K. Sridharan, and A. Tewari, "Smoothness, low noise and fast rates," Advances in Neural Information Processing Systems, vol. 23, 2010.
[46] M. Kloft and G. Blanchard, "The local Rademacher complexity of  $\ell_p$ -norm multiple kernel learning," Advances in Neural Information Processing Systems, vol. 24, 2011.
A Other Perspectives on Affine Model Transfer
A.1 Transformation Functions for General Loss Functions
Here we discuss the optimal transformation function for general loss functions.
Let  $\ell(y, y') \geq 0$  be a convex loss function that returns zero if and only if  $y = y'$ , and let  $g^*(\boldsymbol{x})$  be the optimal predictor that minimizes the expectation of  $\ell$  with respect to the distribution  $p_t$  followed by  $\boldsymbol{x}$  and  $y$  transformed by  $\phi$ :
$$
g ^ {*} (\boldsymbol {x}) = \arg \min  _ {g} \mathbb {E} _ {p _ {t}} \left[ \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y)) \right].
$$
The function  $g$  that minimizes the expected loss
$$
\mathbb {E} _ {p _ {t}} \left[ \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y)) \right] = \iint \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y)) p _ {t} (\boldsymbol {x}, y) d \boldsymbol {x} d y
$$
should be a solution to the Euler-Lagrange equation
$$
\frac {\partial}{\partial g (\boldsymbol {x})} \int \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y)) p _ {t} (\boldsymbol {x}, y) \mathrm {d} y = \int \frac {\partial}{\partial g (\boldsymbol {x})} \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y)) p _ {t} (y | \boldsymbol {x}) \mathrm {d} y p _ {t} (\boldsymbol {x}) = 0. \tag {S.1}
$$
Denote the solution of Eq. (S.1) by  $G(\pmb{x}; \phi_{f_s})$ . While  $G$  depends on the loss  $\ell$  and distribution  $p_t$ , we omit those from the argument for notational simplicity. Using this function, the minimizer of the expected loss  $\mathbb{E}_{\pmb{x}, y}[\ell(g(\pmb{x}), y)]$  can be expressed as  $G(\pmb{x}; \mathrm{id})$ , where  $\mathrm{id}$  represents the identity function.
Here, we consider the following assumption to hold, which generalizes Assumption 2.3 in the main text:
Assumption 2.3(b). For any distribution on the target domain  $p_t(x, y)$  and all  $x \in \mathcal{X}$ , the following relationship holds:
$$
\psi_ {f _ {s}} (g ^ {*} (\boldsymbol {x})) = \arg \min  _ {g} \mathbb {E} _ {\boldsymbol {x}, y} [ \ell (g (\boldsymbol {x}), y) ].
$$
Equivalently, the transformation functions  $\phi_{f_s}$  and  $\psi_{f_s}$  satisfy
$$
\psi_ {f _ {s}} \left(G (\boldsymbol {x}; \phi_ {f _ {s}})\right) = G (\boldsymbol {x}; \mathrm {i d}). \tag {S.2}
$$
Assumption 2.3(b) states that if the optimal predictor  $G(\pmb{x}; \phi_{f_s})$  for the data transformed by  $\phi$  is given to the model transformation function  $\psi$ , it is consistent with the overall optimal predictor  $G(\pmb{x}; \mathrm{id})$  in the target region in terms of the loss function  $\ell$ . We consider all pairs of  $\psi$  and  $\phi$  that satisfy this consistency condition.
Here, let us consider the following proposition:
Proposition A.1. Under Assumption 2.1, 2.2 and 2.3(b),  $\psi_{f_s}^{-1} = \phi_{f_s}$
Proof. The proof is analogous to that of Theorem 2.4 in Section D.1. For any  $y_0 \in \mathcal{V}$ , let  $p_t(y|x) = \delta_{y_0}$ . Combining this with Eq. (S.1) leads to
$$
\frac {\partial}{\partial g (\boldsymbol {x})} \ell (g (\boldsymbol {x}), \phi_ {f _ {s}} (y _ {0})) = 0 (\forall y _ {0} \in \mathcal {Y}).
$$
Because  $\ell(y, y')$  returns the minimum value zero if and only if  $y = y'$ , we obtain  $G(x; \phi_{f_s}) = \phi_{f_s}(y_0)$ . Similarly, we have  $G(x; \mathrm{id}) = y_0$ . From these two facts and Assumption 2.3(b), we have  $\psi_{f_s}(\phi_{f_s}(y_0)) = y_0$ , proving that the proposition is true.
Proposition A.1 indicates that the first statement of Theorem 2.4 holds for general loss functions. However, the second claim of Theorem 2.4 generally depends on the type of loss function. Through the following examples, we describe the optimal class of transformation functions for several loss functions.
Example 1 (Squared loss). Let  $\ell(y, y') = |y - y'|^2$ . As a solution of Eq. (S.1), we can see that the optimal predictor is the conditional expectation  $\mathbb{E}_{p_t}[\phi_{f_s}(Y) | X = x]$ . As discussed in Section 2.1, the transformation functions  $\phi_{f_s}$  and  $\psi_{f_s}$  should be affine transformations.
Example 2 (Absolute loss). Let  $\ell(y, y') = |y - y'|$ . Substituting this into Eq. (S.1), we have
$$
\begin{array}{l} 0 = \int \frac {\partial}{\partial g (\boldsymbol {x})} | g (\boldsymbol {x}) - \phi_ {f _ {s}} (y) | p _ {t} (y | \boldsymbol {x}) d y \\ = \int \operatorname {s i g n} \left(g (\boldsymbol {x}) - \phi_ {f _ {s}} (y)\right) p _ {t} (y | \boldsymbol {x}) \mathrm {d} y \\ = \int_ {\phi_ {f _ {s}} (y) \geq g (\boldsymbol {x})} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y - \int_ {\phi_ {f _ {s}} (y) <   g (\boldsymbol {x})} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y. \\ \end{array}
$$
Assuming that  $\phi_{f_n}$  is monotonically increasing, we have
$$
0 = \int_ {y \geq \phi_ {f _ {s}} ^ {- 1} (g (\boldsymbol {x}))} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y - \int_ {y <   \phi_ {f _ {s}} ^ {- 1} (g (\boldsymbol {x}))} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y.
$$
$$
\int_ {\phi_ {f _ {s}} ^ {- 1} (g (\boldsymbol {x}))} ^ {\infty} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y = \int_ {- \infty} ^ {\phi_ {f _ {s}} ^ {- 1} (g (\boldsymbol {x}))} p _ {t} (y | \boldsymbol {x}) \mathrm {d} y.
$$
The same result is obtained even if  $\phi_{f_s}$  is monotonically decreasing. Consequently,
$$
\phi_ {f _ {s}} ^ {- 1} (g (\boldsymbol {x})) = \operatorname {M e d i a n} [ Y | \boldsymbol {X} = \boldsymbol {x} ],
$$
$$
G (\boldsymbol {x}; \phi_ {f _ {s}}) = \phi_ {f _ {s}} \left(\operatorname {M e d i a n} [ Y | \boldsymbol {X} = \boldsymbol {x} ]\right).
$$
This implies that Eq. (S.2) holds for any  $\phi_{f_s}$  including an affine transformation, and the function form cannot be identified. from this analysis.
Example 3 (Exponential-squared loss). As an example where the affine transformation is not optimal, consider the loss function  $\ell(y, y') = |e^y - e^{y'}|^2$ . Substituting this into Eq. (S.1), we have
$$
\begin{array}{l} 0 = \int \frac {\partial}{\partial g (\boldsymbol {x})} \left| \exp (g (\boldsymbol {x})) - \exp \left(\phi_ {f _ {s}} (y)\right) \right| ^ {2} p _ {t} (y | \boldsymbol {x}) d y \\ = 2 \exp (g (\boldsymbol {x})) \int \left(\exp (g (\boldsymbol {x})) - \exp \left(\phi_ {f _ {s}} (y)\right)\right) p _ {t} (y | \boldsymbol {x}) \mathrm {d} y. \\ \end{array}
$$
$$
G (\boldsymbol {x}; \phi_ {f _ {s}}) = \log \mathbb {E} \left[ \exp \left(\phi_ {f _ {s}} (Y)\right) | \boldsymbol {X} = \boldsymbol {x} \right].
$$
Consequently, Eq. (S.2) becomes
$$
\log \mathbb {E} \left[ \exp \left(\phi_ {f _ {s}} (Y)\right) \right] = \phi_ {f _ {s}} \left(\log \mathbb {E} \left[ \exp (Y) \right]\right).
$$
Even if  $\phi_{f_s}$  is an affine transformation, this equation does not generally hold.
A.2 Analysis of the Optimal Function Class Based on the Upper Bound of the Estimation Error
Here, we discuss the optimal class for the transformation function based on the upper bound of the estimation error.
In addition to Assumptions 2.1 and 2.2, we assume the following in place of Assumption 2.3:
Assumption A.2. The transformation functions  $\phi$  and  $\psi$  are Lipschitz continuous with respect to the first argument, i.e., there exist constants  $\mu_{\phi}$  and  $\mu_{\psi}$  such that,
$$
\phi (a, c) - \phi \left(a ^ {\prime}, c\right) \leq \mu_ {\phi} \| a - a ^ {\prime} \| _ {2}, \quad \psi (b, c) - \psi \left(b ^ {\prime}, c\right) \leq \mu_ {\psi} \| b - b ^ {\prime} \| _ {2},
$$
for any  $a, a' \in \mathcal{V}$  and  $b, b' \in \mathbb{R}$  with any given  $c \in \mathcal{F}_s$ .
Note that each Lipschitz constant is a function of the second argument  $\pmb{f}_{s}$ , i.e.,  $\mu_{\phi} = \mu_{\phi}(\pmb{f}_{s})$  and  $\mu_{\psi} = \mu_{\psi}(\pmb{f}_{s})$ .
Under Assumptions 2.1, 2.2 and A.2, the estimation error is upper bounded as follows:
$$
\begin{array}{l} \underset {\boldsymbol {x}, \boldsymbol {y}} {\mathbb {E}} \left[ | f _ {t} (\boldsymbol {x}) - \hat {f} _ {t} (\boldsymbol {x}) | ^ {2} \right] = \underset {\boldsymbol {x}, \boldsymbol {y}} {\mathbb {E}} \left[ \left| \psi (g (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) - \psi (\hat {g} (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) \right| ^ {2} \right] \\ \leq \mathbb {E} _ {\boldsymbol {x}, y} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \left| g (\boldsymbol {x}) - \hat {g} (\boldsymbol {x}) \right| ^ {2} \right] \\ \leq 3 \mathbb {E} _ {\boldsymbol {x}, y} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \left(\left| g (\boldsymbol {x}) - \phi \left(f _ {t} (\boldsymbol {x}), f _ {s} (\boldsymbol {x})\right) \right| ^ {2} \right. \right. \\ + \left| \phi (f _ {t} (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) - \phi (y, \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) \right| ^ {2} \\ \left. + \left| \phi (y, \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) - \hat {g} (\boldsymbol {x}) \right| ^ {2}\right) \Bigg ] \\ \leq 3 \mathbb {E} _ {\boldsymbol {x}, y} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \left| \psi^ {- 1} \left(f _ {t} (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) - \phi \left(f _ {t} (x), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) \right| ^ {2} \right] \\ + 3 \underset {\boldsymbol {x}, y} {\mathbb {E}} \left[ \mu_ {\psi} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) ^ {2} \mu_ {\phi} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) ^ {2} \left| f _ {t} (\boldsymbol {x}) - y \right| ^ {2} \right] \\ + 3 \underset {\boldsymbol {x}, y} {\mathbb {E}} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} | z - \hat {g} (\boldsymbol {x}) | ^ {2} \right] \\ = 3 \underset {\boldsymbol {x}, y} {\mathbb {E}} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \left| \psi^ {- 1} \left(f _ {t} (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) - \phi \left(f _ {t} (\boldsymbol {x}), \boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) \right| ^ {2} \right] \\ + 3 \sigma^ {2} \mathbb {E} _ {\boldsymbol {x}, y} \left[ \mu_ {\psi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \mu_ {\phi} \left(\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})\right) ^ {2} \right] \\ + 3 \underset {\boldsymbol {x}, y} {\mathbb {E}} \left[ \mu_ {\psi} (\boldsymbol {f} _ {\boldsymbol {s}} (\boldsymbol {x})) ^ {2} | z - \hat {g} (\boldsymbol {x}) | ^ {2} \right]. \\ \end{array}
$$
The derivation of this inequality is based on [12]. We use the Lipschitz property of  $\psi$  and  $\phi$  for the first and third inequalities, and the second inequality comes from the inequality  $(a - d)^2 \leq 3(a - b)^2 + 3(b - c)^2 + 3(c - d)^2$  for  $a, b, c, d \in \mathbb{R}$ .
According to this inequality, the upper bound of the estimation error is decomposed into three terms: the discrepancy between the two transformation functions, the variance of the noise, and the estimation error for the transformed data. Although it is intractable to find the optimal solution of  $\phi, \psi, \hat{g}$  that minimizes all these terms together, it is possible to find a solution that minimizes the first and second terms expressed as the functions of  $\phi$  and  $\psi$  only. Obviously, the first term, which represents the discrepancy between the two transformation functions, reaches its minimum (zero) when  $\phi_{f_s} = \psi_{f_s}^{-1}$ . The second term, which is related to the variance of the noise, is minimized when the differential coefficient  $\frac{\partial}{\partial u}\psi_{f_s}(u)$  is a constant, i.e., when  $\psi_{f_s}$  is a linear function. This is verified as follows. From  $\phi_{f_s} = \psi_{f_s}^{-1}$  and the continuity of  $\psi_{f_s}$ , it follows that
$$
\mu_ {\psi} = \max  \left| \frac {\partial}{\partial u} \psi_ {f _ {s}} (u) \right|, \quad \mu_ {\phi} = \max  \left| \frac {\partial}{\partial u} \psi_ {f _ {s}} ^ {- 1} (u) \right| = \frac {1}{\min  | \frac {\partial}{\partial u} \psi_ {f _ {s}} (u) |},
$$
and thus the product  $\mu_{\phi}\mu_{\psi}$  takes the minimum value (one) when the maximum and minimum of the differential coefficient are the same. Therefore, we can write
$$
\phi (y, \boldsymbol {f} _ {\boldsymbol {s}}) = \frac {y - g _ {1} (\boldsymbol {f} _ {\boldsymbol {s}})}{g _ {2} (\boldsymbol {f} _ {\boldsymbol {s}})}, \quad \psi (g (x), \boldsymbol {f} _ {\boldsymbol {s}}) = g _ {1} (\boldsymbol {f} _ {\boldsymbol {s}}) + g _ {2} (\boldsymbol {f} _ {\boldsymbol {s}}) g (\boldsymbol {x}),
$$
where  $g_{1}, g_{2}: \mathcal{F}_{s} \to \mathbb{R}$  are arbitrarily functions. Thus, the minimization of the third term in the upper bound of the estimation error can be expressed as
$$
\min  _ {g _ {1}, g _ {2}, g} \underset {\boldsymbol {x}, y} {\mathbb {E}} | y - g _ {1} (\boldsymbol {f} _ {\boldsymbol {s}}) - g _ {2} (\boldsymbol {f} _ {\boldsymbol {s}}) g (\boldsymbol {x}) | ^ {2}.
$$
As a result, the suboptimal function class for the upper bound of the estimated function is given as
$$
\mathcal {H} = \left\{x \mapsto g _ {1} (\boldsymbol {f} _ {\boldsymbol {s}}) + g _ {2} (\boldsymbol {f} _ {\boldsymbol {s}}) \cdot g _ {3} (\boldsymbol {x}) \mid g _ {1} \in \mathcal {G} _ {1}, g _ {2} \in \mathcal {G} _ {2}, g _ {3} \in \mathcal {G} _ {3} \right\}.
$$
This is the same function class derived in Section 2.1.
(a) Direct learning
(b) Feature extraction
(c) HTL-offset
Figure S.1: Model architectures for the affine model transfer and related procedures. (a) Direct learning predicts outputs using only the original inputs  $x$ , while (b) feature extraction-based neural transfer predicts outputs using only the source features  $f_{s}$ . (c) The HTL procedure proposed in [10] (HTL-offset) constructs the predictor as the sum of  $g_{1}(f_{s})$  and  $g_{3}(\pmb{x})$ . (d) The affine model transfer encompasses these procedures, computing  $g_{1}$  and  $g_{2}$  as functions of the source features and constructing the predictor as an affine combination with  $g_{3}$ .
(d) Affine model transfer
B.1 Eigenvalue Decay of the Hadamard Product of Two Gram Matrices
We experimentally investigated how the decay rate  $s_2$  in Theorem 4.4 is related to the overlap degree in the spaces spanned by the original input  $x$  and source features  $f_s$ .
For the original input  $\pmb{x} \in \mathbb{R}^{100}$ , we randomly constructed a set of 10 orthonormal bases, and then generated 100 samples from their spanning space. For the source features  $\pmb{f}_s \in \mathbb{R}^{100}$ , we selected  $d$  bases randomly from the 10 orthonormal bases selected for  $\pmb{x}$  and the remaining  $10 - d$  bases from their orthogonal complement space. We then generated 100 samples of  $\pmb{f}_s$  from the space spanned by these 10 bases. The overlap number  $d$  can be regarded as the degree of overlap of two spaces spanned by the samples of  $\pmb{x}$  and  $\pmb{f}_s$ . We generated the 100 different sample sets of  $\pmb{x}$  and  $\pmb{f}_s$ .
We calculated the Hadamard product of the Gram matrices  $K_{2}$  and  $K_{3}$  using the samples of  $\pmb{x}$  and  $\pmb{f}_{s}$ , respectively. For the computation of  $K_{2}$  and  $K_{3}$ , all combinations of the following five kernels were tested:
Linear kernel  $k(\pmb {x},\pmb{x}^{\prime}) = \frac{\pmb{x}^{\top}\pmb{x}}{2\gamma^{2}} +1,$
Matérn kernel  $k(\pmb{x}, \pmb{x}^{\prime}) = \frac{2^{1 - \nu}}{\Gamma(\nu)}\left(\frac{\sqrt{2\nu}\|\pmb{x} - \pmb{x}^{\prime}\|_{2}}{\gamma}\right)^{\nu}K_{\nu}\left(\frac{\sqrt{2\nu}\|\pmb{x} - \pmb{x}^{\prime}\|_{2}}{\gamma}\right)$  for  $\nu = \frac{1}{2}, \frac{3}{2}, \frac{5}{2}, \infty$ ,
where  $K_{\nu}(\cdot)$  is a modified Bessel function and  $\Gamma (\cdot)$  is the gamma function. Note that for  $\nu = \infty$ , the Matérn kernel is equivalent to the Gaussian RBF kernel. The scale parameter  $\gamma$  of both kernels was set to  $\gamma = \sqrt{\dim(\boldsymbol{x})} = \sqrt{10}$ . For a given matrix  $K$ , the decay rate of the eigenvalues was estimated as the smallest value of  $s$  that satisfies  $\lambda_{i}\leq \| K\|_{F}^{2}\cdot i^{-\frac{1}{s}}$  where  $\| \cdot \| _F$  denotes the Frobenius norm. Note that this inequality holds for any matrices  $K$  with  $s = 1$  [32].
Figure S.2 shows the change of the decay rates with respect to varying  $d$  for various combinations of the kernels. In all cases, the decay rate of  $K_{2} \circ K_{3}$  showed a clear trend of monotonically decreasing as the degree of overlap  $d$  increases. In other words, the greater the overlap between the spaces spanned by  $\mathbf{x}$  and  $\mathbf{f}_{s}$ , the smaller the decay rate, and the smaller the complexity of the RKHS  $\mathcal{H}_2 \otimes \mathcal{H}_3$ .
B.2 Lattice Thermal Conductivity of Inorganic Crystals
Here, we describe the relationship between the qualitative differences in source features and the learning behavior of the affine model transfer, in contrast to ordinary feature extraction using neural networks.
Figure S.2: Decay rates of eigenvalues of  $K_{2}$  (blue lines),  $K_{3}$  (green lines) and  $K_{2} \circ K_{3}$  (red lines) for all combinations of the five different kernels. The vertical axis represents the decay rate, and the horizontal axis represents the overlap dimension  $d$  in the space where  $x$  and  $f_{s}$  are distributed.